Controller not detecting a. service

[petar-nikolovski-cif]

We are having problem with HAProxy not detecting services within the cluster, when routing requests to cert-manager pods. For example, HAProxy controller returns:

022/08/16 13:46:12 ERROR   ingress/ingress.go:245 Ingress 'development/my-service': service 'development/cm-acme-http-solver-498h4' does not exist
2022/08/16 13:46:12 INFO    handler/https.go:123 removing client TLS authentication

And the rule from Ingress is like so:

  ingressClassName: external-haproxy
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-498h4
            port:
              number: 8089
        path: /.well-known/path/to/acme-challenge
        pathType: ImplementationSpecific

I've checked haproxy.cfg and I cannot find the internal IP either of the mentioned service or the pod.

We are also having split-horizon dns with two HAProxies - internal and external. I've also found the offending line: https://github.com/haproxytech/kubernetes-ingress/blob/v1.8.3/pkg/ingress/ingress.go#L245

Do you know why this happens? It seems that haproxy cannot find a service, despite service existing in the cluster.

[ocdi]

I too just had this issue happen. I ended up restarting the haproxy controller deployment and the freshly started pods was able to see the service and traffic worked. I should note I'm using the latest ingress version 1.8.3.

[nopsenica]

Yes, this issue occurred at my environment too after adding new service. Error message was the same.
The only help to make new service accessible was to restart haproxy controller.

[ivanmatmati]

Hi thanks for reporting, to help in tracking the issue could you add description of the environment ? Which version of controller is used , how many services and ingresses are there, how old was the controller, if any event occured in the cluster (like a restart of pods, etc), has the service been found and then lost, etc. Thanks in advance.

[ocdi]

I've got quite a few ingresses (many different hostnames pointing to a similar backend service).

k get ingress | wc -l
298

It would appear I've got 297 ingresses currently. This was for a new backend service, but I also noted that certmanager's ingress was not able to find the service to validate the certificate request.

[petar-nikolovski-cif]

• Controller installed via Helm kubernetes-ingress-1.22.4
• Application version is 1.8.3
• 3 pods for external HAProxy, and they are old around 40 days, there are no pod restarts
• I've checked haproxy.cfg service does not appear there
• There are around 55 ingress split between two HAProxy deployments (one controller for internal/vpc, one for external connectivity to cluster)

[petar-nikolovski-cif]

@ivanmatmati It seems that restarting did the job! @ocdi @nopsenica @ivanmatmati thank you! But why did the controller failed to find the configuration?

[dschuldt]

We experience the same issue since version 1.8.x. I mentioned it in #460 originally. Since @ivanmatmati asked me to move to this issue, here are my findings summarized:

• Restart of controller helps
• We see the same ERROR log lines like @petar-nikolovski-cif posted
• memory usage increases over time (see attached image)
• maybe the error originates from store.go (lines 186-189)

[ivanmatmati]

Hi @dschuldt , thanks for that. Actually the code lines you're pointing to are rather the consequence than the cause. We still need to figure out what is happening.We're planning to make a long term running of controller to be in better position to observe the events/issues of this matter. @petar-nikolovski-cif That's the question still to solve. We'll keep you informed. Thanks.

[mblixter]

I'm experiencing the exact same issue with the controller not being able to find certain services. I tried to restart the controller deployment, and also tried to uninstall and install chart version 1.22.4, but it didn't help.
I had to downgrade to chart version 1.21.1 for it to start working again. I did not try any of the charts between 1.22.4 and 1.21.1, so I do not know if a newer version than 1.21.1 works. All I know is that 1.22.4 does not work.

[ocdi]

On a test cluster I tried re-creating the bug by abusing the cluster - creating 1000 services (admittedly all pointing to the same deployment), and 1000 ingresses. Doing that all together was quite intense and it used lots of CPU, causing it to scale up. I think it struggled with all the updating ingress loadbalancer status, but ultimately it worked.

I deleted all those 1000 services, waited for it to settle then created smaller batches of services+ingresses. Turns out my config had a http check configured, which because it was hitting the same pod, it overloaded that and caused that to crash. Oops. Aside from that, the controller appeared to keep everything in sync. It has only been running for 95 minutes so not a full test.

The only thing I was thinking that might cause an issue is perhaps interruptions to the control plane, like when I upgrade it. So I upgraded the control plane to a newer patch version, without upgrading the nodes, but that too was fine.

I kind of wish this triggered the bug, but I suspect it isn't that easy. If there are any other tests like this, that would be helpful, I'm happy to do that.

[ivanmatmati]

Thanks @ocdi for your effort and your report. We still have to run the test we're planning to do.This is indeed not a trivial issue. It can imply multiple actors with cluster, resources and network events, and require a timing condition as well.

[mblixter]

In my case, it had nothing to do with resources or load. There was no real load on the cluster, and I could create working services and ingress' in other namespace. It was just services in one particular NS that the Haproxy controller could not find services in. I tried deleting and recreating the NS, and also tried creating several services that worked in other NS, in the affected NS, but Haproxy could still not find the services.
I do not know if it could have something to do with the name of the NS for instance. My NS was named "vanilla-deploy". Could it have something to do with characters or length of the name? I haven't had time to investigate any further.

[junyeong-huray]

I am struggling with the same issue.

2022/09/06 05:14:58 ERROR   ingress/ingress.go:251 Ingress 'landmark/crate-private': service 'landmark/crate' does not exist

[idonca]

It is happening also for us, we are running 1.8.3 docker image version, and even if we upgraded to 1.8.4 docker image version, is the same behavior. As @ocdi said: seems to be a resolution when a new service is deployed: delete the haproxy pods, delete the service pods, and after that is working. We would love to see a hotfix for this asap.