Services not showing up in intentions tab in the UI

[patoarvizu]

If I access the UI, I can see my services (with the -proxy suffix) plus the consul service, and I can also see the services via the CLI (kubectl exec ... consul catalog services), but when I go to the Intentions tab, the dropdown menu only shows * (All Services) and consul, and if I try to manually enter a service name, it lets me create it but I get Use a future Consul Service called '<servicename>' when entering the name.

The strange thing is that if I create an intention anyway, it does take effect, but only if I create it without the -proxy suffix (i.e. the name of the services based on their annotation and/or the name of the first container). The behavior is the same when creating the intention via the CLI.

Is this the intended Consul behavior? Or is that something specific to consul-helm/consul-k8s?

I'm using consul-helm 0.3.0 (installed with Helm 2.10), consul-k8s 0.2.1, and Kubernetes 1.9.4, running on EC2 (not EKS). I have the sync catalog enabled but defaulted to false and both toConsul and toK8S are set to false.

[johncowen]

Hi @patoarvizu

Thanks for submitting the issue. Just to try to drill down on this more, could you have a look at the output of the /v1/connect/intentions?dc=<your-dc-name> and /v1/internal/ui/services?dc=<your-dc-name> requests?

You can do this either via curl, or by opening the Web Inspector in your browser and clicking the [Network] tab, and then navigating to the page you mention above.

If you could let me know if those two requests give you back any results that would be great!

Thanks,

John

[patoarvizu]

The response for /intentions is [], which is expected since I don't have any intentions, and the (shortened and edited) response for /services is:

[
  {
    "Kind": "connect-proxy",
    "Name": "service1-proxy",
    "Tags": null,
    "Nodes": [
      "ip-1-2-3-4.ec2.internal"
    ],
    "ChecksPassing": 3,
    "ChecksWarning": 0,
    "ChecksCritical": 0,
    "ExternalSources": null
  },
  {
    "Kind": "connect-proxy",
    "Name": "service2-proxy",
    "Tags": null,
    "Nodes": [
      "ip-1-2-3-4.ec2.internal"
    ],
    "ChecksPassing": 3,
    "ChecksWarning": 0,
    "ChecksCritical": 0,
    "ExternalSources": null
  },
  {
    "Kind": "connect-proxy",
    "Name": "service3-proxy",
    "Tags": null,
    "Nodes": [
      "ip-1-2-3-4.ec2.internal"
    ],
    "ChecksPassing": 3,
    "ChecksWarning": 0,
    "ChecksCritical": 0,
    "ExternalSources": null
  },
  {
    "Name": "consul",
    "Tags": null,
    "Nodes": [
      "consul-server-0",
      "consul-server-1",
      "consul-server-2"
    ],
    "ChecksPassing": 3,
    "ChecksWarning": 0,
    "ChecksCritical": 0,
    "ExternalSources": null
  }
...
]

[adilyse]

Thanks for the additional detail! A couple more questions about your setup...

I can tell you're running Connect because of the -proxy instances, but do you have the catalog sync enabled as well? If not, the services won't be available in the Consul cluster.

Additionally, what are the Kubernetes service types of the services you've deployed? We don't currently sync ClusterIP services, so that might be contributing to the situation.

[patoarvizu]

Yes, I'm using Connect, sorry if that wasn't clear. I do have catalog sync enabled, but it's defaulted to false and both toConsul and toK8s are set to false as mentioned above, so I guess it's effectively disabled?

In any case, I understand that ClusterIP services are not being synced, and while that's the way I normally wire my services together, I'm trying to see if I can use Consul's service discovery instead, and that is actually working for me. I'm able to wire together services by using the consul.hashicorp.com/connect-service-upstreams: "serviceA:10000" annotation for example and talking to serviceA via the sidecar proxy on localhost:10000. But that's what's kind of confusing, I'm annotating my pods with e.g. consul.hashicorp.com/connect-service: "serviceB", and those pods are getting the sidecar injected (via the consul.hashicorp.com/connect-inject annotation), so the corresponding serviceB-proxy Consul service is being registered, or at least that's what it looks like from both /v1/internal/ui/services and consul catalog services. But in reality, to do things like connecting services together or to create intentions, I need to refer to them via the actual service name, or at least the one defined by the consul.hashicorp.com/connect-service annotation (e.g. serviceB) and not the name of their proxy service (e.g. serviceB-proxy), so there seems to be some some gap here at least in terms of documentation. In other words, the current state of things gives the impression that only the proxies are being registered as Consul services, but in reality you can reference services directly in annotations or intentions. I hope that made some sense.

To clarify: functionality-wise Connect is doing what I had expected it to do so far (at least in regards to service discovery), but it feels to me that things like this are not very explicit in the documentation.

I'm not sure if there are any actions items here (and if there are, I would be happy to help in any way I can) because nothing is really "broken", but if there is some documentation that maybe I missed and would help me understand this, I would appreciate if you can point me in that direction.

Also, if you think this is more of an issue I should open in the consul-k8s or directly on the consul repo, please let me know.

[johncowen]

Hi @patoarvizu ,

Thanks for the additional info, pretty sure this isn't a UI issue, the dropdown in the intentions tab you talk about about above will only show services that are not a Kind: connect-proxy.

I'll leave it to @adilyse to help you out with this one.

Thanks again!

👋

[adilyse]

Again, thank you for the detailed information, it helps a lot!

If nothing else, there is definitely some opportunity for us to improve the documentation around this situation. The way things work at the moment, proxies are automatically registered directly with the Consul cluster, so they will show up even with catalog sync disabled. This seems unintuitive, so it should be called out in the documentation.

I'm also doing some additional investigation to figure out if we should recommend k8s -> consul syncing for services that are using Connect in Kubernetes. This might make the situation clearer, especially with regards to intentions and service discovery.

[afriha]

Hello,

I'm currently using consul installed with helm chart. I'm facing the same "issue". Well, I wouldn't call it an issue since consul is working as intended and I'm able to write intentions using the name I gave to my services through the service-name annotation. But same as @patoarvizu , I can't list all services (only the -proxy services showing up) in the UI along with k8s' synced services. I'd like to know if there is an issue with my configuration or it's working the way it's meant to.

Thanks in advance.

[adilyse]

After looking at this farther, we've decided that it makes sense to go ahead and register the services behind the proxies as well as just the proxies when using Connect. PR #81 adds support for this and I've verified that it allows the Connect services to be available in the Intentions UI dropdown menus. This will be released as part of v0.7.0.

Thanks for helping us make the user experience better!