Vault deployment with consul sidecar injection doesn't work due to service count

[qzx]

I've installed consul on a kubernetes cluster with helm and got the service mesh working.

Now I'm installing Vault with this helm chart.

These are the configuration parameters

server:
  annotations: |
    "consul.hashicorp.com/connect-inject": "true"
  enabled: true
  dev:
    enabled: true

Pod doesn't start up and stops on Init and this is in the logs:

% kubectl logs vault-0 consul-connect-inject-init
2021-07-22T16:46:29.839Z [INFO]  Unable to find registered services; retrying
2021-07-22T16:46:30.841Z [INFO]  Unable to find registered services; retrying
2021-07-22T16:46:31.845Z [INFO]  Unable to find registered services; retrying
2021-07-22T16:46:32.847Z [INFO]  Unable to find registered services; retrying
2021-07-22T16:46:33.850Z [INFO]  Unable to find registered services; retrying
2021-07-22T16:46:34.852Z [INFO]  Unable to find registered services; retrying
2021-07-22T16:46:35.854Z [INFO]  Unable to find registered services; retrying
2021-07-22T16:46:36.856Z [INFO]  Unable to find registered services; retrying
2021-07-22T16:46:37.857Z [INFO]  Unable to find registered services; retrying
2021-07-22T16:46:38.860Z [INFO]  Unable to find registered services; retrying
2021-07-22T16:46:38.860Z [INFO]  Check to ensure a Kubernetes service has been created for this application.

After 10 failures:

2021-07-22T15:51:59.299Z [ERROR] Timed out waiting for service registration: error="did not find correct number of services: 4"

Looking at consul-k8s source code at https://github.com/hashicorp/consul-k8s/blob/7d7ce5dc3f9ec74bf21f06b172fe3d97bddef62b/subcommand/connect-init/command.go

I can see:

if len(serviceList) != 2 {
			c.logger.Info("Unable to find registered services; retrying")
			// Once every 10 times we're going to print this informational message to the pod logs so that
			// it is not "lost" to the user at the end of the retries when the pod enters a CrashLoop.
			if registrationRetryCount%10 == 0 {
				c.logger.Info("Check to ensure a Kubernetes service has been created for this application.")
			}
			return fmt.Errorf("did not find correct number of services: %d", len(serviceList))
		}

Going by the count of services it actually finds (4) vs the expected value of 2, there's an indication that things are double.

I therefor rendered out the helm templates and deployed each component individually one by one to identify the problem. It seems that this happens because the helm chart deploys two services to front Vault.. and I can't really understand why that's done at all. What's the point of the headless service?

These are functionally identical?

---
# Source: vault/templates/server-service.yaml
# Service for Vault cluster
apiVersion: v1
kind: Service
metadata:
  name: RELEASE-NAME-vault
  namespace: default
  labels:
    helm.sh/chart: vault-0.13.0
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/managed-by: Helm
  annotations:

spec:
  publishNotReadyAddresses: true
  ports:
    - name: http
      port: 8200
      targetPort: 8200
    - name: https-internal
      port: 8201
      targetPort: 8201
  selector:
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: RELEASE-NAME
    component: server

---
# Source: vault/templates/server-headless-service.yaml
# Service for Vault cluster
apiVersion: v1
kind: Service
metadata:
  name: RELEASE-NAME-vault-internal
  namespace: default
  labels:
    helm.sh/chart: vault-0.13.0
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/managed-by: Helm
  annotations:

spec:
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
    - name: "http"
      port: 8200
      targetPort: 8200
    - name: https-internal
      port: 8201
      targetPort: 8201
  selector:
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: RELEASE-NAME
    component: server

I see no clear way to configure the helm chart to only render the server-service and not the server-headless-service.

Why is this headless service, how can we ignore it?

[vauchok]

Hi all,

faced the same issue.

The versions:
EKS(Kubernetes version 1.19)
consul Helm chart - v0.36.0
vault Helm chart - v0.17.1

p.s. I didn't override the image versions in charts

[mike-code]

I have exactly the same issue but with redis chart. Redis chart also installs "regular" and headless service which results in multiple (two) services pointing at the same pod. Looks like consul doesn't like that and expects only 1 service to be pointing at a pod but I can't find any help in docs on how to handle such cases.

btw, do you also get bunch of 403 (ACL not found) when connect inject container starts?

edit: I just noticed that is Vault, not Consul repo. I will repost this question in Consul repo

[lkysow]

Hi All, this is due to hashicorp/consul-k8s#592. Please follow that issue for updates.

Question for y'all just for my knowledge on use-cases: why are you injecting the sidecar into Vault?

[mike-code]

Question for y'all just for my knowledge on use-cases: why are you injecting the sidecar into Vault?

For me personally (not limited to Vault) because I don't know what I want or don't want registered in the Consul mesh so I register everything.

Actually this question could be extended to whether Consul is useful in single k8s cluster at all (with no multi cluster, multi datacenter or k8s <> non-k8s apps setup). Initially I wanted to use Consul as a distributed KV backend and then I thought: while you're here already, why can't you act as my service mesh even if you duplicate what k8s does already on its own 🤓

[lkysow]

@mike-code I'll answer your question in hashicorp/consul-k8s#849 so as to not pollute this ticket.