Container consul-template was using 1004852Ki, which exceeds its request of 0.

[imvishalvyas]

I have deployed vault on kubernetes for secret management for my deployments. Now my deployments are include vault init container, Consul-template container and third one is my app container. i am using multiple deployments in my cluster using consul-template. I am using it since last 2 months, But from last 2 days. My all the pods getting evicted and restarted due to consul template using high resource of memory.

I am getting below error when i search the pod logs.

The node was low on resource: memory.  Container consul-template was using 1004852Ki, which exceeds its request of 0.

I am using 4G ram and 4 core CPU on my kubernetes GCP.

Consul Template version

I am using docker image of consul-template 'hashicorp/consul-template:alpine'

Configuration

I have used consul container along with my app and init container of vault. below is my consul container specs.

      containers:
        # The consul-template container will pull secrets from Vault and expose
        # them as files on disk.
      - name: consul-template
        image: hashicorp/consul-template:alpine
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 100m
        volumeMounts:
        - name: vault-token
          mountPath: /home/vault
        - name: config
          mountPath: /etc/consul-template
        - name: shared-data
          mountPath: /etc/secrets
        - name: vault-tls
          mountPath: /etc/vault/tls
        env:
        - name: HOME
          value: /home/vault
        - name: VAULT_ADDR
          value: https://myvaultip:8200
        - name: VAULT_CACERT
          value: /etc/vault/tls/ca.pem
        args:
          [
            "-config=/etc/consul-template/consul-template-config.hcl",
            #"-log-level=debug",
          ]

My pods getting evicted and restarted 2 or 3 times in a day.

[jboero]

I don't actually see any memory requests in your init container resources? Something like:

    limits:
      memory: 512Mi
    requests:
      memory: 256Mi```

[eikenb]

Hey @imvishalvyas, thanks for the report.

Assuming you are using the latest version available at the time... you might have hit the regression on non-renewable vault secrets. Update to the latest version (0.21.3) that I released today and it may fix the issue.

Please let me know if this resolves it for you. Thanks.

[imvishalvyas]

@eikenb I have tried with latest version of consul template. it worked for some time and after that my pods getting crashed and getting this error. After some time app automatic start and running and throwing this error.

E 2019-09-06T13:06:32.520792808Z * permission denied (retry attempt 6 after "32s")
 
E 2019-09-06T13:07:04.869593484Z 2019/09/06 13:07:04.869372 [WARN] (view) vault.read(secret/kubernetes/cluster/dev/my-app/config): vault.read(secret/kubernetes/cluster/dev/my-app/config): Error making API request.
 
E 2019-09-06T13:07:04.869644198Z 
 
E 2019-09-06T13:07:04.869651580Z URL: GET https://my-vault:8200/v1/secret/kubernetes/cluster/dev/my-app/config
 
E 2019-09-06T13:07:04.869657602Z Code: 403. Errors:
 

@jboero I have also tried with resource limit which you suggest. but not working.

[eikenb]

@imvishalvyas .. That looks different from your original posted issue (the memory issue). Are you still getting the memory error? If not, maybe you could add the missing information (config, template, trace, etc) or file a new issue with it. It's hard to help you diagnose what is wrong with what has been provided. Thanks.

[imvishalvyas]

@eikenb Thanks for the support. memory issue has gone, after update it to the latest version, you can close this for that issue, But after update latest version new issue come which is 403 permission denied.