imagePullSecrets not found?

[jcassee]

I just installed Keel on a Kubernetes 1.12 cluster on DigitalOcean using the provided resources. I have created a deployment with an image from a private Docker Hub repository. The default service account has the correct image pull secrets; the image gets pulled and the pods are started. (imagePullSecrets is set in the pod.)

However, I still see this in the Keel logs over and over again:

time="2018-11-14T00:41:30Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https:/
/index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\
":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\
":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="namespace:default,image:index.docker.
io/goabout/automat,provider:kubernetes,trigger:poll,sched:@every 5m,secrets:[]"

So it seems Keel does not pick up the imagePullSecrets correctly?

Anything I can do to debug the problem further?

[rusenask]

hi, do you see any other logs (before this line) with info/error level that mention secrets?

[jcassee]

I am embarrassed to say that although the default service account has the correct image pull secrets, the pod itself also specified them, and it contained a typo.

Apologies for the confusion! (And I'm really digging Keel, by the way.)

[jcassee]

Ah no, that was a different issue. This one is alive and well, I'm afraid. So, to your question, here is a larger section of the log:

time="2018-11-14T21:22:23Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="namespace:default,image:index.docker.io/goabout/automat,provider:kubernetes,trigger:poll,sched:@every 5m,secrets:[]"
time="2018-11-14T21:22:24Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="goabout/automat:master"
time="2018-11-14T21:22:24Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="namespace:default,image:index.docker.io/goabout/automat,provider:kubernetes,trigger:poll,sched:@every 5m,secrets:[]"
time="2018-11-14T21:22:24Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="goabout/automat:master"
time="2018-11-14T21:22:24Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="namespace:default,image:index.docker.io/goabout/automat,provider:kubernetes,trigger:poll,sched:@every 5m,secrets:[]"
time="2018-11-14T21:22:25Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://index.docker.io/v2/goabout/coconut-backend/manifests/latest: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/coconut-backend\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="goabout/coconut-backend:latest"
time="2018-11-14T21:22:25Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://index.docker.io/v2/goabout/coconut-backend/manifests/latest: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/coconut-backend\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="namespace:default,image:index.docker.io/goabout/coconut-backend,provider:kubernetes,trigger:poll,sched:@every 5m,secrets:[]"
time="2018-11-14T21:22:25Z" level=error msg="trigger.poll.manager: got error(-s) while watching images" error="encountered errors while adding images: Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\"), Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\"), Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\"), Get https://index.docker.io/v2/goabout/coconut-backend/manifests/latest: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/coconut-backend\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")"
time="2018-11-14T21:22:26Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\": \\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="goabout/automat:master"
time="2018-11-14T21:22:26Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="namespace:default,image:index.docker.io/goabout/automat,provider:kubernetes,trigger:poll,sched:@every 5m,secrets:[]"

[rusenask]

There should have been some logs where it's setting up the watcher, right after keel detects that deployment. If you restart keel, it should print out some logs about the missing secret.

[jcassee]

Ah, here are the first logs:

time="2018-11-13T22:41:58Z" level=info msg="extension.credentialshelper: helper registered" name=aws                                
time="2018-11-13T22:41:58Z" level=info msg="bot: registered" name=slack                                                             time="2018-11-13T22:41:58Z" level=info msg="keel starting..." arch=amd64 build_date=2018-10-17T214939Z go_version=go1.10.3 os=linux revision=fedaf23b version=0.12.0                                                                                                    time="2018-11-13T22:41:58Z" level=info msg="extension.notification.slack: sender configured" channels="[notifications]" name=slack  
time="2018-11-13T22:41:58Z" level=info msg="notificationSender: sender configured" sender name=slack                                
time="2018-11-13T22:41:58Z" level=info msg="provider.kubernetes: using in-cluster configuration"                                    
time="2018-11-13T22:41:58Z" level=info msg="provider.defaultProviders: provider 'kubernetes' registered"                            time="2018-11-13T22:41:58Z" level=info msg="extension.credentialshelper: helper registered" name=secrets                            
time="2018-11-13T22:41:58Z" level=info msg="webhook trigger server starting..." port=9300                                           
time="2018-11-13T22:41:58Z" level=info msg="trigger.poll.manager: polling trigger configured"                                       
time="2018-11-13T22:42:00Z" level=info msg=started context=buffer                                                                   
time="2018-11-13T22:42:00Z" level=info msg=started context=watch resource=deployments                                               
time="2018-11-13T22:42:00Z" level=info msg=started context=watch resource=statefulsets                                              
time="2018-11-13T22:42:00Z" level=info msg=started context=watch resource=daemonsets                                                time="2018-11-13T22:42:00Z" level=info msg=started context=watch resource=cronjobs                                                  time="2018-11-13T22:42:03Z" level=info msg="trigger.poll.RepositoryWatcher: new watch tag digest job added" digest="sha256:c47fd9b4eda5fab58835c2dcf6dd6590b69972c9398a6eab84fef0858a6e2e2c" image="datadog/agent:latest" job_name="index.docker.io/datadog/agent:latest
" schedule="@every 60m"                                                                                                             
time="2018-11-13T22:42:04Z" level=warning msg="secrets.defaultGetter: failed to get secret" error="secrets \"dockerhub\" not found" 
image=index.docker.io/goabout/automat namespace=default secret_ref=dockerhub                                                        time="2018-11-13T22:42:04Z" level=warning msg="secrets.defaultGetter.lookupSecrets: docker credentials were not found among secrets"
 image=index.docker.io/goabout/automat namespace=default provider=kubernetes registry=index.docker.io secrets="[dockerhub]"

The deployment itself, however, has different image pull secrets:

$ kubectl get pod master-automat-flask-7f4fdcb9b-kb6p8 -o yaml | grep -A1 imagePullSecrets
  imagePullSecrets:
  - name: docker-hub-549m7fd97t

The deployment has changed since keel was started. So I restarted keel to see what would happen. The new logs do not have the same error about the secret, but still cannot pull the image:

time="2018-11-14T22:48:14Z" level=info msg="extension.credentialshelper: helper registered" name=aws
time="2018-11-14T22:48:14Z" level=info msg="bot: registered" name=slack
time="2018-11-14T22:48:14Z" level=info msg="keel starting..." arch=amd64 build_date=2018-10-17T214939Z go_version=go1.10.3 os=linux revision=fedaf23b version=0.12.0
time="2018-11-14T22:48:14Z" level=info msg="extension.notification.slack: sender configured" channels="[notifications]" name=slack
time="2018-11-14T22:48:14Z" level=info msg="notificationSender: sender configured" sender name=slack
time="2018-11-14T22:48:14Z" level=info msg="provider.kubernetes: using in-cluster configuration"
time="2018-11-14T22:48:14Z" level=info msg="provider.defaultProviders: provider 'kubernetes' registered"
time="2018-11-14T22:48:14Z" level=info msg="extension.credentialshelper: helper registered" name=secrets
time="2018-11-14T22:48:14Z" level=info msg="trigger.poll.manager: polling trigger configured"
time="2018-11-14T22:48:14Z" level=info msg="webhook trigger server starting..." port=9300
time="2018-11-14T22:48:15Z" level=info msg=started context=buffer
time="2018-11-14T22:48:15Z" level=info msg=started context=watch resource=deployments
time="2018-11-14T22:48:15Z" level=info msg=started context=watch resource=statefulsets
time="2018-11-14T22:48:15Z" level=info msg=started context=watch resource=daemonsets
time="2018-11-14T22:48:15Z" level=info msg=started context=watch resource=cronjobs
time="2018-11-14T22:48:18Z" level=info msg="trigger.poll.RepositoryWatcher: new watch tag digest job added" digest="sha256:c47fd9b4eda5fab58835c2dcf6dd6590b69972c9398a6eab84fef0858a6e2e2c" image="datadog/agent:latest" job_name="index.docker.io/datadog/agent:latest" schedule="@every 60m"
time="2018-11-14T22:48:20Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="goabout/automat:master"
time="2018-11-14T22:48:20Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="namespace:default,image:index.docker.io/goabout/automat,provider:kubernetes,trigger:poll,sched:@every 5m,secrets:[]"
time="2018-11-14T22:48:20Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="goabout/automat:master"

[rusenask]

According to this time="2018-11-14T22:48:20Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://index.docker.io/v2/goabout/automat/manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"goabout/automat\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="namespace:default,image:index.docker.io/goabout/automat,provider:kubernetes,trigger:poll,sched:@every 5m,secrets:[]" it seems that it didn't associate any secrets with this image. Are the imagePullSecrets defined in the deployment manifest?

[jcassee]

Yes and no. It is defined in the default service agent. But if you look at the effective deployment manifest (see my previous message, halfway in the text), the image pull secrets are there.

[rusenask]

Could you show the full manifests? feel free to remove anything sensitive, just want to look at the structure. This is the code that gets imagePullSecrets https://github.com/keel-hq/keel/blob/master/internal/k8s/resource.go#L249-L261, maybe needs refreshing :)

[jcassee]

Aha, I inspected the pod, not the deployment. Apparently, if you depend on the service account to inject the image pull secrets, they are injected into the pod, but not the deployment!

What do you think: do you expect Keel to support this use-case?

[rusenask]

so they don't appear in the deployment? This might have changed then, as I was testing this feature with deployments and not pods when I was developing it :/

[jcassee]

Indeed, they do not appear in the deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
    keel.sh/pollSchedule: '@every 5m'
    kubectl.kubernetes.io/last-applied-configuration: |
      [...]
  creationTimestamp: 2018-11-13T22:52:29Z
  generation: 3
  labels:
    app: automat
    component: flask
    deployment: development
    keel.sh/match-tag: "true"
    keel.sh/policy: force
    keel.sh/trigger: poll
    version: master
  name: master-automat-flask
  namespace: default
  resourceVersion: "779113"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/master-automat-flask
  uid: cc2123fe-e796-11e8-9fe6-86176dc8c952
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: automat
      component: flask
      deployment: development
      version: master
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: automat
        component: flask
        deployment: development
        version: master
      name: flask
    spec:
      containers:
      - env:
        [...]
        image: goabout/automat:master
        imagePullPolicy: Always
        name: flask
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2018-11-14T11:19:14Z
    lastUpdateTime: 2018-11-14T11:19:14Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: 2018-11-13T22:52:29Z
    lastUpdateTime: 2018-11-14T13:19:52Z
    message: ReplicaSet "master-automat-flask-7f4fdcb9b" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

And the pod has:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: memory request for container
      flask; memory limit for container flask'
  creationTimestamp: 2018-11-14T13:19:17Z
  generateName: master-automat-flask-7f4fdcb9b-
  labels:
    app: automat
    component: flask
    deployment: development
    pod-template-hash: 7f4fdcb9b
    version: master
  name: master-automat-flask-7f4fdcb9b-kb6p8
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: master-automat-flask-7f4fdcb9b
    uid: e3764a94-e80f-11e8-9fe6-86176dc8c952
  resourceVersion: "779102"
  selfLink: /api/v1/namespaces/default/pods/master-automat-flask-7f4fdcb9b-kb6p8
  uid: e378caa3-e80f-11e8-9fe6-86176dc8c952
spec:
  containers:
  - env:
    [...]
    image: goabout/automat:master
    imagePullPolicy: Always
    name: flask
    ports:
    - containerPort: 80
      name: http
      protocol: TCP
    resources:
      limits:
        cpu: "1"
        memory: 512Mi
      requests:
        cpu: 500m
        memory: 256Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-qf9ch
      readOnly: true
  dnsPolicy: ClusterFirst
  imagePullSecrets:
  - name: docker-hub-549m7fd97t
  nodeName: reverent-kapitsa-1u2
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-qf9ch
    secret:
      defaultMode: 420
      secretName: default-token-qf9ch
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2018-11-14T13:19:17Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2018-11-14T13:19:51Z
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2018-11-14T13:19:51Z
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: 2018-11-14T13:19:17Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://03b7e15e7b7add0720cf6902fff3c2e3573770307d19b7bc5dc300d06dc21388
    image: goabout/automat:master
    imageID: docker-pullable://goabout/automat@sha256:d385a7cb7ca3be5e18364d183ccf12b0ea115d0a1ecaa59d951b7013896e75af
    lastState: {}
    name: flask
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2018-11-14T13:19:50Z
  hostIP: 10.133.78.178
  phase: Running
  podIP: 10.244.58.2
  qosClass: Burstable
  startTime: 2018-11-14T13:19:17Z

[rusenask]

okay, thanks :) I think we have enough info to debug this further. I have added today some initial e2e tests, will be quite easy to add one and start testing. I suspect that this happened recently with the change from extensions to apps for the deployments.

[jcassee]

Thanks, @rusenask! Happy to give you more work... ;-)

[travisghansen]

I think I'm bumping into a similar issue:

time="2018-11-22T19:33:22Z" level=warning msg="secrets.defaultGetter.lookupSecrets: docker credentials were not found among secrets" image=index.docker.io/linuxserver/unifi namespace=default provider=kubernetes registry=index.docker.io secrets="[registry-credentials]"

In my case, I just add the credentials to all deployments but they aren't needed for this particular container or the pod in general (credentials are for a private registry and the container is simply on docker hub). Should this scenario be supported? It seems like since pull secrets are not container specific within a pod this should be gracefully accounted for?

On a related note, since the labels are applied at the deployment level, does keel watch all containers in the deployment and apply the same policy/etc?

[rusenask]

Hi, image pull secrets do get added to the pod spec, it just appears that when queried from a "higher level" via deployments, they do not appear any more there. I will fix this when I get a chance, maybe this weekend, currently too occupied with my main work :)
The idea is that with polling trigger it would just lookup whatever secrets kubernetes is using to access registries and re-use them instead of specifying anything.

On a related note, since the labels are applied at the deployment level, does keel watch all containers in the deployment and apply the same policy/etc?

No, all deployments/daemonsets/statefulsets are just a constructs that manage pod specs, we don't really care about them since if we update that deployment workload, k8s will update individual pods.

[travisghansen]

@rusenask in my case (1.11 installed via rke) the secrets are in the deployment. Any other info I can provide to help or should my case be a separate issue altogether?

[rusenask]

yeah, maybe a bit more info on how you define them would be helpful and in another issue :)

[Antiarchitect]

Have an absolute similar issue with Gitlab registry, chart 0.7.2, keel 0.12.0

[rusenask]

@Antiarchitect did it work before?

[rusenask]

Added e2e test with private registry, besides a problem where it wouldn't like passwords that contain ":", couldn't find anything. Please provide your k8s secret format, I was creating with a kubectl command:

kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

And also through the client-go. I imagine that it doesn't process some secret format. Also, added some more logging to help find the problem.

Please try out 0.13.0-rc2 and provide logs as well, if it doesn't work.

[Antiarchitect]

In yaml it looks like

.dockerconfigjson: <BASE64>

Inside:

{"auths":{"someserver:5555":{"username":"gitlab+deploy-token-1","password":"somepassword","email":"fake@someserver","auth":"<BASE64>"}}}

[Antiarchitect]

Still no luck with 0.13.0-rc2

[Antiarchitect]

@rusenask

[rusenask]

I am out of town for a few days, on a workshop with a client till tomorrow evening. What command are you using to create the secret? I need to replicate it but I suspect that my secret is in the same format as you have.

End-to-end for private registry is here https://github.com/keel-hq/keel/blob/master/tests/acceptance_polling_test.go#L168-L307.

Secret create code is here: https://github.com/keel-hq/keel/blob/master/tests/acceptance_polling_test.go#L227-L240

Also, logs would be useful as I have added a bit more info to the logs.

[Antiarchitect]

kubectl create secret docker-registry gitlab-registry-credentials \
  --docker-server=someserver:5555 \
  --docker-email=fake@someserver \
  --docker-username=gitlab+deploy-token-1 \
  --docker-password=[PASSWORD]

[rusenask]

thanks @Antiarchitect, with your secret on gitlab registry I am getting the same error, digging in :|

[rusenask]

@Antiarchitect could you please provide a sample deployment.yaml manifest that I could use to replicate this? Unfortunately the gitlab test did work for me, I have added and e2e test for the private gitlab repo as well.
Tried creating both through the API and just a yaml file and in both cases keel found the secret and used it. Example:

apiVersion: apps/v1
kind: Deployment
metadata: 
  name: pushwf  
  labels: 
    name: "pushwf"
    # force policy will ensure that deployment is updated
    # even when tag is unchanged (latest remains)
    keel.sh/policy: major
    keel.sh/trigger: poll
    # keel.sh/approvals: "1"
  annotations:
    keel.sh/pollSchedule: "@every 30s"
spec:
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: pushwf
  template:
    metadata:
      name: pushwf
      labels:
        app: pushwf
    spec:
      imagePullSecrets:
        - name: gitlab-registry-credentials
      containers:
        - image: registry.gitlab.com/karolisr/keel:0.1.0          
          imagePullPolicy: Always # this is required to force pull image     
          name: pushwf
          ports:
            - containerPort: 8500
          livenessProbe:
            httpGet:
              path: /
              port: 8500
            initialDelaySeconds: 10
            timeoutSeconds: 5    

[Antiarchitect]

{{ if .Values.core.web.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ template "example-chart.name" . }}-core-web
  labels:
    app: {{ template "example-chart.name" . }}-core-web
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
    keel.sh/match-tag: "true"
    keel.sh/policy: force
    keel.sh/trigger: poll
spec:
  replicas: {{ .Values.core.web.replicaCount }}
  selector:
    matchLabels:
      app: {{ template "example-chart.name" . }}-core-web
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ template "example-chart.name" . }}-core-web
        release: {{ .Release.Name }}
      annotations:
        forcePullImage: {{ now | quote }}
    spec:
      imagePullSecrets:
        - name: gitlab-registry-credentials
      initContainers:
        - name: rails-migration
          image: {{ .Values.core.image }}
          imagePullPolicy: {{ .Values.core.imagePullPolicy }}
          envFrom:
            - configMapRef:
                name: {{ template "example-chart.name" . }}-core-env
            - secretRef:
                name: {{ template "example-chart.name" . }}-core-env
          command: ["bundle"]
          args:
            - "exec"
            - "rails"
            - "db:migrate"
      containers:
        - name: rails-workers
          image: {{ .Values.core.image }}
          imagePullPolicy: {{ .Values.core.imagePullPolicy }}
          ports:
            - name: http
              containerPort: 3000
              protocol: TCP
          envFrom:
            - configMapRef:
                name: {{ template "example-chart.name" . }}-core-env
            - secretRef:
                name: {{ template "example-chart.name" . }}-core-env
          command: ["bundle"]
          args:
            - "exec"
            - "rails"
            - "server"
            - "-b"
            - "0.0.0.0"
            - "-p"
            - "3000"
{{ end }}

[rusenask]

ah, looks a lot like a helm chart :) @jcassee are you also using helm? I haven't tried using helm with secrets. It might be something to do with how we get pods that are managed by helm.

Can someone who's getting this error enable debugging for Keel? Set in the deployment manifest:

DEBUG=true

For helm provider we should start seeing these errors: https://github.com/keel-hq/keel/blob/master/secrets/secrets.go#L104-L128. I am still finishing up other PR but once I am done with it I hope I have enough info to fix this one.
Ideally I will also write some e2e tests for the helm provider.. Could it be that you start experiencing this issue once you switch from extension to apiVersion: apps/v1 for the manifests?

[jcassee]

@rusenask No, I am applying resources directly. I am using Kustomize, but that is just a preprocessor for kubectl apply.

[Antiarchitect]

I turn helm off in keel settings.

[rusenask]

It might have been due to slightly different registry name and secret name (can be scheme, port, path). I have improved matching, please try out 0.13.0-rc3 :)

[jcassee]

@rusenask 0.13.0-rc3 does not solve the problem for me. I still see these logs:

time="2018-12-12T10:06:19Z" level=info msg="extension.credentialshelper: helper registered" name=aws
time="2018-12-12T10:06:19Z" level=info msg="bot: registered" name=slack
time="2018-12-12T10:06:19Z" level=info msg="keel starting..." arch=amd64 build_date=2018-12-12T001601Z go_version=go1.10.3 os=linux revision=d0fc99c5 version=0.13.0-rc3
[...]
time="2018-12-12T10:06:27Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://index.docker.io/v2/.../manifests/master: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":[{\\\"Type\\\":\\\"repository\\\",\\\"Class\\\":\\\"\\\",\\\"Name\\\":\\\"...\\\",\\\"Action\\\":\\\"pull\\\"}]}]}\\n\")" image="..." password= username=

[rusenask]

Hi guys, sorry for the lack of activity but I have been quite busy with my regular job :)

Alright, what I really need in this scenario is:

1. Deployment.yaml for any workload that I could deploy myself
2. Secret example (how do you create it - what command you use or yaml of the secret as a template)
3. Which registry are you using (so I could configure it myself)

From the e2e tests I couldn't replicate it with my own images. There are multiple ways how to create secrets in k8s and their formats, same goes with service accounts.

If I could replicate it, I am sure I could fix it too.

[rusenask]

We had a bunch of similar issues, there were changes to secret management since then, closing this.