GCR PubSub Subscriber only sees one tag when multiple are pushed

[theckman]

Hey there,

We're currently using keel with Google Container Registry (GCR) + PubSub subscription and the helm provider for deployments. We're using the Google Cloud Container Builder (GCCB) to build our images and push them to GCR. In our builds, we create an image with the SemVer tag and one with the name of the git branch the image was built from. We sometimes cut pre-release off branches to facilitate testing of more experimental changes, which is why we've added support for tagging with either.

When these images are being pushed to GCR, we were noticing that keel seemed to only receive a message for one of the images being pushed (the one with the git branch and not the SemVer tag). This was causing keel to, rightfully, refuse to deploy the image:

time="2017-07-25T23:15:36Z" level=debug msg="trigger.pubsub: got message" action=INSERT image_name="dd-engineering/identity:master" tag=master 
time="2017-07-25T23:15:36Z" level=info msg="provider.kubernetes: processing event" registry= repository=gcr.io/dd-engineering/identity tag=master 
time="2017-07-25T23:15:36Z" level=info msg="provider.helm: processing event" registry= repository=gcr.io/dd-engineering/identity tag=master 
time="2017-07-25T23:15:36Z" level=info msg="provider.kubernetes: no impacted deployments found for this event" image=gcr.io/dd-engineering/identity tag=master 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
2017/07/25 23:15:36 warning: destination for Annotations is a table. Ignoring non-table value <nil>
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 

That was when the GCCB configuration for which images to push looked like this:

images:
  - 'gcr.io/$PROJECT_ID/identity:${_GIT_TAG}'
  - 'gcr.io/$PROJECT_ID/identity:${_GIT_BRANCH}'

Switching it to this changed the message delivery so that keel could see the proper tag:

images:
  - 'gcr.io/$PROJECT_ID/identity:${_GIT_BRANCH}'
  - 'gcr.io/$PROJECT_ID/identity:${_GIT_TAG}'

I've not done a deep-enough dive to know whether this is a parsing issue in keel or whether the GCR PubSub integration doesn't contain both tags.

[rusenask]

Thanks, will investigate.

[rusenask]

closing due to age, never seen this in my environment.

[amq]

Experiencing a similar problem here. Two tags are pushed to GCR: latest and semver, and keel sees just one of them.

Update: I have found the cause for this to be the same subscription name. So, if you have GKE clusters with the same name, pulling images from GCR from a single project, you will end up with non-unique subscriptions: #323 (comment)