openshift-ansible release fails if tag already exists

[adambkaplan]

Origin release with ecosystem tests fail if tag for openshift-ansible already exists:

See https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_image_registry/232/

Creating output directory: /tmp/tito
Tagging new version of openshift-ansible: 3.10.0-0.63.0 -> 3.10.1-1
Traceback (most recent call last):
  File "/usr/bin/tito", line 23, in <module>
    CLI().main(sys.argv[1:])
  File "/usr/lib/python2.7/site-packages/tito/cli.py", line 203, in main
    return module.main(argv)
  File "/usr/lib/python2.7/site-packages/tito/cli.py", line 671, in main
    return tagger.run(self.options)
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 114, in run
    self._tag_release()
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 136, in _tag_release
    self._check_tag_does_not_exist(self._get_new_tag(new_version))
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 502, in _check_tag_does_not_exist
    raise Exception("Tag %s already exists!" % new_tag)
Exception: Tag openshift-ansible-3.10.1-1 already exists!
++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace
########## FINISHED STAGE: FAILURE: BUILD AN OPENSHIFT-ANSIBLE RELEASE [00h 00m 02s] ##########

Note - the errant git tag 3.10.1-1 does not exist in the openshift-ansible repo on Github.

[adambkaplan]

@openshift/sig-developer-experience
@openshift/sig-continuous-delivery

[adambkaplan]

@openshift/sig-continuous-infrastructure

[adambkaplan]

odd...with a fresh Jenkins job the build succeeds: https://ci.openshift.redhat.com/jenkins/job/tmp_origin_tito_test/1/
However, existing jobs continue to fail.

[ingvagabund]

For some reason the tag already exists in the repo even before the github.com/openshift/openshift-ansible is pulled/synced.

Right after creating the AWS instance:

$ cd /data/src/github.com/openshift/openshift-ansible
$ git tag | grep 3.10.1-1
openshift-ansible-3.10.1-1
openshift-ansible-3.6.123.1001-1

So something is putting the tag into the repo during the "bake the base image" process.

[ingvagabund]

The job [1] already builds openshift-ansible-3.10.1 and adds the openshift-ansible-3.10.1-1 tag. @stevekuznetsov ? Do you know why?

[1] https://ci.openshift.redhat.com/jenkins/job/ami_build_origin_int_rhel_build/2885/consoleFull

[stevekuznetsov]

/assign @sdodson @smarterclayton

Is this the classic release branch mishap?

[stevekuznetsov]

In the jobs we build tag N+1 from the current branch, so this could also be a workspace caching problem:

• AMI was built from master after 3.9 and therefore created a workspace with 3.10
• master is updated to 3.10
• jobs using the AMI have a stale workspace and collide tags

We can release a new AMI this morning.

[stevekuznetsov]

Yeah, the working job is at image_stage: base and the broken job at image_stage: build. We need to publish a new build stage AMI, which should be happening on every commit to Origin. I will look into it.

/assign

[stevekuznetsov]

The job is building Origin and OpenShift Ansible but is asking for a pre-built AMI at the same time. The job should be reconfigured to use the base stage AMI instead.

/close