FAILURE: BUILD AN OPENSHIFT-ANSIBLE RELEASE #14122

sdodson · 2017-05-10T00:36:25Z

Seen in https://ci.openshift.redhat.com/jenkins/job/test_pull_request_openshift_ansible_extended_conformance_install_with_status_check/260/consoleFull#-157296560358b6e51eb7608a5981914356

I think this happened because the tag is on the stage branch but this PR is against master?

########## STARTING STAGE: BUILD AN OPENSHIFT-ANSIBLE RELEASE ##########
+ [[ -s /var/lib/jenkins/jobs/test_pull_request_openshift_ansible_extended_conformance_install_with_status_check/workspace/activate ]]
+ source /var/lib/jenkins/jobs/test_pull_request_openshift_ansible_extended_conformance_install_with_status_check/workspace/activate
++ export VIRTUAL_ENV=/var/lib/jenkins/origin-ci-tool/b58d70e4ba8547fb716da04deb467ce17ccf4345
++ VIRTUAL_ENV=/var/lib/jenkins/origin-ci-tool/b58d70e4ba8547fb716da04deb467ce17ccf4345
++ export PATH=/var/lib/jenkins/origin-ci-tool/b58d70e4ba8547fb716da04deb467ce17ccf4345/bin:/sbin:/usr/sbin:/bin:/usr/bin
++ PATH=/var/lib/jenkins/origin-ci-tool/b58d70e4ba8547fb716da04deb467ce17ccf4345/bin:/sbin:/usr/sbin:/bin:/usr/bin
++ unset PYTHON_HOME
++ export OCT_CONFIG_HOME=/var/lib/jenkins/jobs/test_pull_request_openshift_ansible_extended_conformance_install_with_status_check/workspace/.config
++ OCT_CONFIG_HOME=/var/lib/jenkins/jobs/test_pull_request_openshift_ansible_extended_conformance_install_with_status_check/workspace/.config
++ mktemp
+ script=/tmp/tmp.SXCuZksfW9
+ cat
+ chmod +x /tmp/tmp.SXCuZksfW9
+ scp -F ./.config/origin-ci-tool/inventory/.ssh_config /tmp/tmp.SXCuZksfW9 openshiftdevel:/tmp/tmp.SXCuZksfW9
+ ssh -F ./.config/origin-ci-tool/inventory/.ssh_config -t openshiftdevel 'bash -l -c "/tmp/tmp.SXCuZksfW9"'
+ cd /data/src/github.com/openshift/openshift-ansible
+ tito_tmp_dir=tito
+ mkdir -p tito
+ tito tag --offline --accept-auto-changelog
Creating output directory: /tmp/tito
Tagging new version of openshift-ansible: 3.6.62-1 -> 3.6.63-1
Traceback (most recent call last):
  File "/usr/bin/tito", line 23, in <module>
    CLI().main(sys.argv[1:])
  File "/usr/lib/python2.7/site-packages/tito/cli.py", line 202, in main
    return module.main(argv)
  File "/usr/lib/python2.7/site-packages/tito/cli.py", line 666, in main
    return tagger.run(self.options)
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 114, in run
    self._tag_release()
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 136, in _tag_release
    self._check_tag_does_not_exist(self._get_new_tag(new_version))
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 501, in _check_tag_does_not_exist
    raise Exception("Tag %s already exists!" % new_tag)
Exception: Tag openshift-ansible-3.6.63-1 already exists!
++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace
########## FINISHED STAGE: FAILURE: BUILD AN OPENSHIFT-ANSIBLE RELEASE ##########

The text was updated successfully, but these errors were encountered:

sdodson · 2017-05-10T12:50:47Z

I think I've fixed this by tagging stage 3.6.65.0 and master 3.6.66

stevekuznetsov · 2017-05-10T13:22:33Z

Yeah, we need to be using that four-number workaround until we have a better solution, but this isn't necessarily a tooling or testing issue -- we just need to have a method for making unique tags.

sdodson · 2017-05-10T18:15:55Z

Well, we should use tools to handle the stage branching, so just making sure you guys are aware of that and we add .0 to the current version while branching and tagging stage.

stevekuznetsov · 2017-05-10T18:18:30Z

Is this problem solved? Are the people reporting this above doing so from old failures? Do we need to take action right now?

sdodson · 2017-05-10T18:25:08Z

Those are old failures. I fixed it at

commit 5bd4ca446711fc7faa53ad0d6d3f2183857938d1
Author: Scott Dodson <sdodson@redhat.com>
Date:   Wed May 10 08:50:14 2017 -0400

    Automatic commit of package [openshift-ansible] release [3.6.66-1].
    
    Created by command:
    
    /usr/bin/tito tag --keep-version

tnozicka · 2017-05-31T11:08:58Z

seems like it's still there: https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_install_update/713/console

mfojtik · 2017-05-31T11:09:40Z

Note that this happened exactly when the new tag was cut for OSE and the versions in error matches, so maybe just bad timing.

tnozicka · 2017-05-31T12:59:25Z

#14322 hit it again - https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/853/

This seem to be serious :)

sdodson · 2017-05-31T13:13:04Z

On stage branch set rpm specfile version to 3.6.88.0, tag and pushed to github.
On master branch set rpm specfile version to 3.6.89, tag and pushed to github.

We need to fix the stage branch tooling to append a fourth version segment when forking that branch.

Looks like @jupierce built the jobs for that, assigning to him.

jupierce · 2017-05-31T14:38:34Z

@sdodson I think the only patch that will hold up through stagecut is adding a .0 to the openshift-ansible.spec in master. A change in the stage branch will be wiped out every time we build stage (since the o-a spec version is always replaced by the ose spec version). When stagecut ends and we return to master, the .0 will similarly be wiped out.
Discussing longer term solutions with @stevekuznetsov to determine the best route.

jupierce · 2017-06-16T20:07:31Z

Should be fixed with openshift-eng/aos-cd-jobs@ca7f5b2

enj · 2017-06-24T17:47:16Z

Seen in #14853 https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/1111/

########## FINISHED STAGE: SUCCESS: DETERMINE THE RELEASE COMMIT FOR ORIGIN IMAGES [00h 00m 14s] ##########
[workspace] $ /bin/bash /tmp/hudson1315612192578537421.sh
########## STARTING STAGE: BUILD AN OPENSHIFT-ANSIBLE RELEASE ##########
+ [[ -s /var/lib/jenkins/jobs/test_pull_request_origin_extended_conformance_install_update/workspace/activate ]]
+ source /var/lib/jenkins/jobs/test_pull_request_origin_extended_conformance_install_update/workspace/activate
++ export VIRTUAL_ENV=/var/lib/jenkins/origin-ci-tool/9aea3b4f81e266b026e21975a3a6a5a1cfddd890
++ VIRTUAL_ENV=/var/lib/jenkins/origin-ci-tool/9aea3b4f81e266b026e21975a3a6a5a1cfddd890
++ export PATH=/var/lib/jenkins/origin-ci-tool/9aea3b4f81e266b026e21975a3a6a5a1cfddd890/bin:/sbin:/usr/sbin:/bin:/usr/bin
++ PATH=/var/lib/jenkins/origin-ci-tool/9aea3b4f81e266b026e21975a3a6a5a1cfddd890/bin:/sbin:/usr/sbin:/bin:/usr/bin
++ unset PYTHON_HOME
++ export OCT_CONFIG_HOME=/var/lib/jenkins/jobs/test_pull_request_origin_extended_conformance_install_update/workspace/.config
++ OCT_CONFIG_HOME=/var/lib/jenkins/jobs/test_pull_request_origin_extended_conformance_install_update/workspace/.config
++ mktemp
+ script=/tmp/tmp.4YCB5OYOuv
+ cat
+ chmod +x /tmp/tmp.4YCB5OYOuv
+ scp -F ./.config/origin-ci-tool/inventory/.ssh_config /tmp/tmp.4YCB5OYOuv openshiftdevel:/tmp/tmp.4YCB5OYOuv
+ ssh -F ./.config/origin-ci-tool/inventory/.ssh_config -t openshiftdevel 'bash -l -c "/tmp/tmp.4YCB5OYOuv"'
+ cd /data/src/github.com/openshift/openshift-ansible
+ tito_tmp_dir=tito
+ mkdir -p tito
+ tito tag --offline --accept-auto-changelog
Creating output directory: /tmp/tito
Tagging new version of openshift-ansible: 3.6.123-1 -> 3.6.124-1
Traceback (most recent call last):
  File "/usr/bin/tito", line 23, in <module>
    CLI().main(sys.argv[1:])
  File "/usr/lib/python2.7/site-packages/tito/cli.py", line 202, in main
    return module.main(argv)
  File "/usr/lib/python2.7/site-packages/tito/cli.py", line 666, in main
    return tagger.run(self.options)
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 114, in run
    self._tag_release()
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 136, in _tag_release
    self._check_tag_does_not_exist(self._get_new_tag(new_version))
  File "/usr/lib/python2.7/site-packages/tito/tagger/main.py", line 501, in _check_tag_does_not_exist
    raise Exception("Tag %s already exists!" % new_tag)
Exception: Tag openshift-ansible-3.6.124-1 already exists!
++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace
########## FINISHED STAGE: FAILURE: BUILD AN OPENSHIFT-ANSIBLE RELEASE [00h 00m 02s] ##########
Build step 'Execute shell' marked build as failure
[PostBuildScript] - Execution post build scripts.
[workspace] $ /bin/bash /tmp/hudson2579696756842582650.sh

jupierce · 2017-06-24T19:26:25Z

Fixed again. The present workaround relies on careful management of Jenkins job scheduling. Working on a more robust solution.

jupierce · 2017-06-24T20:16:32Z

@stevekuznetsov I think I asked this before during our discussion, but I'll lay it out there again. Addressing this completely outside of CI is tricky. I think a simple and effective approach would be for CI to sed append .1000 to any version it found in openshift-ansible.spec before doing tito tag. This is effectively what has to happen outside of CI to avoid this problem, but performing it outside of CI involves git commits and adds complexity to entering and exiting stagecut.

stevekuznetsov · 2017-06-24T22:12:47Z

Not the cleanest thing in the world but we can try that on Monday.

jupierce · 2017-06-28T10:28:24Z

We should now have a reliable fix in place:

master branches will build with three fields (3.6.X)
stage branches will build with four fields (3.6.X.Y). openshift/openshift-ansible@f9c47bf
enterprise branches will build will five fields (3.6.X.Y.Z)

sdodson added the kind/test-flake Categorizes issue or PR as related to test flakes. label May 10, 2017

sdodson assigned stevekuznetsov May 10, 2017

sdodson mentioned this issue May 10, 2017

Remove vim configuration from Python files openshift/openshift-ansible#4123

Merged

mtnbikenc mentioned this issue May 10, 2017

Properly fail on empty repoquery version checks openshift/openshift-ansible#4097

Merged

legionus mentioned this issue May 10, 2017

Force to specify not empty secret for metrics endpoint #13884

Merged

sdodson mentioned this issue May 10, 2017

byo: correct option name openshift/openshift-ansible#4130

Merged

stevekuznetsov mentioned this issue May 10, 2017

Use shared informers for secrets, services, and service accounts #14086

Merged

rootfs mentioned this issue May 10, 2017

backport upstream Cinder fixes #14005

Merged

soltysh mentioned this issue May 10, 2017

Fix godeps #14121

Merged

sdodson mentioned this issue May 10, 2017

polish etcd_common role openshift/openshift-ansible#4124

Merged

bparees mentioned this issue May 10, 2017

strip proxy credentials from proxy env variables when logging them #13751

Merged

sttts mentioned this issue May 10, 2017

Fix default watch cache size to be 1000 instead of upstream's 100 #14052

Merged

miminar mentioned this issue May 10, 2017

Prefer secure connection during image pruning #14114

Merged

juanvallejo mentioned this issue May 10, 2017

Add --local & --dry-run flags to oc set sub-commands #14123

Merged

10 tasks

pweil- added component/internal-tools priority/P1 labels May 10, 2017

tnozicka mentioned this issue May 31, 2017

Set controllerRef in RCs owned by DC #14322

Merged

2 tasks

mfojtik mentioned this issue May 31, 2017

Refactor controller initialization (part 2) #14293

Merged

tnozicka added priority/P0 and removed priority/P1 labels May 31, 2017

sdodson unassigned stevekuznetsov May 31, 2017

sdodson assigned jupierce May 31, 2017

pravisankar mentioned this issue May 31, 2017

Bugs 1439142, 1417641 - network diagnostics fixes #14364

Merged

sttts mentioned this issue Jun 14, 2017

UPSTREAM: 46112: apimachinery: move unversioned registration to metav1 #14593

Merged

jupierce closed this as completed Jun 16, 2017

enj reopened this Jun 24, 2017

enj mentioned this issue Jun 24, 2017

Fix ClusterNetwork REST GetAttrs #14853

Merged

stevekuznetsov added priority/P1 and removed priority/P0 labels Jun 24, 2017

jupierce closed this as completed Jun 28, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FAILURE: BUILD AN OPENSHIFT-ANSIBLE RELEASE #14122

FAILURE: BUILD AN OPENSHIFT-ANSIBLE RELEASE #14122

sdodson commented May 10, 2017

sdodson commented May 10, 2017

stevekuznetsov commented May 10, 2017

sdodson commented May 10, 2017

stevekuznetsov commented May 10, 2017

sdodson commented May 10, 2017

tnozicka commented May 31, 2017

mfojtik commented May 31, 2017

tnozicka commented May 31, 2017

sdodson commented May 31, 2017

jupierce commented May 31, 2017

jupierce commented Jun 16, 2017

enj commented Jun 24, 2017

jupierce commented Jun 24, 2017

jupierce commented Jun 24, 2017

stevekuznetsov commented Jun 24, 2017

jupierce commented Jun 28, 2017

FAILURE: BUILD AN OPENSHIFT-ANSIBLE RELEASE #14122

FAILURE: BUILD AN OPENSHIFT-ANSIBLE RELEASE #14122

Comments

sdodson commented May 10, 2017

sdodson commented May 10, 2017

stevekuznetsov commented May 10, 2017

sdodson commented May 10, 2017

stevekuznetsov commented May 10, 2017

sdodson commented May 10, 2017

tnozicka commented May 31, 2017

mfojtik commented May 31, 2017

tnozicka commented May 31, 2017

sdodson commented May 31, 2017

jupierce commented May 31, 2017

jupierce commented Jun 16, 2017

enj commented Jun 24, 2017

jupierce commented Jun 24, 2017

jupierce commented Jun 24, 2017

stevekuznetsov commented Jun 24, 2017

jupierce commented Jun 28, 2017