Failure signing upgrade image due to transient issue with key management server

[manoj-joseph]

http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build/job/6.0/job/patch/job/post-push/51/ failed like this:

15:21:52  + echo '{"data": "NWM1NjI0MWY4YzkxZmQwOTc3MjI3YTRmN2Y2ZDQyZmNlNDc3MWQzMTg2YzRmZjBkN2ZiMzMyZWY5MjE4M2MzNSAgLi9keF9hcHBseQo1MzgxYjVhZGNlOGJjYWZkOTk4YzM4YjBlN2YyMTg1NzliZjEyNjNmNTIwNTAxZWQzNjAwZTg1YWUzNmYxYjRiICAuL2R4X2RlbGV0ZQpiZmVjMDgwNWU1YjQ3NzkwZTQ5NGFiYzQwOTg2MGJhOTkxNzJkY2UzZmRkZjgyM2EyYWEwNmM2ZDI4MzdjY2YxICAuL2R4X2V4ZWN1dGUKYjU3MmMyMjg4ODMwM2E0Njc2ODQ5OTZkNGQ4MDBiYWIwZjkxN2NmZjFmNzFjOGVmYTVkYzRkMjAxZWUwZWQwNyAgLi9keF9wcmVwYXJlCjZhNGM4YjYzMzE3NTU0YjljMzk2ZjM4Yzk1ZmFmZjA5ZWJjM2NiZTEwNmEwZGZhODc1NzdiNGNjN2Q3NzVlODMgIC4vZHhfdXBnX3BhdXNlX29wdGlvbnMKNzkwYzJmODdlZmRjMzRkMjZlMTEwM2U5ZTUwOGNmNmJhOWE1NjJiNmUyMzg5MmNhYTc4MDhhYmY0ZGNjNzBiYSAgLi9keF91cGdfc3RyZXNzX29wdGlvbnMKOGJmNjM4Y2NlNzE0NDg5ZmViZDIxZDJmZjJiZWJmYjZjOGYzMmMzNzNkNWMzYjc5NWE2NmEzNjhlNDFiMDhkOCAgLi9keF92ZXJpZnkKNGZjOGZmZmU0NDA5MTUwMGVjNzNjNDJlYmU0MmJiZGE5ZjcxNmY5YzcyODgyYzBiNjBjYWMwMzI2OWQ5NDdhYyAgLi9ldGNfc3lzdGVtX3doaXRlbGlzdAozNWM0ODlkNzk3NmVhZWRlZTFjZDllN2Q3YmJkNGVlNjg5YjYzNzg5MWNhZWNhY2NlYTAzNWEwNDY2NDY4MzkyICAuL29zLXJvb3QuY3Bpbwo1M2Q3OWI5NWY2MTAyZjk5OTYxY2YyNzkzMGNmNzI2MWU0ODQ1MDA5NTFkYTI5NTc1OGU3NmIyNTg1MjM1ZDljICAuL29zLXJvb3QuaGFzaGVzCjQxNWUwNmRkM2EzYTc3ZDMxNTgzMzhmNmM4MGQyZWY3Y2I0MzY2MmEyOTBkMjY2MTI0YzUwYTYyY2JmMGRhNjggIC4vdmVyc2lvbi5pbmZvCg=="}'
15:21:52  + curl -s -S -f -H 'Content-Type: application/json' -u **** ****/upgrade/keyVersion/5.3/sign -d @hashes64.payload
15:21:52  curl: (22) The requested URL returned error: 500 
15:21:52  E: config/hooks/90-linux-migration-artifact.binary failed (exit non-zero). You should check for errors.
15:21:52  P: Begin unmounting filesystems...
15:21:52  P: Saving caches...
15:21:52  Reading package lists...
15:21:52  Building dependency tree...
15:21:52  Reading state information...
15:21:52  + kill -9 9500
15:21:52  + [[ ! -f binary/SHA256SUMS ]]
15:21:52  + exit 1
15:21:52  
15:21:52  > Task :live-build:buildInternalDevEsx FAILED

This looks like a transient error. The next run (run 52) passed.

[pzakha]

Another hit: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build/job/master/job/post-push/3251/

[pzakha]

Similar issue, but HTTP 504 instead of 500. Note that the job was modified in the meanwhile, so the output is a bit different:

12:33:48  + VERSION=6.0.3.0-snapshot.20200609150634333+jenkins-selfservice-devops-gate-master-appliance-build-6.0-stage-pre-push-196
12:33:48  + sed -i s/@@VERSION@@/6.0.3.0-snapshot.20200609150634333+jenkins-selfservice-devops-gate-master-appliance-build-6.0-stage-pre-push-196/ version.info
12:33:48  + cp verification-version.info.template verification-version.info
12:33:48  +++ find debs/ -name '****-upgrade-verification*.deb'
12:33:48  +++ head -n 1
12:33:48  ++ dpkg -f debs/****-upgrade-verification_1.0.2_all.deb version
12:33:48  + VERIFICATION_VERSION=1.0.2
12:33:48  + sed -i s/@@VERIFICATION_VERSION@@/1.0.2/ verification-version.info
12:33:48  + sed -i s/@@VERIFICATION_VERSION@@/1.0.2/ version.info
12:33:48  + [[ -n 5.3.0.0 ]]
12:33:48  + sed -i s/@@MINIMUM_VERSION@@/5.3.0.0/ version.info
12:33:48  + [[ -n 5.3.0.0 ]]
12:33:48  + sed -i s/@@MINIMUM_REBOOT_OPTIONAL_VERSION@@/5.3.0.0/ version.info
12:33:48  + sha256sum payload.tar.gz version.info verification-version.info prepare
12:34:01  + set +o xtrace
12:35:10  curl: (22) The requested URL returned error: 504 

http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage0/job/pre-push/415/console

[sdimitro]

Disregard the content below see Pavel's comment

=======
Hit again: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage1/job/post-push/16206/console

06:25:05  + for dir in /dev /proc /sys
06:25:05  + umount -R /mnt/****.CqlknvK/sys
06:25:05  + umount /mnt/****.CqlknvK/var/log
06:25:05  + umount /mnt/****.CqlknvK/var/****
06:25:05  + umount /mnt/****.CqlknvK/export/home
06:25:05  + umount /var/crash
06:25:05  + zfs umount rpool/ROOT/****.CqlknvK/root
06:25:05  umount: /mnt/****.CqlknvK: target is busy.
06:25:05  cannot unmount '/mnt/****.CqlknvK': umount failed
06:25:05  E: config/hooks/90-raw-disk-image.binary failed (exit non-zero). You should check for errors.
06:25:05  P: Begin unmounting filesystems...
06:25:05  P: Saving caches...
06:25:05  Reading package lists...
06:25:05  Building dependency tree...
06:25:05  Reading state information...
06:25:05  + kill -9 9604
06:25:05  + [[ ! -f binary/SHA256SUMS ]]
06:25:05  + exit 1

[pzakha]

@sdimitro This looks more like #431

[sdimitro]

03:21:13  + curl -s -S -f -H 'Content-Type: application/json' -u **** ****/upgrade/keyVersion/5.3/sign -d @hashes64.payload
03:21:13  curl: (22) The requested URL returned error: 500 
03:21:13  E: config/hooks/90-linux-migration-artifact.binary failed (exit non-zero). You should check for errors.
03:21:13  P: Begin unmounting filesystems...
03:21:13  P: Saving caches...
03:21:13  Reading package lists...
03:21:13  Building dependency tree...
03:21:13  Reading state information...
03:21:13  + kill -9 9466
03:21:13  + [[ ! -f binary/SHA256SUMS ]]
03:21:13  + exit 1

http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage1/job/post-push/17401/console

[pzakha]

15:18:50  + curl -s -S -f -H 'Content-Type: application/json' -u **** ****/upgrade/keyVersion/5.3/sign -d @hashes64.payload
15:19:48  curl: (22) The requested URL returned error: 504 
15:19:48  E: config/hooks/90-linux-migration-artifact.binary failed (exit non-zero). You should check for errors.

http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage1/job/post-push/69046/console

[pzakha]

http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage0/job/post-push/4052/console