Hanging when copying artifacts to S3

[pzakha]

The aws s3 sync operation seems to have hanged, resulting in a timeout of the job:

16:55:03 [artifacts] Running shell script
16:55:04 + aws s3 sync --delete --only-show-errors . s3://snapshot-de-images/builds/jenkins-ops/devops-gate/master/appliance-build/master/post-push/319
00:31:36 Sending interrupt signal to process
00:31:36 Cancelling nested steps due to timeout

This may be somewhat related to issue #107.

[pzakha]

Another instance here: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage1/job/master/job/post-push/3158/console

I was wondering if using cp --recursive might have been more reliable, since sync has a bunch of extra logic in it, however I see tickets open for hangs in both cases: aws/aws-cli#2477, aws/aws-cli#951.

We had added some debugging in the past with this devops-gate commit:

commit 3e581572f3d78a0e614a9bf4645e62fb4f814e86
Author: Prakash Surya <prakash.surya@delphix.com>
Date:   Mon Sep 17 09:36:09 2018 -0700

    Use "--debug" with "aws s3 sync" command

However we had to remove the debugging as it flooded the Jenkins logs with several hundreds of megabytes of debug logs. Perhaps we could only print the lines with the ERROR tag.

[shartse]

Hit again here: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage1/job/master/job/post-push/12145/consoleFull

23:43:17  + aws s3 sync --delete --only-show-errors . s3://snapshot-de-images/builds/jenkins-ops/devops-gate/master/appliance-build/master/nightly/171/internal-qa-aws/
03:00:16  Calling Pipeline was cancelled
03:00:16  Sending interrupt signal to process
03:00:16  Calling Pipeline was cancelled
03:00:20  Sending interrupt signal to process
03:00:21  /var/tmp/jenkins/workspace/devops-gate/master/appliance-build-stage1/master/post-push/appliance-build/live-build/build/artifacts@tmp/durable-8252f486/script.sh: line 1: 10461 Terminated              aws s3 sync --delete --only-show-errors . s3://snapshot-de-images/builds/jenkins-ops/devops-gate/master/appliance-build/master/nightly/171/internal-qa-aws/
03:00:21  script returned exit code 143

[prashks]

Another occurance: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build/job/master/job/post-push/1958/consoleFull

[jwk404]

Hit again: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage1/job/master/job/post-push/25748/console

14:01:12  + aws s3 sync --delete --only-show-errors . s3://snapshot-de-images/builds/jenkins-ops/devops-gate/master/appliance-build/master/post-push/2205/internal-qa-azure/
17:23:18  Calling Pipeline was cancelled
17:23:18  Sending interrupt signal to process
17:23:19  Calling Pipeline was cancelled
17:23:23  Sending interrupt signal to process
17:23:33  /var/tmp/jenkins/workspace/devops-gate/master/appliance-build-stage1/master/post-push/appliance-build/live-build/build/artifacts@tmp/durable-ccef5ee5/script.sh: line 1:  5628 Terminated              aws s3 sync --delete --only-show-errors . s3://snapshot-de-images/builds/jenkins-ops/devops-gate/master/appliance-build/master/post-push/2205/internal-qa-azure/
17:23:33  script returned exit code 143

[pzakha]

Another hit: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage1/job/master/job/post-push/27509/console

[pzakha]

Another hit: #430

[pzakha]

http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build/job/6.0/job/release/job/post-push/205/console

[pzakha]

http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage1/job/post-push/10368/console

[pzakha]

http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage1/job/post-push/69130/console

00:02:25  + aws s3 sync --delete --only-show-errors . s3://snapshot-de-images/builds/jenkins-ops/devops-gate/master/appliance-build/master/post-push/4639/internal-dev-azure/
03:05:08  Calling Pipeline was cancelled
03:05:08  Sending interrupt signal to process
03:05:08  Calling Pipeline was cancelled
03:05:11  Sending interrupt signal to process
03:05:12  /var/tmp/jenkins/workspace/devops-gate/master/appliance-build-stage1/post-push/appliance-build/live-build/build/artifacts@tmp/durable-4db4c2b4/script.sh: line 1:  1822 Terminated              aws s3 sync --delete --only-show-errors . s3://snapshot-de-images/builds/jenkins-ops/devops-gate/master/appliance-build/master/post-push/4639/internal-dev-azure/
03:05:12  script returned exit code 143