Incorrect exit status #19

Closed
sylvinus opened this Issue Jan 6, 2012 · 17 comments

Projects

None yet
@sylvinus
sylvinus commented Jan 6, 2012

It seems like s3cmd suffers from the same issue as :
http://code.google.com/p/s3sync-s3cmd/issues/detail?id=1

It would be great for use in script to have an exit status >0 when an upload or any other operation fails. Currently I have to parse stderr for "\nERROR:" which is a dirty hack :-/

Thanks!

srs81 commented Jan 14, 2013

+1 for non-zero return code on failed transfers. This would be really useful for automated scripts.

+1 for exit codes indicating status what happened

+1 for non-zero return code on failed transfers.

trunet commented May 13, 2013

+1 on every exception that actually exit

TJC commented May 16, 2013

+1 as well, because of the usage in scripts.

+1

Looks like the way this is implemented right now, the same function is used for PUT-ing single files and multiple files. There seems to be a decision to let individual files fail without stopping the whole operation. This might be better for first time users, but definitely not UNIX-y.

Doesn't look to hard to add add a --strict option that changes this behavior. I'll take a first crack at it that may not catch all cases.

There was a patch posted for this, over a year ago on the Debian BTS and forwarded to the S3 list.

tszming commented Jul 16, 2013

+1, currently it is barely usable for automatic backup job.

Have made some progress on revising the above patch. Currently working through the unit tests.

For those of you who want to try/comment on my branch, it's more strict than the Debian BTS patch https://github.com/ryanwitt/s3cmd/compare/strict

Having some trouble with unit tests on
OS X that are probably unrelated to the patch. I'll try to do more testing on Linux later today.

rgarcia commented Mar 11, 2014

👍 this behavior is baffling

Member
mdomsch commented May 8, 2014

https://github.com/mdomsch/s3cmd/tree/bug/exitcodes
has a stab at adding proper exit codes to s3cmd. This attempts to address bug #19, #65, and #219 from a systematic POV.

If you hate using s3cmd because is almost always returned exit code 0 regardless, please take a look at this (short) branch and comment.

Thanks,
Matt

Member
mdomsch commented May 9, 2014

pushed to master branch now. Please try and comment.

@mdomsch mdomsch closed this May 9, 2014

I just cloned and installed s3cmd. First, it appears that error messages are inconsistent.

Neither bucket test10 or test11 exists:

$ s3cmd ls s3://test10
ERROR: Access to bucket 'test10' was denied
ERROR: S3 error: 403 (AccessDenied): Access Denied
$ echo $?
70

$ s3cmd ls s3://test11
ERROR: S3 error: 403 (AllAccessDisabled): All access to this object has been disabled
$ echo $?
70

It appears that in some cases attempting to put a file to a nonexistent bucket gives a 0 return code and sometimes not.

$ s3cmd put log s3://test11
log -> s3://test11/log [1 of 1]
61440 of 512978 11% in 1s 46.02 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
log -> s3://test11/log [1 of 1]
57344 of 512978 11% in 0s 90.33 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
log -> s3://test11/log [1 of 1]
53248 of 512978 10% in 0s 78.20 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
log -> s3://test11/log [1 of 1]
32768 of 512978 6% in 1s 24.78 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
log -> s3://test11/log [1 of 1]
24576 of 512978 4% in 1s 16.81 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
log -> s3://test11/log [1 of 1]
8192 of 512978 1% in 1s 5.50 kB/s failed
ERROR: Upload of 'log' failed too many times. Skipping that file.

$ echo $?
0

However, sometimes it fails differently:

$ s3cmd put log s3://test10
log -> s3://test10/log [1 of 1]
512978 of 512978 100% in 0s 1375.94 kB/s done
ERROR: S3 error: 403 (AccessDenied): Access Denied

$ echo $?
70

And it's not some undescribed difference between test10 and test11 because just a little earlier:

$ s3cmd put /home/sexton/Test/log s3://test10
/home/sexton/Test/log -> s3://test10/log [1 of 1]
77824 of 512978 15% in 1s 40.91 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
/home/sexton/Test/log -> s3://test10/log [1 of 1]
65536 of 512978 12% in 1s 36.42 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
/home/sexton/Test/log -> s3://test10/log [1 of 1]
61440 of 512978 11% in 1s 33.32 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
/home/sexton/Test/log -> s3://test10/log [1 of 1]
49152 of 512978 9% in 1s 27.44 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
/home/sexton/Test/log -> s3://test10/log [1 of 1]
28672 of 512978 5% in 1s 15.91 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
/home/sexton/Test/log -> s3://test10/log [1 of 1]
8192 of 512978 1% in 1s 6.26 kB/s failed
ERROR: Upload of '/home/sexton/Test/log' failed too many times. Skipping that file.

$ echo $?
0

Thank you for your help.

Member
mdomsch commented May 14, 2014

There are cases not currently handled where some actions succeed while some
fail - exactly as you are describing. Right now we aren't returning a
failing exit code. That needs to be added yet.

How are you causing the Broken Pipers to occur? That actually would be a
useful test scenario if it can be done programmaticly.

On Wed, May 14, 2014 at 9:42 AM, matt-sexton notifications@github.comwrote:

I just cloned and installed s3cmd. First, it appears that error messages
are inconsistent.

Neither bucket test10 or test11 exists:

$ s3cmd ls s3://test10
ERROR: Access to bucket 'test10' was denied
ERROR: S3 error: 403 (AccessDenied): Access Denied
$ echo $?
70

$ s3cmd ls s3://test11
ERROR: S3 error: 403 (AllAccessDisabled): All access to this object has
been disabled
$ echo $?
70

It appears that in some cases attempting to put a file to a nonexistent
bucket gives a 0 return code and sometimes not.

$ s3cmd put log s3://test11
log -> s3://test11/log [1 of 1]
61440 of 512978 11% in 1s 46.02 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
log -> s3://test11/log [1 of 1]
57344 of 512978 11% in 0s 90.33 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
log -> s3://test11/log [1 of 1]
53248 of 512978 10% in 0s 78.20 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
log -> s3://test11/log [1 of 1]
32768 of 512978 6% in 1s 24.78 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
log -> s3://test11/log [1 of 1]
24576 of 512978 4% in 1s 16.81 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
log -> s3://test11/log [1 of 1]
8192 of 512978 1% in 1s 5.50 kB/s failed
ERROR: Upload of 'log' failed too many times. Skipping that file.

$ echo $?
0

However, sometimes it fails differently:

$ s3cmd put log s3://test10
log -> s3://test10/log [1 of 1]
512978 of 512978 100% in 0s 1375.94 kB/s done
ERROR: S3 error: 403 (AccessDenied): Access Denied

$ echo $?
70

And it's not some undescribed difference between test10 and test11 because
just a little earlier:

$ s3cmd put /home/sexton/Test/log s3://test10
/home/sexton/Test/log -> s3://test10/log [1 of 1]
77824 of 512978 15% in 1s 40.91 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
/home/sexton/Test/log -> s3://test10/log [1 of 1]
65536 of 512978 12% in 1s 36.42 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
/home/sexton/Test/log -> s3://test10/log [1 of 1]
61440 of 512978 11% in 1s 33.32 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
/home/sexton/Test/log -> s3://test10/log [1 of 1]
49152 of 512978 9% in 1s 27.44 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
/home/sexton/Test/log -> s3://test10/log [1 of 1]
28672 of 512978 5% in 1s 15.91 kB/s failed
WARNING: Upload failed: /log ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
/home/sexton/Test/log -> s3://test10/log [1 of 1]
8192 of 512978 1% in 1s 6.26 kB/s failed
ERROR: Upload of '/home/sexton/Test/log' failed too many times. Skipping
that file.

$ echo $?
0

Thank you for your help.


Reply to this email directly or view it on GitHubhttps://github.com/s3tools/s3cmd/issues/19#issuecomment-43089196
.

I'm not aware of doing anything special other than attempting to put a file into a nonexistent bucket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment