Make sure high level retries show with -q since the errors will have been #648

gdubicki · 2016-08-26T08:11:23Z

I have a Bash script with set -euo pipefail, so it should end immediately after non-zero exit code.

One of the commands in it is:

rclone --quiet --config="/root/myconfig" --timeout=120m0s move /mydir/myfile gcs:mybucket/subdir/myfile

..and in the logs I generated from 2 runs of this script on 2 different machines I got a similar error:

Aug 26 03:01:28 Moving file /tmp/stuff.eWne9t/myfile1_2016-07-23.tar.xz.gpg to GCS to mybucket/subdir...
2016/08/26 03:01:38 myfile1_2016-07-23.tar.xz.gpg: Failed to copy: googleapi: Error 500: Backend Error, backendError
2016/08/26 03:01:38 myfile1_2016-07-23.tar.xz.gpg: Not deleting as copy failed: googleapi: Error 500: Backend Error, backendError

...and the script continues to run, so rclone didn't return non-zero exit code, as I expected.

On the second machine I got 502 (with HTML body printed out (!) - actually I didn't expect that also, especially with --quiet flag enabled):

Aug 26 03:01:56 Moving file /tmp/stuff.mTIBiQ/myfile2_2016-06-30.tar.xz.gpg to GCS to mybucket/subdir...
2016/08/26 03:01:56 myfile2_2016-06-30.tar.xz.gpg: Failed to copy: googleapi: got HTTP response code 502 with body: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/bran
 ding/goo
 glelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>
2016/08/26 03:01:56 myfile2_2016-06-30.tar.xz.gpg: Not deleting as copy failed: googleapi: got HTTP response code 502 with body: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/bran
 ding/goo
 glelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>

...and here too the script continued to run.

What is your rclone version (eg output from rclone -V)

rclone 1.32

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS 7, 64-bits

Which cloud storage system are you using? (eg Google Drive)

GCS

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --quiet --config="/root/myconfig" --timeout=120m0s move /mydir/myfile gcs:mybucket/subdir/myfile

A log from the command with the -v flag (eg output from rclone -v copy /tmp remote:tmp)

Can't reproduce right now

The text was updated successfully, but these errors were encountered:

ncw · 2016-08-26T08:27:17Z

Firstly rclone should always return a non zero exit code if there was an error, however...

The default behaviour of rclone would be to retry the whole operation 3 times or until successful. So what I guess happened is that the first try failed but the second succeeded - hence the non-0 exit code.

... and because you are using -q you won't be seeing the "Attempt %d/%d failed with %d errors" about to retry errors in the log. You'll see these without -q.

Instead of using -q you could use --stats=0 for a little bit more log but not much and you'll get to see the retries.

Maybe the retry logs should be visible with -q since there will already have been a report of the error.

Note that with -q mode you'll still get all error level reports. It is a bit unfortunate (and unexpected) that that error message is a massive blob of html - that is Google's API doing something unexpected!

gdubicki · 2016-09-09T13:23:07Z

The default behaviour of rclone would be to retry the whole operation 3 times or until successful. So what I guess happened is that the first try failed but the second succeeded - hence the non-0 exit code.

You mean: hence the 0 exit code?

This would make sense.

And thanks for the tip about using --stats=0 instead of -q.

ncw · 2016-09-09T19:43:10Z

Glad that is sorted!

I'm going to use this issue to remind me to fix the logs with -q to shoe the retries

ncw · 2016-09-12T17:27:12Z

I've fixed this here: http://pub.rclone.org/v1.33-19-g83ba597%CE%B2/

Please re-open if you find any problems with it

ncw changed the title ~~rclone doesn't return non-zero exit code in case of unsuccessful move to GCS~~ Make sure high level retries show with -q since the errors will have been Sep 9, 2016

ncw added the bug label Sep 9, 2016

ncw added this to the v1.34 milestone Sep 9, 2016

ncw closed this as completed in 4001e21 Sep 12, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make sure high level retries show with -q since the errors will have been #648

Make sure high level retries show with -q since the errors will have been #648

gdubicki commented Aug 26, 2016

ncw commented Aug 26, 2016

gdubicki commented Sep 9, 2016

ncw commented Sep 9, 2016

ncw commented Sep 12, 2016

Make sure high level retries show with -q since the errors will have been #648

Make sure high level retries show with -q since the errors will have been #648

Comments

gdubicki commented Aug 26, 2016

ncw commented Aug 26, 2016

gdubicki commented Sep 9, 2016

ncw commented Sep 9, 2016

ncw commented Sep 12, 2016