Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drive does weird things on 5xx HTTP responses when pulling #734

Closed
cdown opened this Issue Sep 14, 2016 · 9 comments

Comments

Projects
None yet
4 participants
@cdown
Copy link

cdown commented Sep 14, 2016

When trying to pull I got this:

% drive pull
Resolving...                                                                                                                                        
+ XXX.pdf
+ YYY.pdf
Addition count 2 src: 77.37MB
Proceed with the changes? [Y/n]:y
 0 / 81128125 [--------------------------------------------------------------------------------------------------------------------------------------]   0.00%
pull: /XXX.pdf err: download: failed for url "https://googledrive.com/host/<redacted>". StatusCode: 502
 81064099 / 81128125 [============================================================================================================================]  99.92% 4s
% echo $?
0
% ls -l XXX.pdf
-rw-r--r-- 1 chris chris 0 Sep 14 15:01 XXX.pdf
% file XXX.pdf
XXX.pdf: empty

Two weird things here:

  1. drive returns exit code 0 even when it got a 5xx during pull
  2. drive creates an empty file for the errored file. This seems bad since it creates the illusion that the file was successfully copied
% drive version
drive version: 0.3.7
Commit Hash: 'bd7e3f5ad5a67a27892e47da5c36452bfdfb8513'
Go Version: go1.6.2
OS: linux/amd64
BuildTime: 2016-06-11 19:52:06.74253692 +0100 BST
@dermesser

This comment has been minimized.

Copy link

dermesser commented Sep 14, 2016

Hey there, I'm working on Drive (but not here in my employer's mission). tl;dr: Drive Web Hosting, which provides the https://googledrive.com/host/... urls is being turned down. The recommended path forward is to use the API's download method.

I created #735 , and highly recommend that it is merged soon, as the official deprecation date has already passed (https://support.google.com/drive/answer/2881970?hl=en)

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Sep 14, 2016

@cdown thank you for the issue, definitely a bug.

@dermesser found the cause of it for starters #734 (comment), and made a patch to fix it #735.

Please get the latest from master.

@cdown

This comment has been minimized.

Copy link
Author

cdown commented Sep 14, 2016

This solves the "intentional 502" situation, but not the actual issue that occurs on 500s, no? Maybe I'm missing something, though.

@dermesser

This comment has been minimized.

Copy link

dermesser commented Sep 14, 2016

@cdown, you're correct. This won't solve the actual handling of errors.

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Sep 14, 2016

@cdown, yes in deed, for starters this helps mask the problem that led you to discover that errors on download aren't being handled alright. Your issue still stands, that's why I haven't yet closed it.

@odeke-em odeke-em added the bug label Sep 15, 2016

@odeke-em odeke-em added this to the v0.3.8 milestone Sep 15, 2016

odeke-em added a commit that referenced this issue Sep 15, 2016

pull: relay along errors encountered
Fixes #734.

Capture errors after parallelized pulls.
Also avoid shadowing errors on result waiting.

The changed code was legacy of an oversight after the old days
in which we didn't send back exit codes to the outside world.
After exit codes were added to drive, the pull code was never touched.
Even still though, we had logging for errors and users could see their
errors.

Exhibits:
* Unsuccessful pull
```shell
 drive pull --export pdf,rtf t1; echo $?
Resolving...
+ /test-docs/t1
Addition count 1
Proceed with the changes? [Y/n]:y
outErr: download: failed for url "". StatusCode: 400
download: failed for url "". StatusCode: 400
download: failed for url "". StatusCode: 400
download: failed for url "". StatusCode: 400
255
```

* Successful pull
```shell
$ drive pull --export pdf,rtf t1; echo $?
Resolving...
M /test-docs/t1
Modification count 1
Proceed with the changes? [Y/n]:y
Exported '/Users/emmanuelodeke/emm.odeke@gmail.com/test-docs/t1' to '/Users/emmanuelodeke/emm.odeke@gmail.com/test-docs/t1_exports/t1.rtf'
Exported '/Users/emmanuelodeke/emm.odeke@gmail.com/test-docs/t1' to '/Users/emmanuelodeke/emm.odeke@gmail.com/test-docs/t1_exports/t1.pdf'
0
```

odeke-em added a commit that referenced this issue Sep 15, 2016

pull: relay along errors encountered
Fixes #734.

Capture errors after parallelized pulls.
Also avoid shadowing errors on result waiting.

The changed code was legacy of an oversight after the old days
in which we didn't send back exit codes to the outside world.
After exit codes were added to drive, the pull code was never touched.
Even still though, we had logging for errors and users could see their
errors.

Exhibits:
* Unsuccessful pull
```shell
 drive pull --export pdf,rtf t1; echo $?
Resolving...
$ drive pull --export pdf,rtf t1; echo $?
Resolving...
M /test-docs/t1
Modification count 1
yroceed with the changes? [Y/n]: –
/test-docs/t1 err: download: failed for url "". StatusCode: 400
download: failed for url "". StatusCode: 400
255
```

* Successful pull
```shell
$ drive pull --export pdf,rtf t1; echo $?
Resolving...
M /test-docs/t1
Modification count 1
Proceed with the changes? [Y/n]:y
Exported '/Users/emmanuelodeke/emm.odeke@gmail.com/test-docs/t1' to '/Users/emmanuelodeke/emm.odeke@gmail.com/test-docs/t1_exports/t1.rtf'
Exported '/Users/emmanuelodeke/emm.odeke@gmail.com/test-docs/t1' to '/Users/emmanuelodeke/emm.odeke@gmail.com/test-docs/t1_exports/t1.pdf'
0
```
@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Sep 15, 2016

@cdown thank you for reporting this issue. I've addressed it with #737. Please get the latest from master.

Also thanks @dermesser for covering the first bases.

@yottabit42

This comment has been minimized.

Copy link

yottabit42 commented Sep 20, 2016

I just pulled from master as I had just encountered the StatusCode 502 error.

Upon restarting my pull, the client appears to be pulling all files again instead of only pulling the changes.

Clashing modification count 12 src: 113.38MB
Deletion count 8 dest: 9.03MB
Addition count 15774 src: 31.77GB

@yottabit42

This comment has been minimized.

Copy link

yottabit42 commented Sep 20, 2016

And then after that pull finished:

34215725118 / 34229412077 [======================================================================================================================================================================] 99.96% 56m32s

/Google Photos/2015/06/IMG_6786.JPG err: googleapi: got HTTP response code 503 with body: Service Unavailable

/Google Photos/2013/09/IMG_1732.JPG err: googleapi: got HTTP response code 500 with body: {
"error": { "errors": [ { "domain": "global",
"reason": "internalError", "message": "Internal Error" } ],
"code": 500, "message": "Internal Error" } }
/Google Photos/2013/09/IMG_2686.JPG err: googleapi: got HTTP response code 500 with body: {
"error": { "errors": [ { "domain": "global",
"reason": "internalError", "message": "Internal Error" } ],
"code": 500, "message": "Internal Error" } }
/Google Photos/2013/07/2013-07-31 19_09_22.jpg err: unexpected EOF
/Google Photos/2012/06/IMG_5485.JPG err: unexpected EOF
/Google Photos/2009/08/IMG_8764.jpg err: unexpected EOF
/Google Photos/2009/01/CIMG1962.JPG err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2001/09/09060114_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2001/09/09100002_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2001/09/09070020_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2001/08/08250074_G.jpg err: unexpected EOF
/Google Photos/2001/08/08240079_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2001/07/07030030_G-edited.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2001/03/03310033_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2000/12/12260001_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2000/12/12230005_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2000/10/10270002_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2000/10/10110018_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2000/08/08240010_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2000/08/08250005_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }
/Google Photos/2000/06/06270063_G.jpg err: googleapi: got HTTP response code 500 with body: {
"error": { "code": 500, "message": null } }

Exited with Errorlevel 255.

@odeke-em odeke-em referenced this issue Oct 2, 2016

Closed

StatusCode: 503 #752

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Oct 3, 2016

@yottabit42 this is kind of weird, seems like that 503 at the beginning caused it to trip more hence the 500s. Unfortunately, I don't have any insights for why we'll be getting those 500s as they are on Google Drive's backend :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.