gdrive remote strange behaviour

[RomanVeretenov]

DVC version - 0.80.0, Installed via pip
Ubuntu 18.04.2 LTS

To continue situation described in this issue:

I've created 2 remotes in .dvc/config

• local remote in local network called lremote. Set as default.
• gdrive remote called gremote. All gdrive preparations are done (app created inside my account, key & secret generated as described in 'dvc remote add' reference

Syncing with local remote works fine.
Syncing with gdrive remote behaves in some strange way:

Situation 1:
I do dvc push -r gremote, wait until all data is uploaded (it really appears in gdrive folder) and after it do git clean -dxf and then dvc pull -r gremote (also I can clone the git repo to another place, behaviour will be the same)
Expected: dvc progress bar appears and pull begins.
Got:

• I must input access key again
• After I copy the link to browser and get new access key, I get a warning
  WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: plates/***/706.zip, md5: ebd148595ce875c5754e95604d05d550
  and so on and here follows list of all files that must be pulled
  ERROR: failed to pull data from the cloud - Checkout failed for following targets:
plates/***/706.zip
  and so on
  Did you forget to fetch?

dvc fetch says the same
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:

As the result, no files are pulled

Situation 2:
I do dvc push -r gremote and then repeat it.
Expected: everything is up to date message
Got: push runs again and takes same time as on first run.

[efiop]

Hi @RomanVeretenov !

We are still actively working on gdrive, and have recently released some important changes. Please try upgrading to 0.82.2 and check if the issue persists. 🙂

[shcheklein]

@RomanVeretenov could you try to specify dvc remote modify gremote no_traverse true as a workaround for this "missing files" issue, please? I've seen the same problem with the team drives. Do you use a regular Google Drive, btw or something special?

[RomanVeretenov]

I've seen the same problem with the team drives. Do you use a regular Google Drive, btw or something special?

Seems that it's a regular personal drive (but with unlimited storage). At least I login to regular google account via standart google login page. If you know how I could check it more detailed, please let me know.

[RomanVeretenov]

So I have updated dvc to 0.82.3 and added no_traverse true

Now I'm getting following error

failed to upload '.dvc/cache/95/3c8aa52bcd20381f00f96bfd891a32' to 'gdrive://root/dvc_root/*myrepo*/95/3c8aa52bcd20381f00f96bfd891a32' - <HttpError 403 when requesting https://www.googleapis.com/drive/v2/files?q=%28%27root%27+in+parents%29+and+trashed%3Dfalse+and+title%3D%27dvc_root%27&maxResults=1&corpus=DEFAULT&supportsTeamDrives=true&includeTeamDriveItems=true&alt=json returned "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=1013921526305">

But if I follow the link, I see that quotas aren't exceeded

[shcheklein]

@RomanVeretenov could you please try the latest version (remove no_traverse setting from the .dvc/config)? I hope we've fixed the missing files issue.

@RomanVeretenov not sure what's up with rate limits. Will try to reproduce. 👀

Btw, I would also recommend to remove the files on the google drive remote you don't need anymore (from the time you was pushing them one by one). Just create a clean one and push again, or use dvc gc -c (be careful with this, read the docs). The size (number of files) on the remote storage might affect certain operations (especially with the default no_traverse value).

[RomanVeretenov]

@RomanVeretenov could you please try the latest version (remove no_traverse setting from the .dvc/config)? I hope we've fixed the missing files issue.

@RomanVeretenov not sure what's up with rate limits. Will try to reproduce. 👀

Btw, I would also recommend to remove the files on the google drive remote you don't need anymore (from the time you was pushing them one by one). Just create a clean one and push again, or use dvc gc -c (be careful with this, read the docs). The size (number of files) on the remote storage might affect certain operations (especially with the default no_traverse value).

I will try ti reproduce it on latest DVC version with new gdrive folder and without no_traverse this week and report the result

[RomanVeretenov]

So it still does not work
I have recreated the gremote destination folder in gdrive (deleted it and created an empty one)

% dvc --version
0.82.9

% dvc push -r gremote
ERROR: unexpected error - <HttpError 400 when requesting https://www.googleapis.com/drive/v2/files?q=%28%27root%27+in+parents%29+and+trashed%3Dfalse+and+title%3D%27dvc_root%27&maxResults=1&corpora=default&corpus=DEFAULT&supportsTeamDrives=true&includeTeamDriveItems=true&alt=json returned "Corpus and corpora are mutually exclusive options.">


Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

[shcheklein]

@RomanVeretenov could you please check what version of PyDrive2 is installed? (pip freeze ) ... it should the latest one - 1.4.4

[shcheklein]

I'm still not sure what is happening wit the exceeded limits. btw ... still looking into it and trying to reproduce

[RomanVeretenov]

@RomanVeretenov could you please check what version of PyDrive2 is installed? (pip freeze ) ... it should the latest one - 1.4.4

% pip freeze | grep -i pydrive
PyDrive==1.3.1
PyDrive2==1.4.0

[RomanVeretenov]

@RomanVeretenov could you please check what version of PyDrive2 is installed? (pip freeze ) ... it should the latest one - 1.4.4

% pip freeze | grep -i pydrive
PyDrive==1.3.1
PyDrive2==1.4.0

After updating PyDrive2 to 1.4.4, dvc push -r gremote has sucessfully started. But it often reports following error:

failed to upload '.dvc/cache/4a/dd94f47d5a6f3328fdda6ff6d62fc1' to 'gdrive://root/dvc_root/inex_plates/4a/dd94f47d5a6f3328fdda6ff6d62fc1' - [Errno 101] Network is unreachable

and the cache value differ always. I have limited the --jobs to 4, but it does not help.

I'm working on a remote Ubuntu pc via ssh, the pc is located somewhere in USA, but I newer faced any problems with network connection on this pc. Usual gdrive upload/download speed it there 10 MBytes/sec.