drive: fix scan of whole file system - unworkable for large numbers of files

[Edke]

Hey.

Found rclone and it's not working at all even with single file:

$ rclone -v copy  rclone-v1.07-linux-amd64.zip  drive:RCLONE/
2015/02/03 20:21:23 Google drive root 'RCLONE': Modify window is 1ms
2015/02/03 20:21:23 Google drive root 'RCLONE': Building file list
2015/02/03 20:22:23 
Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:  1m6.178419486s

2015/02/03 20:23:23 
Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0 
Checks:                 0
Transferred:            0
Elapsed time:  2m6.17841706s

2015/02/03 20:24:23 
Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:  3m6.17849835s

2015/02/03 20:25:23 
Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:  4m6.17840109s

Any thoughts ?

[Edke]

Tested it a bit further, it works against my personal drive account, but not against my google apps drive.

[ncw]

Hmm, interesting! I don't have a google apps drive account to test it with unfortunately, so it isn't tested with them at all.

Can you do some other operations like ls, lsl, lsd with the -v flag to see if it prints any interesting debug?

Is there something special about authenticating with a google apps account?

Thanks

Nick

[Edke]

I can create one account for one month for you if that would help.

[zeshanb]

Hello Eduard,

Please can you also try with a local source folder configuration by using
config wizard.(rclone config) http://rclone.org/drive/

For local source set the path to location of the file you are trying to
transfer. If the file is located /home/username/ then set the path to
/home/username/ in When you run rclone config. Set the name for local
source as local. This will allow you to use local: when transferring files.

Once configured try this:

rclone -v copy local:rclone-v1.07-linux-amd64.zip drive:RCLONE

You don't need a ending slash after drive:RCLONE

Zeshan Bilal

On Wed, Feb 4, 2015 at 5:43 AM, Eduard Kracmar notifications@github.com
wrote:

I can create one account for one month for you if that would help.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[ncw]

@Edke I think I may have figured out the problem. Do you by any chance have a great deal of files or directories in your google drive account?

If so this may fix it

rclone --drive-full-list=false -v copy  rclone-v1.07-linux-amd64.zip  drive:RCLONE

I should probably make that the default - I can't remember the rationale now so I'll have to re-investigate

[Edke]

@ncw that's it, it works like it should now, great!

@zeshanb didn't test it, ncw's solution works

[ncw]

@Edke thanks for testing that - I'll try to make that flag not necessary for you in the next release.

[zeshanb]

@Edke that's super cool. I also got to learn a new option in the process.

On Wed, Feb 4, 2015 at 11:40 AM, Nick Craig-Wood notifications@github.com
wrote:

@Edke https://github.com/Edke thanks for testing that - I'll try to
make that flag not necessary for you in the next release.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[Edke]

Finally I have an option how to sync against Google Drive and move 100GB folder structure there and back again. Triied all current solutions, gsync, insync, fuse etc, none of them worked in my scenario. This is awesome, it just works!

Transferred:   2686751451 Bytes (1214.10 kByte/s)
Errors:                58
Checks:                 0
Transferred:         2466
Elapsed time:  36m1.09202875s
Transferring:  photos/2008/08/28/img_6855.jpg, photos/2008/08/28/img_6858.jpg, photos/2008/08/28/img_6857.jpg, photos/2008/08/28/img_6856.jpg

[ncw]

Excellent! There is a problem with drive and 403 errors (see #20) which I'm working on at the moment too. I can see you got 58 errors above which were probably caused by it - if you try the sync/copy again it should transfer the missing files very quickly.

[Edke]

I didnt finish yet :)))

Transferred:   20204142538 Bytes (1980.77 kByte/s)
Errors:               126
Checks:                 0
Transferred:        10879
Elapsed time:  2h46m1.091942083s
Transferring:  photos/2011/05/14/img_9806.jpg, photos/2011/05/14/img_9805.jpg, photos/2011/05/14/img_9802.jpg, photos/2011/05/14/img_9807.jpg

[zeshanb]

Eduard, How do you let the process continue to run in the background? Do
you use screen or something else?

On Wed, Feb 4, 2015 at 2:32 PM, Eduard Kracmar notifications@github.com
wrote:

I didnt finish yet :)))

Transferred: 20204142538 Bytes (1980.77 kByte/s)
Errors: 126
Checks: 0
Transferred: 10879
Elapsed time: 2h46m1.091942083s
Transferring: photos/2011/05/14/img_9806.jpg, photos/2011/05/14/img_9805.jpg, photos/2011/05/14/img_9802.jpg, photos/2011/05/14/img_9807.jpg

—
Reply to this email directly or view it on GitHub
#23 (comment).

[Edke]

@zeshanb screen is great option of course

[ncw]

I've fixed this and made a new release v1.08.

If this doesn't work, please re-open the ticket.

Thank you very much for your bug report.

-- Nick

[ChrisSLT]

Reporting back per your request. Now using v 1.09. I tested this with a "check" of a local file system against google drive. Total size about 2GB. It worked great first time.

[ncw]

Great, thanks for testing it for me :-)