make gsutil rsync recursive #297
Conversation
|
Unsure of why that failed. Is there anything I can do further to help with this? |
|
Thanks @KBodolai. Rebasing on to the latest |
|
ok, I've done that, hopefully it'll be it! |
|
ok, so these passed (yay!) There's another parameter that can be fed to gsutil for threaded processing (-m), which provides huge performance boosts, although they mention in some cases it may slow the syncing down. I believe it will typically only result in speedups for the setting this is used in, but not quite certain, what do you think? |
|
@KBodolai I agree that 99% of the use cases here will be cloud <-> local, not local <-> local so |
|
right, I'll add it and set up a test project to double check it does well, I'll ping you when it's ready :) |
|
@pjbull , actually, I stayed a bit late after work and tested if with a bunch of heavy lidar files, all working fine! |
* make gsutil rsync recursive * reformatted with black * add threaded flag for gsutil rsync
Small change to allow gsutil to do rsync recursively, syncing all the folders.
Context: We had some issues using it with gcloud, since
make sync_data_upwould not upload all the data folders. Adding the -r flag fixes this issue.