Option to save remote directory structure to file

[ealgase]

What is your current rclone version (output from rclone version)?

v1.44

What problem are you are trying to solve?

I have around fifty thousands files in a directory, which adds around two hundred files each day. While each file is under a megabyte and would upload very quickly, Google Drive's rate limiting makes it take a few hours. My current solution is to tar the entire directory and upload that, which is very inefficient.

How do you think rclone should be changed to solve that?

Add a flag --cache-file-structure that stores all files, dates last modified, and hashes, that are in the remote.

Example: running rclone copy Pictures Drive:Pictures --cache-file-structure=structurecache.txt would read the previously cached file structure from structurecache.txt. If any files have been modified locally, it would upload (and overwrite) to the remote (and update the data in structurecache.txt. If any files have been created, it would upload them (and add the data to structurecache.txt).

[ncw]

There are two ways of solving this...

One is to use the cache backend

The other might be to do something like this

rclone lsf -R --max-age 1d Pictures  > files.txt

to get a list of potentials (files modified within one day) then feed that into

rclone copy --files-from files.txt Pictures Drive:Pictures

that doesn't do any remote directory scanning (which has been improved further in the latest beta) so that is quite a quick way of getting incremental updates.

[ealgase]

Looks like either of those will work for me, thank you for your speedy response.