Big syncs with millions of files

The problem

Rclone syncs on a directory by directory basis. If you have 10,000,000 directories with 1,000 files in and it will sync fine, but if you have a directory with 100,000,000 files in you will need a lot of RAM to process it.

Until the OOM killer kills the process, the log is then filled by, :

2023/07/06 15:30:35 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       1m0.0s

... although HTTP REQUEST requests are made, with HTTP RESPONSE 200 in response (--dump-headers option), no copy is made.

This problem exists until at least version rclone v1.64.0-beta.7132.f1a842081.

Workaround

We can get around the problem as follows.

• First read file or object names

rclone lsf --files-only -R src:bucket | sort > src
rclone lsf --files-only -R dst:bucket | sort > dst

• Now use comm to find what files/objects need to be transferred

comm -23 src dst > need-to-transfer
comm -13 src dst > need-to-delete

You now have a list of files you need to transfer from src to dst and another list of files in dst that aren't in src so should likely be deleted.

Then break the need-to-transfer file up into chunks of (say) 10,000 lines with something like split -l 10000 need-to-transfer and run this on each chunk to transfer 10,000 files at a time. The --files-from and the --no-traverse means that this won't list the source or the destination:

rclone copy src:bucket dst:bucket --files-from need-to-transfer --no-traverse
rclone delete src:bucket dst:bucket --files-from need-to-delete --no-traverse

If you need to sync changes, you can include hash and/or size in the listing. For example, with hashes:

rclone lsf --files-only --format "ph" -R src:bucket | sort -t';' -k1 > src
rclone lsf --files-only --format "ph" -R dst:bucket | sort -t';' -k1 > dst

The comm tool will then filter the two fields as one.