New sync method in v1.36 and Backblaze B2 Class C Transaction Cap

[rasssimo]

The new sync method in v1.36 is creating a lot of class C transactions in Backblaze B2 and using up the daily free allowance (2,500 transactions). Syncing just under 500,000 files is creating nearly 60,000 transactions - if I apply the --old-sync-method flag about 400 transactions are created.

See forum thread

[jasongill]

I am having this issue as well; on 1.35, our Class C costs were free. On 1.36, we are spending ~$20 per day in Class C transactions - more than we spend in storage! - so our bill has more than doubled.

We do have a large number of files (and a large number of directories). Is it possible to create some kind of "cache" file (upon first run of a new Rclone version), and store it in the root of the bucket - then use that cache instead of scanning each directory with every run?

Obviously this would become a problem if you were manipulating files outside of rclone, but for many enterprise users like us, we just use B2 for rclone storage only.

[jwiegley]

I agree with @jasongill: when using rclone mainly for archival purposes, any optimization the new sync method is trying to gain in terms of speed is just not a benefit, and comes at a $$ cost.

[ncw]

I'm just about to fix this, so watch this space! Use --old-sync-method in the mean time, but I've pushed my changes you'll use --fast-list.

[jasongill]

@ncw thank you so much for looking at this. We are faced with "spend thousands of dollars per month on Class C transactions" or "struggle with memory usage" so I'm very thankful that you've hopefully found a good middle ground

[jwiegley]

Yes, this is all about "choosing which resource you want to spend".

[ncw]

Please try using this beta with the --fast-list flag. It will still use more memory that without but it will definitely use less transactions.

https://beta.rclone.org/v1.36-185-g64662bef/ (uploaded in 15-30 imins)

[joshslater]

has --fast-list flag made it into a release yet? This is affecting me on B2 as well...

[joeytwiddle]

Yes --fast-list is released. It does appear to have reduced the number of C transactions made during the checking phase, and hence reduced my bill. 👍

For that reason, I suggest that --fast-list should be the default for b2. Or perhaps gentler, rclone could recommend --fast-list in the log, if the service is b2 (and the number of files is greater than 1000).

[ncw]

Yes --fast-list is released. It does appear to have reduced the number of C transactions made during the checking phase, and hence reduced my bill.

For that reason, I suggest that --fast-list should be the default for b2. Or perhaps gentler, rclone could recommend --fast-list in the log, if the service is b2 (and the number of files is greater than 1000).

There is this bit in the docs: https://rclone.org/b2/#fast-list