Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--b2-versions doesn't work for crypt backed by B2 #1627

Open
jikamens opened this issue Aug 26, 2017 · 10 comments

Comments

Projects
None yet
6 participants
@jikamens
Copy link

commented Aug 26, 2017

Hidden file versions are not listed when I do rclone ls --b2-versions on a crypt bucket that is backed by a B2 bucket. This is highly inconvenient, to say the least. :-/

@jikamens

This comment has been minimized.

Copy link
Author

commented Aug 26, 2017

Does rclone cleanup on a B2-backed crypt bucket even work? Not sure.

@ncw

This comment has been minimized.

Copy link
Owner

commented Aug 30, 2017

I'm not 100% sure how to fix this! Perhaps by getting the crypt remote to recognise the suffixes added for the versions.

Does rclone cleanup on a B2-backed crypt bucket even work? Not sure.

I'm guessing yes as the crypt remote wraps the cleanup call. If it doesn't, please make another issue!

@keithgh1

This comment has been minimized.

Copy link

commented Aug 31, 2017

Info from dupe #1632

When you have B2 "lifecycle settings" set to include versioning, and you upload a new file in the same directory, with the same filename, then the B2 server has more than one version of the file with different timestamps.

The documented behavior is here

https://www.backblaze.com/b2/docs/file_versions.html

and

https://www.backblaze.com/b2/docs/lifecycle_rules.html (describes which versions to keep etc)

Due to the filenames being encrypted locally by rclone prior to uploading, these additional versions now available on the server causes the associated filenames to be a mix of ciphertext and a plaintext date/time stamp.

When rclone attempts to decrypt this now-modified filename, the decryption fails and rclone throws an error (visible with more verbose debugging enabled) saying that the file has an "invalid filename" and skips processing this file.

Ideally, rclone could identify when a B2 encrypted remote, using filename encryption, with B2 versioning enabled server-side, sees an "invalid filename", could have an additional decryption attempt, by properly parsing the now-modified filename into two parts. The first part, the original encrypted filename, could run through the decryption process normally. The second part, the date/time stamp, would be appended onto the original plaintext filename (result of part1 decryption)

This would allow rclone filename encryption and B2 versioning to work together.

This issue is being filed per ncw. Some discussion is in this thread https://forum.rclone.org/t/backblaze-b2-humming-along-as-a-replacement-for-acd/2580/25

@ivlis

This comment has been minimized.

Copy link

commented Sep 7, 2017

That would be an extremely useful feature for b2, since it allows a versioned backup with zero cost from the user-side.

@normsland

This comment has been minimized.

Copy link

commented Sep 16, 2017

Would be great to get this implemented as b2 does not support server side move or copy. So we can't use the --backup-dir switch.

@cowwoc

This comment has been minimized.

Copy link

commented Mar 8, 2018

This feature, or lack thereof, is a very very big deal to me (the practical implication is data loss in case I want to recover an older version of my data).

Can we expect a fix in the short-term? Or should I consider drastic steps like disabling encryption?

Also, does this really affect all crypt configuratons? Or does it only affect filename encryption?

I would be okay losing filename encryption as a workaround so long as the actual contents are encrypted and I get versioning. Does that work?

@cowwoc

This comment has been minimized.

Copy link

commented Mar 8, 2018

I just tested it. It looks like versioning works fine if filename encryption is disabled.

@ncw

This comment has been minimized.

Copy link
Owner

commented Mar 11, 2018

I just tested it. It looks like versioning works fine if filename encryption is disabled.

Yes that is a reasonable work-around I think.

@keithgh1

This comment has been minimized.

Copy link

commented Mar 12, 2018

Yes, the filename encryption is exactly the problem that this bug covers. See my previous message above. I did describe a potential fix above --- whether that's the most prudent one, I have no idea.

Thanks

@cowwoc

This comment has been minimized.

Copy link

commented Jul 23, 2019

@ncw You added a note on the website saying that the recommended workaround is --backup-dir but above @normsland wrote that:

So we can't use the --backup-dir switch.

So I'm a bit confused :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.