Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--b2-versions doesn't work for crypt backed by B2 #1627

Closed
jikamens opened this issue Aug 26, 2017 · 14 comments
Closed

--b2-versions doesn't work for crypt backed by B2 #1627

jikamens opened this issue Aug 26, 2017 · 14 comments

Comments

@jikamens
Copy link

@jikamens jikamens commented Aug 26, 2017

Hidden file versions are not listed when I do rclone ls --b2-versions on a crypt bucket that is backed by a B2 bucket. This is highly inconvenient, to say the least. :-/

@jikamens
Copy link
Author

@jikamens jikamens commented Aug 26, 2017

Does rclone cleanup on a B2-backed crypt bucket even work? Not sure.

Loading

@ncw
Copy link
Member

@ncw ncw commented Aug 30, 2017

I'm not 100% sure how to fix this! Perhaps by getting the crypt remote to recognise the suffixes added for the versions.

Does rclone cleanup on a B2-backed crypt bucket even work? Not sure.

I'm guessing yes as the crypt remote wraps the cleanup call. If it doesn't, please make another issue!

Loading

@keithgh1
Copy link

@keithgh1 keithgh1 commented Aug 31, 2017

Info from dupe #1632

When you have B2 "lifecycle settings" set to include versioning, and you upload a new file in the same directory, with the same filename, then the B2 server has more than one version of the file with different timestamps.

The documented behavior is here

https://www.backblaze.com/b2/docs/file_versions.html

and

https://www.backblaze.com/b2/docs/lifecycle_rules.html (describes which versions to keep etc)

Due to the filenames being encrypted locally by rclone prior to uploading, these additional versions now available on the server causes the associated filenames to be a mix of ciphertext and a plaintext date/time stamp.

When rclone attempts to decrypt this now-modified filename, the decryption fails and rclone throws an error (visible with more verbose debugging enabled) saying that the file has an "invalid filename" and skips processing this file.

Ideally, rclone could identify when a B2 encrypted remote, using filename encryption, with B2 versioning enabled server-side, sees an "invalid filename", could have an additional decryption attempt, by properly parsing the now-modified filename into two parts. The first part, the original encrypted filename, could run through the decryption process normally. The second part, the date/time stamp, would be appended onto the original plaintext filename (result of part1 decryption)

This would allow rclone filename encryption and B2 versioning to work together.

This issue is being filed per ncw. Some discussion is in this thread https://forum.rclone.org/t/backblaze-b2-humming-along-as-a-replacement-for-acd/2580/25

Loading

@ivlis
Copy link

@ivlis ivlis commented Sep 7, 2017

That would be an extremely useful feature for b2, since it allows a versioned backup with zero cost from the user-side.

Loading

@normsland
Copy link

@normsland normsland commented Sep 16, 2017

Would be great to get this implemented as b2 does not support server side move or copy. So we can't use the --backup-dir switch.

Loading

@cowwoc
Copy link

@cowwoc cowwoc commented Mar 8, 2018

This feature, or lack thereof, is a very very big deal to me (the practical implication is data loss in case I want to recover an older version of my data).

Can we expect a fix in the short-term? Or should I consider drastic steps like disabling encryption?

Also, does this really affect all crypt configuratons? Or does it only affect filename encryption?

I would be okay losing filename encryption as a workaround so long as the actual contents are encrypted and I get versioning. Does that work?

Loading

@cowwoc
Copy link

@cowwoc cowwoc commented Mar 8, 2018

I just tested it. It looks like versioning works fine if filename encryption is disabled.

Loading

@ncw
Copy link
Member

@ncw ncw commented Mar 11, 2018

I just tested it. It looks like versioning works fine if filename encryption is disabled.

Yes that is a reasonable work-around I think.

Loading

@keithgh1
Copy link

@keithgh1 keithgh1 commented Mar 12, 2018

Yes, the filename encryption is exactly the problem that this bug covers. See my previous message above. I did describe a potential fix above --- whether that's the most prudent one, I have no idea.

Thanks

Loading

@cowwoc
Copy link

@cowwoc cowwoc commented Jul 23, 2019

@ncw You added a note on the website saying that the recommended workaround is --backup-dir but above @normsland wrote that:

So we can't use the --backup-dir switch.

So I'm a bit confused :)

Loading

@ncw
Copy link
Member

@ncw ncw commented Jul 30, 2019

@ncw You added a note on the website saying that the recommended workaround is --backup-dir but above @normsland wrote that:

So we can't use the --backup-dir switch.

So I'm a bit confused :)

b2 does support --backup-dir now from rclone v1.48 since b2 added the capability for server side copies.

Loading

@jikamens

This comment has been minimized.

wlritchi added a commit to wlritchi/rclone that referenced this issue Mar 2, 2020
Relies on identifying the version string that backend/b2 adds to the
filename. There are pathological cases where an obfuscated filename
could end in what looks like a version string purely by chance. In such
a case, this code will incorrectly leave that portion of the filename
obfuscated. There's not really a clean fix for that, other than keeping
track of versions out-of-band instead of in the filename string.

Fixes rclone#1627
wlritchi added a commit to wlritchi/rclone that referenced this issue Mar 2, 2020
Relies on identifying the version string that backend/b2 adds to the
filename. There are pathological cases where an obfuscated filename
could end in what looks like a version string purely by chance. In such
a case, this code will incorrectly leave that portion of the filename
obfuscated. There's not really a clean fix for that, other than keeping
track of versions out-of-band instead of in the filename string.

Fixes rclone#1627
wlritchi added a commit to wlritchi/rclone that referenced this issue Mar 2, 2020
Relies on identifying the version string that backend/b2 adds to the
filename. There are pathological cases where an obfuscated filename
could end in what looks like a version string purely by chance. In such
a case, this code will incorrectly leave that portion of the filename
obfuscated. There's not really a clean fix for that, other than keeping
track of versions out-of-band instead of in the filename string.

Fixes rclone#1627
wlritchi added a commit to wlritchi/rclone that referenced this issue Mar 2, 2020
Relies on identifying the version string that backend/b2 adds to the
filename. There are pathological cases where an obfuscated filename
could end in what looks like a version string purely by chance. In such
a case, this code will incorrectly leave that portion of the filename
obfuscated. There's not really a clean fix for that, other than keeping
track of versions out-of-band instead of in the filename string.

Fixes rclone#1627
@steakhutzeee
Copy link

@steakhutzeee steakhutzeee commented Dec 19, 2020

Hi, i have a couple of questions:

rclone cleanup command will not work with --backup-dir, cleaning it, right?
I suppose it only works with "original" b2 old versions of files.

And also, if a bucket is only destined to work with crypt, at this time it's useless to use the "lifecycle settings" for that bucket because using crypt they are usless and i need to use --backup-dir instead.

Correct me if i am wrong.

Thanks in advance!

Loading

@ivandeex
Copy link
Member

@ivandeex ivandeex commented Mar 16, 2021

Unfortunately the patch #4024, which could resolve this, was abandoned by maintainers :-(

Loading

ncw added a commit that referenced this issue Apr 12, 2021
With the file version format standardized in lib/version, `crypt` can
now treat the version strings separately from the encrypted/decrypted
file names. This allows --b2-versions to work with `crypt`.

Fixes #1627

Co-authored-by: Luc Ritchie <luc.ritchie@gmail.com>
negative0 added a commit to negative0/rclone that referenced this issue Aug 13, 2021
With the file version format standardized in lib/version, `crypt` can
now treat the version strings separately from the encrypted/decrypted
file names. This allows --b2-versions to work with `crypt`.

Fixes rclone#1627

Co-authored-by: Luc Ritchie <luc.ritchie@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants