Database reset with encrypted folders causes a full resync

[maximilize]

What happened?

1. Sync a folder with Receive Encrypted
2. Wait until it's fully synced
3. On the encrypted side, run syncthing --reset-database
4. Restart syncthing

Syncthing will detect all already synced data as "Locally changed" and will do a full resync of all data.
After this, storage consumption is 2x. This can be fixed with "Revert local changes".

Aside from this it happens that there are still "Locally changed items", even after pressing on "Revert local changes". Pause and unpause the folder sometimes helps.

Syncthing version

v1.27.4

Platform & operating system

Linux arm64

[calmh]

I was with you up to

After this, storage consumption is 2x

how did that happen? AFAIK the file names should be the same after the resync.

[maximilize]

Well, I'm not joking.

Global State 	 789   120   ~925 MiB
 Local State  1,578   120   ~1.81 GiB
 Locally Changed Items 	[789 items, ~927 MiB]

Size on disk: 1.8 GB. Before DB reset it was 920 MB.

After the DB reset, syncthing made a full resync.

After clicking on "Revert local changes":
Locally Changed Items [789 items, ~0 B]

After pause and then unpause the folder, "Locally changed items" disappears.

Edit:
Is it possible that there is a local salt or so which get's reset together with the DB?

[calmh]

Probably conflict copies, now that I think about it, which I think is also expected.

[maximilize]

Possible, I'll check that. But why are the files conflicting? No key or file was changed.

Additionally, the bug with not resetting the counter for locally changed files does not reset after clicking on revert. Just the file size is changed to 0. After pause and unpause, this is fixed. Possible related to PR #9319?!

Edit: Should be file conflicts a thing with "Receive encrypted" folders at all?

[maximilize]

I understand now that encrypted folders are never hashed (see https://github.com/syncthing/syncthing/blob/main/lib/model/folder.go#L664).

So syncthing is always assuming that the encrypted side is outdated when the index DB is reset/broken/corrupted. Index DB corruption happened to me a few times already.

Downsides:

• A full resync is required (problematic on big amounts of data)
• Double amount of space will be used.
• Loss of data. If no remote has the data anymore (and a user clicks on "Revert Local Changes")

[calmh]

Resetting the database is not a normal operational thing, I would rather focus on the cause of the corruption you're seeing. I've been running this thing for a decade now on uncounted boxes and haven't yet got a corrupted database.

[maximilize]

It happened more than once after one VPS provider did some overselling and the HDD performance dropped very much during peek hours.
And it happened on an external SSD where the USB was unstable.

Of course it should not be corrupted, but things can break always. Also a HDD can be corrupted, sectors can break, RAM can malfunction or bits can flip. Without deep verification of the data stored on the disk, the integrity is not guaranteed.

Sadly for me the encrypted folders sound like a half production ready solution.

[er-pa]

Sadly for me the encrypted folders sound like a half production ready solution.

Well, not just for you as the docs regarding this feature are very clear about its state;

Warning

This feature should still be considered beta / testing only.

I'm not entirely sure what's needed to get it out of that state, but I can imagine that the lack of gracefully handling such corner cases is a part of it. Together with some pending bugs.

[maximilize]

Thanks for pointing out the documentation. As I understand it right, the current protocol makes it impossible to verify single blocks.

The hash is encrypted using AES-SIV (with the file key)

The resulting encrypted hash can’t be used for data verification by the untrusted device, but it can be used as a form of “token” referring to a given data block for reuse purposes.

The only way I see is to implement this is by storing the hash of the encrypted data on the trusted side, alongside the hash of the plain data.
This needs to be done for every password in use.

The cause will be a bigger index db, more memory usage (?), and more CPU usage during the hashing process.

Is such a change wanted to syncthing? Or are there any other ideas or plans?

[calmh]

I don't think it's desired in general, no.

[maximilize]

So what other plans are there to get the encryption feature out of the beta status?

[calmh]

I don't see that this is related to that in any way? I think the beta-ness is about the amount of testing, and possibly a couple of known bugs with the storage format that should be fixed. Your issue, paraphrasing, is that you used a nuclear-level debugging option (that we arguably shouldn't even have) because of potential hardware issues, and the resulting user experience wasn't smooth, which isn't surprising to me.

[maximilize]

It's not only not smooth, I believe the missing and unplanned untrusted-side verification is a design flaw and makes syncthing an bad choice for some use cases. As said before, corruption of storage media can and does happen, as well as db corruption.

I rather switch to tools like restic to store my data on an untrusted server since this tool provides industry standards for data safety.