New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Moving files/folders from/to shared folders causes Encryption Errors #16419
Comments
It seems that we have the same issue (see nextcloud/desktop#1168). We're currently looking into this issue as the corresponding restores take a long time and we'd like to have this issue resolved. We're trying to reproduce this problem by creating two shared folders and moving huge subfolders between those two shared folders. One thing we've seen is that at some point the web UI issues as warning message that moving the subfolder failed. However, when watching the target folder we see that files are still moved after this message has been issued. Even after all files seem to have been moved on disk, the corresponding PHP process is still hard at work and the database is active as well. (As long as this task isn't finished, the moved files don't show up in the web UI.) As we've learned, when moving files around the Furthermore, it seems as if files get completely re-encrypted when copying or moving them between shared folder. In contrast, they "only" get re-encrypted when being copied within the same shared folder. |
We now found a reliable way that breaks the signature in 100% of our test cases. Howto:
There are some more things we found out in the meantime: We were wondering why the files got re-encrypted when moving them from one shared folder to another shared folder. The reason for the re-encryption is that the file is not really moved but it is rather copied over and then deleted. This leads to several things happening:
|
Final revelation: Encrypted files must have an We found out that Nextcloud stores the value The problem persists that moved files may be signed with other |
To be able to debug all this we have written two helpful tools by reimplementing the signature checks (and updates) of Nextcloud and by reimplementing the decryption process of Nextcloud. These are standalone scripts that need some configuration information of Nextcloud, but not the Nextcloud codebase itself:
|
Thanks for the extensive debugging, that will sure help finding and fixing the issue! |
cc @icewind1991 |
I just wanted to document some more thoughts on the design of the encryption and signature scheme:
|
To get a complete understanding of the inner workings of the default encryption module we now also implemented the support for public sharing keys, recovery keys and user keys in our nextcloud-tools. Our goal is to create a document containing our newly-gained knowledge about how the encryption works. |
We're currently in the process of testing our Nextcloud dataset with the written tools. We stumbled upon lots of files that seemingly have an
We're not sure yet where these duplicate entries come from. Even in our test installation some files have duplicate entries in the mentioned formats. |
We also found that you should not use |
Uff, yeah files:scan should be really used with care. I would argue to even disable it when encryption is on. |
It's hard to know when a file is encrypted ... it could also be just a normal file with similar headers. Thus the file:scan can't do much here. Also moving files on the hard disk is just not supported. Doing so causes always (also non-encrypted) issues, because it is from our side just guessing and files are simply just removed and newly added and thus shares are also lost. Long story short: don't move files on filesystem level if you do not want to loose metadata (like shares, encryption state, tags, activities, versions, ...). |
@MorrisJobke Don't misunderstand. This is just one thing we noticed while testing how the encryption module reacts. The problem with bad signature checks also occurs when moving files on the application level (see the mentioned text file example above). As there doesn't seem to be an extensive description of how the encryption works we have to find this out ourselves. |
@MorrisJobke @nextcloud/encryption @nickvergessen @rullzer Due to this issue why dug pretty deep into the default encryption module and gathered at lot of knowledge about the inner workings of the encryption and signature processes. We created a document called server-side-encryption.md that contains the general knowledge and tiny details that we learned. Is there the possibility to add this to the official documentation of the encryption module so that this knowledge is easier to find for others? |
I guess it could either go into https://github.com/nextcloud/documentation/blob/master/admin_manual/configuration_files/encryption_configuration.rst or in a new page next to it. |
Now that we know how to calculate the MACs and decrypt files we started looking into the actual problems of the encryption module more closely: We started by creating a reproducible failure again. To do this we created three files called
Fileid Now we try to move
Fileid Fileid Fileids The actual file containing
As you can see the actual file containing |
Some more things we learned about copying and moving files around:
This leads us to the conclusion that files seem to be handled differently depending on whether they are handled by the owner of the shared folder or by the recipient of a shared folder. Putting broken files in the trashbin and failing to properly move a file to a shared folder seem to be two different problems. One seems to be related with the ownership of the source folder while the other seems to be related to the ownership of the target folder. There are also some database inconsistencies that we saw during our tests:
|
Could you please avoid such strong language ? It makes me feel unsafe. |
I can also confirm that this bug still exists (Version 18.0.6). |
I think this bug still exists as of Nextcloud 19.0.4 So this means that there is also an invisible side effect of this, the database contains lots of duplicates in the cache table? Also to the Nextcloud maintainers: because of that very issue, I tried to get away from nextcloud all together. They create dataloss. That's not acceptable for a cloud solution. |
Yes, me and my company also wanted to leave because of that issue. Let's hope one day it will get solved… |
Same here, still not working correctly and no feedback from the security chiefs at Nextcloud. I would be glad if someone can fix this after so many years. |
@inthreedee looking forward to these fixes and hopefully this can happen soon. Please also think about backporting these fixes to Nextcloud 20, in case it ever makes it into Nextcloud 21. |
Starting some days ago, I get more and more broken files. Yesterday three files got broken, today already 8 files. I did not change anything on my nextcloud. Nextcloud can no longer sync those file to my clients and I can no longer download or open those files. When I click on download, I get ERR_INVALID_RESPONSE, the website can not be reached. I have encryption enabled, with master key, recovery key and encryption keys for every single user. The broken files seem to only occur in shared folders. But they also occur in shared folders, that are only shared by a dummy user to me alone (only me accessing those files). I am not sure, if I really lost those files or if I have already renamed and moved those files, but the sync was partially broken, resulting in broken leftover files on the cloud. It seems to primarily affect new scanned documents starting with S30C-xxxxxxxxxxxx.pdf or S30C-xxxxxxxxxxxx.jpg. Server configuration: |
What is the actually reason that this one is not "simple" fixed? Adding a FAQ-entry for an security issue (data loss is security) where a nearly ready patch is available sound weird to me. |
I presume this issue still persists in Nextcloud 20.
The PR from @yahesh seems concise, what is the reason is is still blocked? I'd really like to have this fixed ;) In case there is still work to be done, could someone please elaborate on that? I mean, sharing files is a fundamental part of Nextcloud, isn't it? |
@yahesh The problem is that since I wrote the fix some changes in the codebase or the test suite lead to some tests failing for this change. However, as all tests have passed at the time when I created the pull request, I don't feel obliged to debug that problem on my own. |
@yahesh could you please see and mention the persons did these changes in the codebase which breaks your code? We have to ping them so that they fix what they broke asap else your hard work will just be useless :( |
Same problem here. Dataloss with moving encrypted files/folders occurred first with owncloud, years ago, and still occurs with nextcloud (we're stuck with NC 15, but as far as I can see there's no progress in newer versions). |
We started on 15 with the server side encryption and are now on 20. On our side the issue still exists. Some weeks ago a user crashed roughly 25% of all data in one moment. Luckily I could restore full file system and db backups which took me about 48 hrs restless work. But especially with this experience I fear to deactivate the server side encryption or to fresh install. Our system is hosting millions of files of like 10 years. The good thing of this event was that I now know that my backup strategy works and where to optimize it. |
I've looked into Roeland's fix in #25249 and discovered that the problem is not reproducible any more in recent Nextcloud versions, so it is likely that the fix is not needed any more as it was fixed through a different way. If you do have files with "bad signature", please run the command mentioned in https://docs.nextcloud.com/server/latest/admin_manual/issues/general_troubleshooting.html#problems-when-downloading-or-decrypting-files for the affected files/folders to repair their signature. After that, I expect that move operations won't cause any troubles. After that, please let me know if anyone is still experiencing "bad signature" errors on files after moving. |
moving a folder structure worked fine for me in NC 20.0.10 and 21.0.4, but it used to be broken. I'll close this for now. |
Hi, I don't fully understand why this has been closed, as the issue is still there. Are you saying it has been fixed for the latest release only? If so the issue should be kept open until the patch has been backported for older (but still supported) releases. Thank you! |
NC 20.0.12 is the latest supported version and in my tests I couldn't reproduce the issue there (but could on older NC 20.0 versions), so the fix must be there already. @redtux what version are you on ? |
Hi @PVince81, thank you for your quick reply! I am using NC 21.0.3 on a Hetzner instance - apart from Nextcloud GmbH probably the biggest provider of NC instances atm. When trying to open an affected PDF inside the browser (I have kept them untouched for the last years just in case this gets fixed some day, so I could test the patch with the affected files), I get the following error:
Or are those files just unrecoverable crypto junk now that can be deleted? |
see https://docs.nextcloud.com/server/latest/admin_manual/issues/general_troubleshooting.html#problems-when-downloading-or-decrypting-files and run it on the file in question this is only if the error in log is "bad signature", for anything else it could be a different issue |
Hi, I already tried - but it is not working…
Verifying the content of whole directory shows for every single file status The file is listed when connecting via WebDAV, but accessing (opening/downloading) it results in an error 500… |
@redtux can you check:
Depending what you find, a manual repair might be necessary as the "fix-encrypted-version" command didn't seem to work. |
Steps to reproduce
Expected behaviour
Folder should move without errors
Actual behaviour
Encryption errors
Server configuration
Operating system: Debian 9
Web server: NGINX
Database: MariaDB, Redis
PHP version: 7.3
Nextcloud version: 16.0.3
Updated from an older Nextcloud/ownCloud or fresh install: fresh
Where did you install Nextcloud from: latest archive
Signing status:
Signing status
List of activated apps:
App list
Nextcloud configuration:
Config report
Are you using external storage, if yes which one: no
Are you using encryption: yes
Are you using an external user-backend, if yes which one: no
Client configuration
Browser: Chrome
Operating system: Ubuntu 19.04
Logs
Nextcloud log GUI
Nextcloud log
The text was updated successfully, but these errors were encountered: