Allow low-level decryption of files (without signature check)

[tflidd]

We had several reports of people having problems to restore encrypted files due to a failing signatrue check. It was added to enhance file integrity. However, if this options prevents you from recovering encrypted files, this is a serious problem. Here just a few github issues related to this problem:
#2206
#5359
#3958
and the forum: https://help.nextcloud.com/search?q=bad%20signature

Nobody seems to really care how to fix this problem. In some cases, this could be related to restoring data or with versioning??

So I propose to provide a low-level decryption tool, where you need only the encrypted file, the encryption key, and the user's password. Or the occ-decryption command with an option to ignore bad signatures, or even on the web-interface.

@schiessle

[berho]

I think the better solution is to stabilize the source code for server-side encryption! Also in the current version (13.0.2) there must be some errors. I have uploaded 300 GB images in the last weeks. Some folders could not create thumbnails and the download failed..... Have to find out this files and upload again!

I've wasted a lot of time and money!

[suntorytimed]

This is a crucial fallback command. In my case I switched off the encryption following some documentation on the internet without knowing that this doesn't decrypt all the files. After that I renamed a folder on the machine instead of Nextcloud and did a file:sync. This resulted in a change of the database and loss of all information that those files are indeed encrypted. Now all those files can't be accessed anymore as the decrypt:all step can't decrypt those files.

[suntorytimed]

Having an occ option like "--no-signature-check" for recovery purposes is a must in my opinion. I understand that there are reasons why you want to check the signature and that not checking them might be less secure in terms of integrity. But there is nothing to gain in terms of integrity if I loose all my files because I can't decrypt them without a signature that changes way too often.

@schiessle @tflidd

[suntorytimed]

I did find a way to turn off the signature check by adding return true; to the checkSignature() in apps/encryption/lib/Crypto/Crypt.php. I added it in the if clause right before the exception is thrown. But switching it off isn’t enough.

The reason is that the server reports a different filesize to the client and breaks off the download too early. The client therefore thinks that the connection was lost and reports an error. But the file is already downloaded successfully (f.e. in Chrome you just have to remove .crdownload at the end of the downloaded file). I have written a small Python 3 script that can download the files via WebDav. It is a dirty hack, but at least I could recover my files.

You can find the script including an explanation in my gitea repository:
https://gitea.hibiki.eu/suntorytimed/nc-downloader

After checking the downloads I discovered that while the JPEGs open without any problem my RAW files didn't. Looking closer at the JPEGs I could see that in the last pixel line there were some blocks missing. So the download wasn't finished. Following up on the error message that gets displayed in Nextcloud in the hasSignature() call of splitMetaData() I discovered that the encrypted data field was empty and therefore there can't be a signature in the file. To bypass this I have added following if clause into the function symmetricDecryptFileContent() in apps/encryption/lib/Crypto/Crypt.php:

            if ($keyFileContents == '') {
                    return '';
            }

I have put this code as the first command in the symmetricDecryptFileContent(). Together with disabling the signature check (putting return true; in the checkSignature() function in the same file):

    private function checkSignature($data, $passPhrase, $expectedSignature) {
            $signature = $this->createSignature($data, $passPhrase);
            if (!hash_equals($expectedSignature, $signature)) {
                    return true;
                    throw new GenericEncryptionException('Bad Signature', $this->l->t('Bad Signature'));
            }
    }

I can now see the previews in the web interface and download all files decrypted and even download the folders as zip-files. My script is not necessary anymore 😃

[yahesh]

@tflidd I don't know if this is still relevant for you but we've written such a tool: decrypt-file.php

[yahesh]

As a small update to this topic: I've got several support requests in the last weeks because people were not able to use the decrypt-file.php on their own. I therefore went a step further now and implemented the decrypt-all-files.php script now. Its main advantage is that it scans the whole data directory for files and tries to decrypt them.

It is also possible to decrypt files of a mixed-up encryption setup where some files are encrypted with the master-key encryption while others are encrypted with the user-key encryption or the pubShare key or the recovery key. The script will try to gather as many decryption keys as possible.

I intend to add an elaborate description to the file so that it serves as a stand-alone rescue script.

P.S.: I don't understand why such a standalone script hasn't been developed by the core maintainers years ago as there obviously seems to be a need for such a rescue tool.

[PVince81]

you can fix files with "bad signature" using this new command: https://docs.nextcloud.com/server/latest/admin_manual/issues/general_troubleshooting.html#problems-when-downloading-or-decrypting-files

[joshtrichards]

So between both of the following now being in place, is this Issue still in need of further work or should it be closed?

• https://docs.nextcloud.com/server/latest/admin_manual/issues/general_troubleshooting.html#troubleshooting-downloading-or-decrypting-files
• https://github.com/nextcloud/encryption-recovery-tools

Perhaps some additional documentation is still needed? If so, what's the scope and let's get an Issue open for it in https://github.com/nextcloud/documentation

[suntorytimed]

@joshtrichards I personally stopped using Nextcloud after this issue and a partial data loss, so I can’t give any feedback.