OCC Decrypt all always fails to complete.

[wipeoutdude]

Firstly let me say that Owncloud is brilliant..

When I first installed Owncloud back in the 8.x days I didn't properly grasp how the encryption module works and that locally encrypted files are a little pointless since the keys are sitting there too.. 100% of our storage is local so I really shouldn't have even enabled the module in the first place..

For quite a while now I have been trying to run the decrypt-all OCC command to remove encryption from the local files but it always fails.. I upgraded to OC 9.1 over the weekend and tried again but it still failed.. There are no specific error messages it simply appears to stop after a number of hours..

Another issue that seems to be causing the process to run a LOT longer than it needs to is the attempted re-decrypting of shared files over and over again for each user who has access to those files.. 99% of the files stored on the server are shared between most/all of the users and the decryption is running through the same files over and over again.. Surely if the file has been decrypted in step 1 of x it doesn't need to be re-decrypted for each user that has access to that file?

Perhaps I need to decrypt user by user individually then i won't try re-decrypt the files? Issue there is that it seems the first step is to disable the encryption module so not sure how that would work out.. Or maybe I have to un-share all files then decrypt-all and then re-share the files again but that too would be quite a job and would cause a lot of resyncing of data to the PC's..

The documentation doesn't suggest there is a special process to follow to remove and disable encryption..

Steps to reproduce

1. run ./occ encryption:decrypt-all

Expected behaviour

All files for all users decrypted and encryption module no longer required.

Actual behaviour

Process starts and runs for a very long time through step 1 of 7, 2 of 7.... Then it falls over..
Last attempt made it to 6 of 7 before stopping but other attempts haven't made it as far.. This took about 3 hours.. All files then totally inaccessible through the web interface so I then have to completely restore the server from backup..

Server configuration

Operating system:
Cpanel server running Centos

Web server:
Apache under Cpanel

Database:
Mysql under Cpanel

PHP version:
5.6.20

ownCloud version: (see ownCloud admin page)
Currently 9.1 but have had the issue through all the 9.0.x versions since I started trying to remove the encryption module.

Updated from an older ownCloud or fresh install:
Upgraded

Where did you install ownCloud from:
Owncloud website

Signing status (ownCloud 9.0 and above):
No errors found

List of activated apps:
Enabled:

• activity: 2.3.2
• comments: 0.3.0
• dav: 0.2.5
• encryption: 1.3.0
• federatedfilesharing: 0.3.0
• files: 1.5.1
• files_pdfviewer: 0.8.1
• files_sharing: 0.10.0
• files_texteditor: 2.1
• files_trashbin: 0.9.0
• files_versions: 1.3.0
• files_videoplayer: 0.9.8
• firstrunwizard: 1.1
• notifications: 0.3.0
• provisioning_api: 0.5.0
• templateeditor: 0.1
• updatenotification: 0.2.1
  Disabled:
• external
• federation
• files_antivirus
• files_external
• gallery
• systemtags
• user_external
• user_ldap

The content of config/config.php:

Default config

Are you using external storage, if yes which one: local/smb/sftp/...
No

Are you using encryption: yes/no
Yes

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

Client configuration

Browser:
SSH

[DeepDiver1975]

@owncloud/qa please verify - THX

[davitol]

@wipeoutdude when you run ./occ encryption:decrypt-all command via CLI, did you get the following message in your screen?

Files for following users couldn't be decrypted,
maybe the user is not set up in a way that supports this operation:
username

[wipeoutdude]

@davitol - No, I don't get that message, it appears to be processing through all users just fine but has never completed yet and as I said appears to be going thorough the same files (shared files) over and over again..

[schiessle]

@wipeoutdude this shouldn't happen. We skip files which are already decrypted, see https://github.com/owncloud/core/blob/master/lib/private/Encryption/DecryptAll.php#L220

Maybe because of some caching issues it might happen that your server still reads the old value? I couldn't reproduce it. For many large shared folders it might even have a performance impact if it works as expected, just because we look at every shared file just to decide that nothing has to be done. Therefore I proposed this PR at the Nextcloud repository, maybe you want to give it a try if this changes anything for you: nextcloud/server#608

@DeepDiver1975 feel free to copy it if it turns out to be useful. If we ignore the tests, the changes should be small enough to avoid copyright issues.

[DeepDiver1975]

@DeepDiver1975 feel free to copy it if it turns out to be useful. If we ignore the tests, the changes should be small enough to avoid copyright issues.

Thanks for the pointer @schiessle - much appreciated!

[pluempyx]

@wipeoutdude I had a simular issue while decrypting all files. It takes very long and shared files are decrypted multiple times.
@schiessle I think the code for skipping decrypted files is not working. the code you are referring to expectes "$file->isEncrypted()" to be a boolean. In my test this seems to be a different data type. As a quick and dirty workaround i removed one "=" (use equal comparison instead of identical) to "fix" the condition. This solves the problem for me, but is not a valid/solid solution, obviously. Maybe this should be handled as separate issue.

[DeepDiver1975]

In my test this seems to be a different data type. As a quick and dirty workaround i removed one "=" (use equal comparison instead of identical) to "fix" the condition.

What type did you see? THX

[pluempyx]

gettype($file->isEncrypted()) says it is a string

[DeepDiver1975]

gettype($file->isEncrypted()) says it is a string

what are the values you are observing?

[pluempyx]

what are the values you are observing?

I get "1" for encrypted and "0" for unencrypted files.
The values are equal to the field "encrypted" in the "oc_filecache" database table.

[DeepDiver1975]

THX

[jknockaert]

While this was obviously a valid bug report, I would like to point out that the motivation for decrypting as stated by @wipeoutdude does not make much sense to me. There seem to be a couple of misunderstandings as to how encryption works that keep coming back:

1. "local encryption doesn't serve a purpose": it does. One cannot decrypt an encrypted file using the stored keys as the only input, one also needs the user password. This is not different from how full disk encryption works.
2. (point not made by @wipeoutdude but elsewhere) "encryption is cpu intensive": it is not. All common hardware and software (i.e. openssl and intel cpus) support hardware accelerated encryption/decryption transparently since half a decade now. I never ran a benchmark, but the order of magnitude of the impact of enabling encryption on processing time should be around 1%.

Maybe we should consider documenting this somewhere?

[guruz]

@jknockaert You can edit and send a pull request https://github.com/owncloud/documentation/blob/master/admin_manual/configuration_files/encryption_configuration.rst :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.