OCC Decrypt all always fails to complete. #25599

Closed
wipeoutdude opened this Issue Jul 25, 2016 · 13 comments

Projects

None yet

7 participants

@wipeoutdude

Firstly let me say that Owncloud is brilliant..

When I first installed Owncloud back in the 8.x days I didn't properly grasp how the encryption module works and that locally encrypted files are a little pointless since the keys are sitting there too.. 100% of our storage is local so I really shouldn't have even enabled the module in the first place..

For quite a while now I have been trying to run the decrypt-all OCC command to remove encryption from the local files but it always fails.. I upgraded to OC 9.1 over the weekend and tried again but it still failed.. There are no specific error messages it simply appears to stop after a number of hours..

Another issue that seems to be causing the process to run a LOT longer than it needs to is the attempted re-decrypting of shared files over and over again for each user who has access to those files.. 99% of the files stored on the server are shared between most/all of the users and the decryption is running through the same files over and over again.. Surely if the file has been decrypted in step 1 of x it doesn't need to be re-decrypted for each user that has access to that file?

Perhaps I need to decrypt user by user individually then i won't try re-decrypt the files? Issue there is that it seems the first step is to disable the encryption module so not sure how that would work out.. Or maybe I have to un-share all files then decrypt-all and then re-share the files again but that too would be quite a job and would cause a lot of resyncing of data to the PC's..

The documentation doesn't suggest there is a special process to follow to remove and disable encryption..

Steps to reproduce

  1. run ./occ encryption:decrypt-all

Expected behaviour

All files for all users decrypted and encryption module no longer required.

Actual behaviour

Process starts and runs for a very long time through step 1 of 7, 2 of 7.... Then it falls over..
Last attempt made it to 6 of 7 before stopping but other attempts haven't made it as far.. This took about 3 hours.. All files then totally inaccessible through the web interface so I then have to completely restore the server from backup..

Server configuration

Operating system:
Cpanel server running Centos

Web server:
Apache under Cpanel

Database:
Mysql under Cpanel

PHP version:
5.6.20

ownCloud version: (see ownCloud admin page)
Currently 9.1 but have had the issue through all the 9.0.x versions since I started trying to remove the encryption module.

Updated from an older ownCloud or fresh install:
Upgraded

Where did you install ownCloud from:
Owncloud website

Signing status (ownCloud 9.0 and above):
No errors found

List of activated apps:
Enabled:

  • activity: 2.3.2
  • comments: 0.3.0
  • dav: 0.2.5
  • encryption: 1.3.0
  • federatedfilesharing: 0.3.0
  • files: 1.5.1
  • files_pdfviewer: 0.8.1
  • files_sharing: 0.10.0
  • files_texteditor: 2.1
  • files_trashbin: 0.9.0
  • files_versions: 1.3.0
  • files_videoplayer: 0.9.8
  • firstrunwizard: 1.1
  • notifications: 0.3.0
  • provisioning_api: 0.5.0
  • templateeditor: 0.1
  • updatenotification: 0.2.1
    Disabled:
  • external
  • federation
  • files_antivirus
  • files_external
  • gallery
  • systemtags
  • user_external
  • user_ldap

The content of config/config.php:

Default config

Are you using external storage, if yes which one: local/smb/sftp/...
No

Are you using encryption: yes/no
Yes

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

Client configuration

Browser:
SSH

@DeepDiver1975
Member

@owncloud/qa please verify - THX

@davitol
Contributor
davitol commented Jul 27, 2016

@wipeoutdude when you run ./occ encryption:decrypt-all command via CLI, did you get the following message in your screen?

Files for following users couldn't be decrypted,
maybe the user is not set up in a way that supports this operation:
username

@wipeoutdude

@davitol - No, I don't get that message, it appears to be processing through all users just fine but has never completed yet and as I said appears to be going thorough the same files (shared files) over and over again..

@schiessle
Member
schiessle commented Jul 27, 2016 edited

@wipeoutdude this shouldn't happen. We skip files which are already decrypted, see https://github.com/owncloud/core/blob/master/lib/private/Encryption/DecryptAll.php#L220

Maybe because of some caching issues it might happen that your server still reads the old value? I couldn't reproduce it. For many large shared folders it might even have a performance impact if it works as expected, just because we look at every shared file just to decide that nothing has to be done. Therefore I proposed this PR at the Nextcloud repository, maybe you want to give it a try if this changes anything for you: nextcloud/server#608

@DeepDiver1975 feel free to copy it if it turns out to be useful. If we ignore the tests, the changes should be small enough to avoid copyright issues.

@DeepDiver1975
Member

@DeepDiver1975 feel free to copy it if it turns out to be useful. If we ignore the tests, the changes should be small enough to avoid copyright issues.

Thanks for the pointer @schiessle - much appreciated!

@DeepDiver1975 DeepDiver1975 self-assigned this Jul 28, 2016
@DeepDiver1975 DeepDiver1975 added this to the 9.2 milestone Jul 28, 2016
@DeepDiver1975 DeepDiver1975 added the bug label Jul 28, 2016
@pluempyx92

@wipeoutdude I had a simular issue while decrypting all files. It takes very long and shared files are decrypted multiple times.
@schiessle I think the code for skipping decrypted files is not working. the code you are referring to expectes "$file->isEncrypted()" to be a boolean. In my test this seems to be a different data type. As a quick and dirty workaround i removed one "=" (use equal comparison instead of identical) to "fix" the condition. This solves the problem for me, but is not a valid/solid solution, obviously. Maybe this should be handled as separate issue.

@DeepDiver1975
Member

In my test this seems to be a different data type. As a quick and dirty workaround i removed one "=" (use equal comparison instead of identical) to "fix" the condition.

What type did you see? THX

@pluempyx92

gettype($file->isEncrypted()) says it is a string

@DeepDiver1975
Member

gettype($file->isEncrypted()) says it is a string

what are the values you are observing?

@pluempyx92
pluempyx92 commented Aug 14, 2016 edited

what are the values you are observing?

I get "1" for encrypted and "0" for unencrypted files.
The values are equal to the field "encrypted" in the "oc_filecache" database table.

@DeepDiver1975
Member

THX

@PVince81 PVince81 closed this in #25639 Aug 17, 2016
@jknockaert
Contributor

While this was obviously a valid bug report, I would like to point out that the motivation for decrypting as stated by @wipeoutdude does not make much sense to me. There seem to be a couple of misunderstandings as to how encryption works that keep coming back:

  1. "local encryption doesn't serve a purpose": it does. One cannot decrypt an encrypted file using the stored keys as the only input, one also needs the user password. This is not different from how full disk encryption works.
  2. (point not made by @wipeoutdude but elsewhere) "encryption is cpu intensive": it is not. All common hardware and software (i.e. openssl and intel cpus) support hardware accelerated encryption/decryption transparently since half a decade now. I never ran a benchmark, but the order of magnitude of the impact of enabling encryption on processing time should be around 1%.

Maybe we should consider documenting this somewhere?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment