Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed moving of master key encrypted and versioned files between shared folders #16696

Closed
wants to merge 2 commits into from

Conversation

yahesh
Copy link
Member

@yahesh yahesh commented Aug 8, 2019

This fixes #16419 for master key encrypted files.

@yahesh yahesh changed the title fixed moving of encrypted and versioned files between shared folders fixed moving of master key encrypted and versioned files between shared folders Aug 8, 2019
@yahesh
Copy link
Member Author

@yahesh yahesh commented Aug 8, 2019

This fixes the problem for master key encrypted files and improves the situation a bit for user key encrypted files.

@yahesh
Copy link
Member Author

@yahesh yahesh commented Aug 8, 2019

Copied over from #16419 as suggested by @kesselb:

It seems that moving user key encrypted folders leads into some weird locking issues in the $this->file->getAccessList() call of lib/private/Files/Stream/Encryption.php:stream_open() (which we can reproduce reliably but haven't fully understood yet). At least, this locking issue doesn't break files. Copying user key encrypted folders works.

This is a write-down of what the general problem was:
Let's shortly re-iterate on what happens when a file is moved from one shared folder to another shared folder:

  • a copy of the file is put into the target folder
  • a copy of the file is put into the trashbin of the user moving the file
  • the original copy of the file is put into the trashbin of the file owner

The general logic can be found in lib/private/Files/Storage/Wrapper/Encryption.php: moveFromStorage() which first copies the file/folder with the help of lib/private/Files/Storage/Wrapper/Encryption.php:copyBetweenStorage() and then deletes the file/folder.

One task that has to be done is to update the oc_filecache table. Most of this is done by underlying classes but the encrypted value in the database is handled here in lib/private/Files/Storage/Wrapper/Encryption.php:updateEncryptedVersion() and this is also where the problem is buried.

When copyBetweenStorage is first called in moveFromStorage it provides true as the $isRename value. This later prompts updateEncryptedVersion to overwrite the encrypted value of the source file with 1. So back to the process:

  • When the source file is copied to the target file, the encrypted value of the source file is changed into a wrong value.
  • When the source file is copied to the trashbin of the user moving the file, the encrypted value of the source file doesn't match and the file copy fails because the signature checks fails while reading the content.

What we did for now to remedy this problem:

  • At the start of copyBetweenStorage we backup the original encrypted value.
  • We let the rest of the program logic untouched because some underlying code seems to rely on this broken behaviour.
  • At the end of copyBetweenStorage we restore the original encrypted value so that it is the correct value for the next copy task.

@jknockaert
Copy link
Contributor

@jknockaert jknockaert commented Aug 12, 2019

@yahesh Thanks for the PR. Ideally there would be one or more tests added to the codebase in order to avoid future regressions (and they also help to some extent in documenting the PR).
@nextcloud/encryption Reviews needed. Any other groups to invite?

@yahesh
Copy link
Member Author

@yahesh yahesh commented Aug 12, 2019

@jknockaert As you may have read in the corresponding issue, this problem occurs some 32 levels down into the call stack. I really have no idea how to properly mock this misbehaviour so that the requested test isn't much more than a dummy placeholder.

@jknockaert
Copy link
Contributor

@jknockaert jknockaert commented Aug 12, 2019

@yahesh Ja this is pretty low-level stuff.
@schiessle Do you have any advise as to how to design a proper test for this fix?

@papimla
Copy link

@papimla papimla commented May 27, 2020

This looks like a lot of good work (diagnostic + solution) to solve a severe problem. It would be great to see some progress. I'm stuck with files that are most likely in a bad state because of this.

@hostingnuggets
Copy link

@hostingnuggets hostingnuggets commented May 28, 2020

@papimla same here! It seems like @yahesh's work was a waste of time for him...

@kesselb
Copy link
Contributor

@kesselb kesselb commented May 28, 2020

@yahesh mind to have a look at the conflicting files? I don't know how soon one has time to review it but would be nice to have at least a recent ci run.

@papimla
Copy link

@papimla papimla commented May 29, 2020

@papimla same here! It seems like @yahesh's work was a waste of time for him...

It certainly is frustrating when good work is not put to use and is abandoned for some time. But the excellent diagnostic is still valid and will be a great help to get this fixed. I would not go as far as to say it was wasted 😄

@papimla
Copy link

@papimla papimla commented May 29, 2020

@yahesh thanks a lot for updating the pull request !

@hostingnuggets
Copy link

@hostingnuggets hostingnuggets commented May 29, 2020

@papimla exactly, now we only need to hope that the Nextclouders will take their precious time to check this PR.

@juliushaertl
Copy link
Member

@juliushaertl juliushaertl commented Aug 5, 2020

Thanks a lot @yahesh for looking into this and providing such a detailed issue description together with the fix. The changes make sense and the fix works like a charm.

@juliushaertl juliushaertl added the 3. to review label Aug 5, 2020
@juliushaertl juliushaertl requested a review from MorrisJobke Aug 5, 2020
Copy link
Member

@MorrisJobke MorrisJobke left a comment

Code looks good 👍

@MorrisJobke
Copy link
Member

@MorrisJobke MorrisJobke commented Aug 5, 2020

Rebased to trigger CI again. Then this can be merged.

@MorrisJobke MorrisJobke added this to the Nextcloud 20 milestone Aug 5, 2020
@MorrisJobke MorrisJobke removed the 3. to review label Aug 5, 2020
@MorrisJobke
Copy link
Member

@MorrisJobke MorrisJobke commented Aug 5, 2020

Unfortunately the tests fail :/

@MorrisJobke MorrisJobke mentioned this pull request Aug 11, 2020
57 tasks
@MorrisJobke MorrisJobke added 2. developing and removed 4. to release labels Aug 13, 2020
@rullzer rullzer mentioned this pull request Aug 21, 2020
19 tasks
@rullzer rullzer mentioned this pull request Aug 27, 2020
21 tasks
@rullzer rullzer mentioned this pull request Sep 3, 2020
13 tasks
@rullzer rullzer mentioned this pull request Sep 11, 2020
5 tasks
@@ -286,7 +286,7 @@ public function fopen($path, $mode) {
}
}
$info = [
'target' => $this->getMountPoint() . $path,
'target' => $this->getMountPoint() . '/' . $path,
Copy link
Member

@MorrisJobke MorrisJobke Sep 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to #22902 as it is unrelated to this PR.

yahesh and others added 2 commits Sep 17, 2020
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
@MorrisJobke
Copy link
Member

@MorrisJobke MorrisJobke commented Sep 17, 2020

I extracted the unrelated stuff out in #22898, #22902 and #22903. After rebasing this PR and testing it again it doesn't solve the issue for me. But also on master there were already some fixes. @juliushaertl do you mind to check it out again as well? Maybe I have overseen something?

@MorrisJobke MorrisJobke removed this from the Nextcloud 20 milestone Sep 17, 2020
@MorrisJobke MorrisJobke added this to the Nextcloud 21 milestone Sep 17, 2020
@juliushaertl juliushaertl self-requested a review Sep 17, 2020
@rullzer rullzer mentioned this pull request Dec 14, 2020
59 tasks
@rullzer rullzer mentioned this pull request Dec 18, 2020
39 tasks
@kesselb
Copy link
Contributor

@kesselb kesselb commented Dec 25, 2020

@yahesh mind to do another rebase to have recent CI run? Heads up: Your branch was changed in the meantime.

@rullzer rullzer mentioned this pull request Dec 28, 2020
39 tasks
@rullzer rullzer mentioned this pull request Jan 4, 2021
5 tasks
@rullzer rullzer removed this from the Nextcloud 21 milestone Jan 8, 2021
@rullzer rullzer added this to the Nextcloud 22 milestone Jan 8, 2021
@MorrisJobke MorrisJobke mentioned this pull request May 20, 2021
106 tasks
@MorrisJobke
Copy link
Member

@MorrisJobke MorrisJobke commented May 20, 2021

Let's close this one for now. If it still solves an issue we can reopen it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment