Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use sync instead of copy for blob storage #6298

Merged
merged 3 commits into from Oct 22, 2019

Conversation

@stsewd
Copy link
Member

stsewd commented Oct 16, 2019

We use rsync for our normal storage, but for blob storage we are just doing a copy.

Full explanation at #6186 (comment)

This is heavily borrowed from a snippet from @davidfischer :)

Fixes #6069
Closes #6186

@stsewd stsewd requested review from davidfischer and readthedocs/core Oct 16, 2019
dest_folders, dest_files = self.listdir(self._dirpath(destination))
for folder in dest_folders:
if folder not in copied_dirs:
self.delete_directory(self.join(destination, folder))

This comment has been minimized.

Copy link
@stsewd

stsewd Oct 16, 2019

Author Member

I changed this, because set1 - set2 creates another set. not in is O(1).


tree = [
# Cloud storage generally doesn't consider empty directories to exist
('api', []),

This comment has been minimized.

Copy link
@stsewd

stsewd Oct 16, 2019

Author Member

I was about to fix this to delete all dirs on FileSystemStorage, but looks like we expect that and probably isn't a big deal.

# We don't check "dirs" here - in filesystem backed storages
# the empty directories are not deleted
# Cloud storage generally doesn't consider empty directories to exist

- Linting fix
- Check against suspicious operations
- Additional logging
Copy link
Contributor

davidfischer left a comment

I fixed the linting error and added a few other cleanup parts. This is good to go and should fix the lingering storage issues. Thanks for doing this @stsewd!

Copy link
Member

ericholscher left a comment

This makes sense, and I'm guessing will help fix the lingering issues we have with files sticking around after deletion?

@stsewd

This comment has been minimized.

Copy link
Member Author

stsewd commented Oct 22, 2019

This makes sense, and I'm guessing will help fix the lingering issues we have with files sticking around after deletion?

No completely, we still need to remove files when a version is deactivated, but I'm making another PR about that.

@stsewd stsewd merged commit 8ab13f4 into readthedocs:master Oct 22, 2019
2 checks passed
2 checks passed
continuous-documentation/read-the-docs Read the Docs build succeeded!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@stsewd stsewd deleted the stsewd:stsewd/sync-instead-of-copy branch Oct 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.