fix(dump): Uncompress the dump IN the data.ms #2057
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
irevoire
force-pushed
the
fix-docker-dump-import
branch
from
348dd1f
to
c05c8c7
Compare
When loading a dump with docker, we had two problems. After creating a tempdirectory, uncompressing and re-indexing the dump: 1. We try to `move` the new “data.ms” onto the currently present one. The problem is that if the `data.ms` is a mount point because that's what peoples do with docker usually. We can't override a mount point, and thus we were throwing an error. 2. The tempdir is created in `/tmp`, which is usually quite small AND may not be on the same partition as the `data.ms`. This means when we tried to move the dump over the `data.ms`, it was also failing because we can't move data between two partitions. ============== 1 was fixed by deleting the *content* of the `data.ms` and moving the *content* of the tempdir *inside* the `data.ms`. If someone tries to create volumes inside the `data.ms` that's his problem, not ours. 2 was fixed by creating the tempdir *inside* of the `data.ms`. If a user mounted its `data.ms` on a large partition, there is no reason he could not load a big dump because his `/tmp` was too small. This solves the issue; now the dump is extracted and indexed on the same partition the `data.ms` will lay. fix #1833
irevoire
force-pushed
the
fix-docker-dump-import
branch
from
c05c8c7
to
c9c7da3
Compare
irevoire
changed the title
MarinPostma
approved these changes
Jan 10, 2022
|
bors merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When loading a dump with docker, we had two problems.
After creating a tempdirectory, uncompressing and re-indexing the dump:
movethe new “data.ms” onto the currently presentone. The problem is that if the
data.msis a mount point becausethat's what peoples do with docker usually. We can't override
a mount point, and thus we were throwing an error.
/tmp, which is usually quite small AND may notbe on the same partition as the
data.ms. This means when we tried to movethe dump over the
data.ms, it was also failing because we can't move databetween two partitions.
1 was fixed by deleting the content of the
data.msand moving the contentof the tempdir inside the
data.ms. If someone tries to create volumes insidethe
data.msthat's his problem, not ours.2 was fixed by creating the tempdir inside of the
data.ms. If a user mountedits
data.mson a large partition, there is no reason he could not load a bigdump because his
/tmpwas too small. This solves the issue; now the dump isextracted and indexed on the same partition the
data.mswill lay.fix #1833