Don't fsync every file when installing gems #556
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
d84090b introduced "similar sync behavior from old tar code". It looks like the old code was fsyncing directories, but the commit was fsyncing every single file.
drbrain
added a commit
that referenced
this pull request
Jun 26, 2013
Don't fsync every file when installing gems
drbrain
added a commit
that referenced
this pull request
Jun 26, 2013
This assumes writes by the filesystem will be ordered which may not be a safe, but does ensure at least the spec is written to disk before returning from installation.
|
In @801ba9e I moved the fsync() to a single file, the gem's specification. As noted this assumes that writes are ordered and that the fsync() for the specification will force all the files installed by the gem to also be synced to disk which may be unsafe, but I would like to make a reasonable effort to commit the gem to permanent storage in case of a crash. Let me know if this removes the performance benefit of your commit and we can reopen. |
|
Thanks! I'll run a few tests to see if we see the same performance gain with fsyncing the gemspecs. |
drbrain
added a commit
that referenced
this pull request
Jun 27, 2013
This assumes writes by the filesystem will be ordered which may not be a safe, but does ensure at least the spec is written to disk before returning from installation.
|
@drbrain Whoops, sorry for the late reply, completely forgot about it. Everything is fast now, even with added fsync for specifications. Thanks again! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
d84090b introduced "similar sync behavior from old tar code". It looks like the old code was fsyncing directories, but the commit was fsyncing every single file.
This fixes a severe performance degradation in our virtualized environment. Gem install takes ~40 seconds on rubygems 2.0 compared to ~1 second on rubygems 1.8.
I looked through the last version that didn't have those performance issues - 1.8.25 - and the Gem::Package::FSyncDir module is dead code there.
I don't know rubygems well enough to say if its absolutely safe to remove this code, but from my limited testing it seems that it is. I installed a few gems with checking if any TarReader::Entry is a directory, and none of them is.