Use buffered IO when writing the output jar. #666
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Simplify and optimize manifest modification logic We only need to modify a file in the Jar but end up manually copying all entries. For small jars this is not an issue, but for larger jars it has terrible performance (specially in ci). E.g on my M1 mac, stamping buckCompiler jar takes ~20 seconds with the naive implementation. With the changes in this commit it now takes ~1.5s. The performance is even worse on ci which has terrible io (+10min just to stamp this jar) * avoid creating file with bad attrs
mauriciogg
requested review from
c-parsons,
cheister,
jin and
shs96c
as code owners
|
The test failures look related to this change. Could you please make sure that |
This improves io performance for large jars Running this code on my machine with a compressed 70MB zip with ~40K entries now takes ~5s from ~25s. For an uncompressed jar this is a subsecond operation.
mauriciogg
changed the title
The failing tests would have been hard to fix using the FileSystem API, since the manifest has to be the first (not sure why java needs it there) entry and FileSystem hides the zip impl details and operates at a higher level. Turns out that simply wrapping the JarOutputStream around a buffered stream makes the performance good enough so I ended up just going that route. I updated the PR to reflect this change. |
shs96c
approved these changes
Feb 19, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This improves io performance for large jars
Running this code on my machine with a compressed 70MB zip with ~40K
entries now takes ~5s from ~25s. For the same uncompressed jar goes from ~20s to a subsecond operation.