[Build] Build hdfs fixture faster and less #118801
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
breskeby
added
>non-issue
:Delivery/Build
Collaborator
|
Pinging @elastic/es-delivery (Team:Delivery) |
The building of the shadowed hdfs2 and hdfs3 fixtures takes quite long time due to being 51 and 80mb in size. By removing non used dependencies from the shadow jar creation we can speed up this significantly
breskeby
force-pushed
the
make-hdfsfixture-effecient
branch
from
90d5754 to
f0a0d87
Compare
breskeby
changed the title
Contributor
Author
|
simple benchmark on my local machine of building those hdfs fixture jars: 
|
breskeby
added
the
auto-merge-without-approval
mark-vieira
reviewed
Dec 18, 2024
| exclude(dependency("commons-codec:commons-codec:.*")) | ||
| exclude(dependency("org.apache.httpcomponents:httpclient:.*")) | ||
| exclude(dependency("org.apache.httpcomponents:httpcore:.*")) | ||
| // exclude(dependency("org.apache.httpcomponents:httpclient:.*")) |
| } | ||
| } | ||
|
|
||
| tasks.register('showHdfs3Jar', Copy) { |
Contributor
There was a problem hiding this comment.
I don't see what these tasks are used for? Did you add them just to debug what was in these configurations?
Contributor
Author
There was a problem hiding this comment.
cleaning up. only used for some debugging
mark-vieira
approved these changes
Dec 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The building of the shadowed hdfs2 and hdfs3 fixtures takes quite long time due to being 51 and 80mb in size.
By removing non used dependencies from the shadow jar creation we can speed up this significantly.
Also by tweaking the dependency definitions in the dependent projects we can avoid building the expensive shadowed jars for compile dependencies. this safes us a lot especially for precommit checks