gradle: improve airbyte-docker performance #30299
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context: https://airbytehq-team.slack.com/archives/C03VDJ4FMJB/p1694426749391339
Problem statement:
When running
./gradlew format
, all projects are loaded and, sinceformat
relies ongenerate
due to the need to apply formatting rules on generated artifacts which are checked-in, andgenerate
is a dependency onairbyteDocker
, the latter task would be created for all projects. This would trigger expensive calls todocker
for all projects, incurring significant overhead.Mitigation:
First of all, reducing the number of
generate
tasks to a minimum seems sensible, since the set of projects which need to define these is unlikely to change over time (it's, like,airbyte-api
and a couple of others). Next, amortize the calls todocker
in the airbyte-docker plugin and get rid of calls to the system shell to evaluate env vars (also somewhat expensive).