MINOR: Enable Gradle Remote Build Cache

[nicktelford]

We enable the remote build cache, hosted by the ASF Gradle Enterprise instance.

We only cache tasks during builds on trunk, to ensure that pushes to PRs always re-run all Tasks needed by that PR.

PRs will read from the build cache, enabling tasks for modules that haven't been changed in the PR to be read from the cache.

This should ensure that changes "leaf" modules (e.g. connect, streams, etc.) don't run the entire test suite from dependent modules (i.e. core, clients, storage, etc.), which can be very costly.

[nicktelford]

I don't have access to the ASF Gradle Enterprise, so a committer will need to verify that this works as intended.

[divijvaidya]

Hey @nicktelford , you don't need committer permissions to look at ASF gradle. Try using this link: https://ge.apache.org, it should be publicly accessible.

For this PR, here's the build - https://ge.apache.org/s/bdwopqcoxwge2 which shows Build Cache as On which is unlike this build on trunk https://ge.apache.org/s/ywmt7vaghr62u where Build Cache is Off.

I am not sure how useful this will be since CI for trunk may build on different boxes but it's definitely worth keeping it on.

@ijuma is our gradle expert in the community. I will wait to hear his thoughts regarding this.

[nicktelford]

Hi @divijvaidya , this enables the remote build cache, which allows different machines to share the cached output of tasks.

Thanks for the links to the builds, I haven't had a chance to look at them yet, but the thing that requires access to ge would be too simulate a trunk CI run and push to the cache, which I won't be able to do locally.

[ijuma]

Could we do a test with a new branch (not trunk) where we validate the behavior before we roll it out to everyone else?

[ijuma]

I guess one way would be to enable the cache only for CI & trunk by default at first. Then we could manually enable the cache locally and for one PR to test the behavior. If it all looks good, we would then enable the cache for everyone (but leave pushes for trunk && CI only).

[nicktelford]

@ijuma I've temporarily reconfigured it to enable the cache for this branch (gradle-remote-build-cache) instead of trunk, so we can test that it works as expected using the CI builds on this PR.

Pushing to the cache from these builds should be fine, since there are no code changes, so the task outputs should be the same as in trunk anyway.

[nicktelford]

Looks like Jenkins doesn't have permission (by default at least) to write entries to the ASF Gradle Enterprise Build Cache:

Could not store entry 744bd685bdd1d149ec93934b276b7698 in remote build cache: Storing entry at 'https://ge.apache.org/cache/744bd685bdd1d149ec93934b276b7698' response status 403: Forbidden

It might require some explicit authentication.

[divijvaidya]

apache/beam#27015 is how Apache Beam enabled this. Seems like we need to request a build node from Apache Infra and change our jenkins to pick up creds similar to how beam did it (from env vars which are deployed by infra). I have created a ticket to talk to Infra at https://issues.apache.org/jira/browse/INFRA-25336

[clayburn]

Looks like Jenkins doesn't have permission (by default at least) to write entries to the ASF Gradle Enterprise Build Cache:

This is correct. The reason for this is that we do not currently intend for projects to use the built-in cache node (which is what this PR configures). If used this way, the node would be shared by all projects, which is not desirable because we would not want project A to inadvertently read a cache entry from project B. In the future, we will support such a use case without that risk, but for today, the correct option is to stand up a remote cache node specifically for Kafka, as requested in INFRA-25336

[github-actions]

This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch)

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.