airbyte-ci: disable gradle step caching on source-postgres

[alafanechere]

What

We want to disable the dagger/gradle caching on source-postgres gradle steps in order to collect unbiased CI performance metrics for source-postgres.
This change will be reverted once the project is over.

How

• Burst dagger layers in GradleTask with a CACHEBUSTER env var. The layers are bursted on each new pipeline, but not on each gradle task execution within the pipeline
• Create pipeline specific gradle-cache volume name to guarantee that the gradle remote cache volume (powered by dagger) is never used for source-postgres.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 12, 2023 2:23pm

[alafanechere]

Current dependencies on/for this PR:

• master
  • PR airbyte-ci: disable gradle step caching on source-postgres #31332 👈

This comment was auto-generated by Graphite.

[github-actions]

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage	71.72%	🍏

[airbyte-oss-build-runner]

source-postgres test report (commit 5ade428ff0) - ✅

⏲️ Total pipeline duration: 29mn27s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

[postamar]

I'd prefer a smaller and less risky change, which is going to make it easer to undo after Q4a.

[postamar]

Assuming that these volumes outlive the pipeline execution (which I have doubts about) isn't this dangerous? Won't this leave a bunch of garbage lying around in the dagger engine?

Irrespective of whether I'm right or not about cache volume persistence, it's simpler to reimplement this PR by modifying _get_gradle_command instead:

    def _get_gradle_command(self, task: str) -> List[str]:
        return sh_dash_c(
            [
                # The gradle command is chained in between a couple of rsyncs which load from- and store to the cache volume.
                # TODO: remove test after Q4a2023 (github issue #123456)
                f"(test '{self.context.connector.technical_name}' != 'source-postgres' && rsync -a --stats /root/gradle-cache/ /root/.gradle || true)",
                f"./gradlew {' '.join(self.DEFAULT_GRADLE_TASK_OPTIONS)} {task}",
                f"(test '{self.context.connector.technical_name}' != 'source-postgres' && rsync -a --stats /root/.gradle/ /root/gradle-cache || true)",
            ]
        )

I can't foresee us wanting to disable caching for anything other than source-postgres.

[alafanechere]

What we want to avoid to get reproducible metrics is the persistent (remote) caching which could potentially make other runs feed the cache and bias the results on master.
If we use a completely different cache volume while keeping rsync we get unbiased performance with intra-pipeline gradle caching which is working correctly today.

The volumes will outlive pipeline execution, but the dagger-engine on our infra is run on a K8S that is not supposed to idle forever. In other word we have disk renewal.

[postamar]

Ah, you're right. OK!

[postamar]

s/TBD/31332/