[EPIC] Connectors CI/CD Improvements (Q2B)

[alafanechere]

Definition of done:

Nice DX

• The pipelines should be stable and usable by developers flawlessly

Speed tracking

• We can monitor connectors test duration on a daily basis
• We can compare these durations to the legacy test pipeline

Caching

• Repeated operation without code change should be cached whenever possible: unit tests, qa checks, format check, metadata validation.

Auto changes

New pipelines are ready to commit automatic changes on a branch to:

• Format code
• Bump connector version
• Change log update in connector doc

[alafanechere]

Weekly project update:

Done

• We can consider the original Connector CI/CD to production epic [EPIC] - Connectors CI/CD Pipeline in Production  #24403 done! Full CI/CD for connectors is now a reality: we have automated connector testing on PR + passing tests are required to merge a PR
• We implemented code auto formatting for Python and Java connectors

Coming next (on the improvements epic #27310):

• Stabilize nightly/weekly builds: We occasionally face workflow failures. Dagger team is actively investigating root causes (see our support project)
• Improve UX for Test debugging: Dagger logs are a bit hard to digest for developers.
• Improve and investigate caching.
• Monitor connector test speed gain.
• Infrastructure changes for better cost control and tracking.
• Implement automatic version bump and changeling entry generation

[alafanechere]

Weekly project update

Wins:

• We drastically improved the logging / debugging experience of test pipelines. An HTML test report is now generated and logs are less noisy with meaningful messages. Log example / Report example
• Conor revamped the infrastructure for better node lifecycle management with Carpenter: no more idle nodes, significant cost reduction.
• The scheduled nightly builds ran successfully without requiring a manual retrigger.
• We can now track connector test performance in this dashboard. The takeaway metric is that we’ve reduced the test speed by a third compared to /test so far.
• We were able to identify improvements in our pipeline code to maximize caching and boost performance! PR

What got in the way:

• The auto formatter led to unexpected code change. We disabled it.
• After the latest infra change some jobs wait a very long time for a runner (~3H). We’re investigating this. Some Karpenter / ARC fine tuning needs to happen.

What’s in for this week:

• Infra fine tuning to get a good balance of node availability vs test cold start duration
• Better monitor resource usage of Dagger engines to improve instance sizing
• Mainline the caching improvements to assess performance gains
• Fix glitches in the CI UX

[alafanechere]

Closing this epic in favor of airbytehq/airbyte-internal-issues#1707
Issues were migrated to the new one.