Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EPIC] Connectors CI/CD Improvements (Q2B) #27310

Closed
alafanechere opened this issue Jun 13, 2023 · 3 comments
Closed

[EPIC] Connectors CI/CD Improvements (Q2B) #27310

alafanechere opened this issue Jun 13, 2023 · 3 comments

Comments

@alafanechere
Copy link
Contributor

alafanechere commented Jun 13, 2023

Definition of done:

Nice DX

  • The pipelines should be stable and usable by developers flawlessly

Speed tracking

  • We can monitor connectors test duration on a daily basis
  • We can compare these durations to the legacy test pipeline

Caching

  • Repeated operation without code change should be cached whenever possible: unit tests, qa checks, format check, metadata validation.

Auto changes

New pipelines are ready to commit automatic changes on a branch to:

  • Format code
  • Bump connector version
  • Change log update in connector doc
@alafanechere
Copy link
Contributor Author

alafanechere commented Jun 20, 2023

Weekly project update:

Done

  • We can consider the original Connector CI/CD to production epic [EPIC] - Connectors CI/CD Pipeline in Production  #24403 done! Full CI/CD for connectors is now a reality: we have automated connector testing on PR + passing tests are required to merge a PR
  • We implemented code auto formatting for Python and Java connectors

Coming next (on the improvements epic #27310):

  • Stabilize nightly/weekly builds: We occasionally face workflow failures. Dagger team is actively investigating root causes (see our support project)
  • Improve UX for Test debugging: Dagger logs are a bit hard to digest for developers.
  • Improve and investigate caching.
  • Monitor connector test speed gain.
  • Infrastructure changes for better cost control and tracking.
  • Implement automatic version bump and changeling entry generation

@alafanechere
Copy link
Contributor Author

Weekly project update

Wins:

  • We drastically improved the logging / debugging experience of test pipelines. An HTML test report is now generated and logs are less noisy with meaningful messages. Log example / Report example
  • Conor revamped the infrastructure for better node lifecycle management with Carpenter: no more idle nodes, significant cost reduction.
  • The scheduled nightly builds ran successfully without requiring a manual retrigger.
  • We can now track connector test performance in this dashboard. The takeaway metric is that we’ve reduced the test speed by a third compared to /test so far.
  • We were able to identify improvements in our pipeline code to maximize caching and boost performance! PR

What got in the way:

  • The auto formatter led to unexpected code change. We disabled it.
  • After the latest infra change some jobs wait a very long time for a runner (~3H). We’re investigating this. Some Karpenter / ARC fine tuning needs to happen.

What’s in for this week:

  • Infra fine tuning to get a good balance of node availability vs test cold start duration
  • Better monitor resource usage of Dagger engines to improve instance sizing
  • Mainline the caching improvements to assess performance gains
  • Fix glitches in the CI UX

@alafanechere
Copy link
Contributor Author

Closing this epic in favor of airbytehq/airbyte-internal-issues#1707
Issues were migrated to the new one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant