airbyte-ci: upgrade to dagger 0.9.5 #33582

alafanechere · 2023-12-18T16:17:15Z

What

Closes #33429

How

Bump Dagger to 0.9.5 in: airbyte-ci, base-images, CAT
Update code where broken due to backward-incompatible APIs
Update GHA workflows to use the 0.9.5 engines on the infra

Manual testing

(manual testing should be done using the dev airbyte ci binary - this is why I used [skip ci] on all commits)

connectors --name=source-postgres --name=source-faker test
connectors --name=source-pokeapi publish --pre-release
format check all
format fix all (local)
connectors --name=source-faker bump_version (local)
nightly builds
airbyte-ci test

🚨 User Impact 🚨

None, if all manual tests succeed

Dependencies:

This should not be merged until the magicache set up is complete on the infra side.
Update: Distributed caching is currently enabled on our runners: waiting for Dagger team to confirm it's working correctly.

2 This should not be merged until CI can run on current branche's binary
Depends on #33519

Related infra PRs

vercel · 2023-12-18T16:17:24Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 5, 2024 11:03am

alafanechere · 2023-12-18T16:17:28Z

Current dependencies on/for this PR:

master
- PR airbyte-ci: upgrade to dagger 0.9.5 #33582 👈

This stack of pull requests is managed by Graphite.

github-actions · 2023-12-21T10:54:34Z

Warning

Soft code freeze is in effect until 2024-01-02. Please avoid merging to master. #freedom-and-responsibility

postamar

Does this mean we can follow this up with a PR which removes the dirty hack I put in that installs some secrets as regular env vars with --is-local is set so that the gradle integration test output wouldn't be all messed up?

postamar · 2024-01-04T18:04:39Z

airbyte-ci/connectors/pipelines/pyproject.toml

@@ -10,7 +10,7 @@ authors = ["Airbyte <contact@airbyte.io>"]

 [tool.poetry.dependencies]
 python = "~3.10"
-dagger-io = "^0.6.4"
+dagger-io = "==0.9.5"


Shouldn't you bump the major version counter of airbyte-ci as well?

postamar · 2024-01-04T18:07:59Z

airbyte-ci/connectors/pipelines/pipelines/dagger/actions/system/docker.py

    if docker_hub_username_secret and docker_hub_password_secret:
        # Docker login happens late because there's a cache buster in the docker login command.
        dockerd_container = docker_login(dockerd_container, docker_hub_username_secret, docker_hub_password_secret)
    return dockerd_container.with_exec(
        ["dockerd", "--log-level=error", f"--host=tcp://0.0.0.0:{DOCKER_HOST_PORT}", "--tls=false"], insecure_root_capabilities=True
-    )
+    ).as_service()


bnchrch

A few very small things but LGTM!

bnchrch · 2024-01-05T02:09:40Z

.github/actions/run-dagger-pipeline/action.yml

@@ -142,14 +145,14 @@ runs:
        export _EXPERIMENTAL_DAGGER_RUNNER_HOST="unix:///var/run/buildkit/buildkitd.sock"
        airbyte-ci --disable-dagger-run --is-ci --gha-workflow-run-id=${{ github.run_id }} ${{ inputs.subcommand }} ${{ inputs.options }}
      env:
-        _EXPERIMENTAL_DAGGER_CLOUD_TOKEN: "p.eyJ1IjogIjFiZjEwMmRjLWYyZmQtNDVhNi1iNzM1LTgxNzI1NGFkZDU2ZiIsICJpZCI6ICJlNjk3YzZiYy0yMDhiLTRlMTktODBjZC0yNjIyNGI3ZDBjMDEifQ.hT6eMOYt3KZgNoVGNYI3_v4CC-s19z8uQsBkGrBhU3k"


❤️ Nice to see this go to an input!

bnchrch · 2024-01-05T02:09:55Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/pipeline.py

@@ -94,11 +94,6 @@ async def run_connectors_pipelines(
    default_connectors_semaphore = anyio.Semaphore(concurrency)
    dagger_logs_output = sys.stderr if not dagger_logs_path else create_and_open_file(dagger_logs_path)
    async with dagger.Connection(Config(log_output=dagger_logs_output, execute_timeout=execute_timeout)) as dagger_client:
-
-        # HACK: This is to get a long running dockerd service to be shared across all the connectors pipelines
-        # Using the "normal" service binding leads to restart of dockerd during pipeline run that can cause corrupted docker state


bnchrch · 2024-01-05T02:11:13Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/pipeline.py

-                    )
-            # When the connectors pipelines are done, we can stop the dockerd service
-            tg_main.cancel_scope.cancel()
+        await dockerd_service.start()


💅 nit: Should/could we turn this to an async with

async with dockerd_service.start():

It'd be nice but I don't think we can as it's not returning a context manager

bnchrch · 2024-01-05T02:12:12Z

airbyte-ci/connectors/pipelines/pipelines/dagger/actions/connector/hooks.py

 from pipelines.airbyte_ci.connectors.context import ConnectorContext


+def get_sdk_version() -> str:


Not immediately clear what SDK its refering to when you call it.

Suggested change

def get_sdk_version() -> str:

def get_dagger_sdk_version() -> str:

alafanechere · 2024-01-05T11:41:29Z

Does this mean we can follow this up with a PR which removes the dirty hack I put in that installs some secrets as regular env vars with --is-local is set so that the gradle integration test output wouldn't be all messed up?

@postamar I think it's worth a try, if my not mistaken the issues you opened on Dagger repo are closed now .

alafanechere requested a review from a team December 18, 2023 16:17

alafanechere requested review from lazebnyi and oustynova as code owners December 18, 2023 16:17

alafanechere force-pushed the augustin/12-14-dagger_upgrade_to_0.9.4 branch from 282778e to 40df54a Compare December 18, 2023 18:42

alafanechere changed the title ~~airbyte-ci: upgrade to dagger 0.9.4~~ [do not merge] airbyte-ci: upgrade to dagger 0.9.4 Dec 18, 2023

alafanechere force-pushed the augustin/12-14-dagger_upgrade_to_0.9.4 branch 2 times, most recently from 2015dcb to bfdd6de Compare December 19, 2023 12:54

alafanechere mentioned this pull request Dec 19, 2023

airbyte-ci: upgrade to dagger 0.9.5 #33429

Closed

vercel bot deployed to Preview December 21, 2023 10:56 View deployment

alafanechere force-pushed the augustin/12-14-dagger_upgrade_to_0.9.4 branch from 402811b to 4c8a6a4 Compare January 3, 2024 13:01

alafanechere changed the title ~~[do not merge] airbyte-ci: upgrade to dagger 0.9.4~~ [do not merge] airbyte-ci: upgrade to dagger 0.9.5 Jan 3, 2024

alafanechere force-pushed the augustin/12-14-dagger_upgrade_to_0.9.4 branch 2 times, most recently from 08c193c to cc757ef Compare January 3, 2024 14:52

alafanechere changed the title ~~[do not merge] airbyte-ci: upgrade to dagger 0.9.5~~ airbyte-ci: upgrade to dagger 0.9.5 Jan 3, 2024

alafanechere force-pushed the augustin/12-14-dagger_upgrade_to_0.9.4 branch from cc757ef to 8de621f Compare January 3, 2024 16:42

vercel bot deployed to Preview January 4, 2024 07:35 View deployment

alafanechere force-pushed the augustin/12-14-dagger_upgrade_to_0.9.4 branch from 4e586dc to 83c228e Compare January 4, 2024 10:30

postamar approved these changes Jan 4, 2024

View reviewed changes

bnchrch approved these changes Jan 5, 2024

View reviewed changes

alafanechere added 4 commits January 5, 2024 11:50

dagger: upgrade to 0.9.4

f2fc345

fix typing

52a54a1

rename get_sdk_version to get_dagger_sdk_version

f3fe18b

bump packages versions

d0a4b5c

alafanechere force-pushed the augustin/12-14-dagger_upgrade_to_0.9.4 branch from c62d4fe to d0a4b5c Compare January 5, 2024 11:00

alafanechere merged commit 617075d into master Jan 5, 2024
23 checks passed

alafanechere deleted the augustin/12-14-dagger_upgrade_to_0.9.4 branch January 5, 2024 11:41

postamar mentioned this pull request Jan 5, 2024

airbyte-ci: remove connector secrets hack for --is-local #33972

Merged

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

airbyte-ci: upgrade to dagger 0.9.5 (airbytehq#33582)

cd1f1e6

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

airbyte-ci: upgrade to dagger 0.9.5 (airbytehq#33582)

f870e39

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

airbyte-ci: upgrade to dagger 0.9.5 (airbytehq#33582)

2168c3e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

airbyte-ci: upgrade to dagger 0.9.5 #33582

airbyte-ci: upgrade to dagger 0.9.5 #33582

alafanechere commented Dec 18, 2023 •

edited

Loading

vercel bot commented Dec 18, 2023 •

edited

Loading

alafanechere commented Dec 18, 2023

github-actions bot commented Dec 21, 2023

postamar left a comment

postamar Jan 4, 2024

postamar Jan 4, 2024

bnchrch left a comment

bnchrch Jan 5, 2024

bnchrch Jan 5, 2024

bnchrch Jan 5, 2024

alafanechere Jan 5, 2024

bnchrch Jan 5, 2024

alafanechere Jan 5, 2024

alafanechere commented Jan 5, 2024

		from pipelines.airbyte_ci.connectors.context import ConnectorContext


		def get_sdk_version() -> str:

	def get_sdk_version() -> str:
	def get_dagger_sdk_version() -> str:

airbyte-ci: upgrade to dagger 0.9.5 #33582

airbyte-ci: upgrade to dagger 0.9.5 #33582

Conversation

alafanechere commented Dec 18, 2023 • edited Loading

What

How

Manual testing

🚨 User Impact 🚨

Dependencies:

Related infra PRs

vercel bot commented Dec 18, 2023 • edited Loading

alafanechere commented Dec 18, 2023

github-actions bot commented Dec 21, 2023

postamar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnchrch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alafanechere commented Jan 5, 2024

alafanechere commented Dec 18, 2023 •

edited

Loading

vercel bot commented Dec 18, 2023 •

edited

Loading