airbyte-ci: Pass env vars to poetry container in test command

[flash1293]

This PR allows to pass through environment variables of the host sytem to the container executing tests:

airbyte-ci test my_package --pass-env-var="GCM_CREDENTIALS"

Behavior:

• Multiple env vars can be passed in
• As they might be confidental, they are treated as secrets as far as dagger is concerned
• If they are not set on the host system, a descriptive error is thrown

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 18, 2024 11:35am

[sweep-ai]

Apply Sweep Rules to your PR?

• Apply: All new business logic should have corresponding unit tests.
• Apply: Refactor large functions to be more modular.
• Apply: Add docstrings to all functions and file headers.

This is an automated message generated by Sweep AI.

[flash1293]

@aaronsteers we can use this to pass the GCM credentials to the airbyte-lib test to download secrets for integration tests using external resources.

[sweep-ai]

Fixing PR: track the progress here.

I'm currently fixing this PR to address the following:

[Sweep GHA Fix] The GitHub Actions run failed with the following error logs:

[!CAUTION]

An error has occurred: Cmd('git') failed due to: exit code(128)
cmdline: git clone -v --branch=flash1293/pass-env-var -- https://:@github.com/airbytehq/airbyte.git /tmp/cache/repos/airbytehq/airbyte/base/flash1293--pass-env-var
stderr: 'fatal: could not create work tree dir '/tmp/cache/repos/airbytehq/airbyte/base/flash1293--pass-env-var': No space left on device
' (tracking ID: 1f88dff9e8)

[sweep-ai]

Fixing PR: track the progress here.

I'm currently fixing this PR to address the following:

[Sweep GHA Fix] The GitHub Actions run failed with the following error logs:

[!CAUTION]

An error has occurred: Cmd('git') failed due to: exit code(128)
cmdline: git clone -v --branch=flash1293/pass-env-var -- https://:@github.com/airbytehq/airbyte.git /tmp/cache/repos/airbytehq/airbyte/base/flash1293--pass-env-var
stderr: 'fatal: could not create work tree dir '/tmp/cache/repos/airbytehq/airbyte/base/flash1293--pass-env-var': No space left on device
' (tracking ID: e82a4b0b13)

[alafanechere]

Nice improvement!
I'm wondering what the more convenient:

1. Explicitly passing env var like in docker run with --env MY_ENV_VAR='my-value'
2. Passing already set env vars

1 has the benefit of not pre-setting the env var on your host
2 has the benefit of not passing secret as CLI input

Please bump the package version in pyproject.toml and README.md.

[alafanechere]

Should we set this at a more higher level during container declaration?
Let's say we want to override DOCKER_VERSION or if poetry install behavior depends on some env var value? The drawback of setting this earlier is that it'll clear downstream layers (re-running poetry install on env var changes).

[alafanechere]

Suggested change

	"--pass-env-var",
	"--pass-env-var", "-e", "passed_env_vars"

To have a shorter -e option and a context object key named with plural passed_env_vars

[aaronsteers]

Looks good for my part!