Skip to content

Commit

Permalink
airbyte-ci: docker login on dockerd container to avoid rate limit (#3…
Browse files Browse the repository at this point in the history 
…1718)
  • Loading branch information
@alafanechere
alafanechere committed Oct 23, 2023
1 parent cfcc70c commit f8ae6f3
Show file tree
Hide file tree
Showing 13 changed files with 98 additions and 93 deletions.
30 changes: 16 additions & 14 deletions airbyte-ci/connectors/pipelines/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -133,18 +133,21 @@ Available commands:
* `airbyte-ci connectors publish`: Publish a connector to Airbyte's DockerHub.

#### Options
| Option | Multiple | Default value | Description |
| -------------------------------------------------------------- | -------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--use-remote-secrets` | False | True | If True, connectors configuration will be pulled from Google Secret Manager. Requires the GCP_GSM_CREDENTIALS environment variable to be set with a service account with permission to read GSM secrets. If False the connector configuration will be read from the local connector `secrets` folder. |
| `--name` | True | | Select a specific connector for which the pipeline will run. Can be used multiple time to select multiple connectors. The expected name is the connector technical name. e.g. `source-pokeapi` |
| `--support-level` | True | | Select connectors with a specific support level: `community`, `certified`. Can be used multiple times to select multiple support levels. |
| `--metadata-query` | False | | Filter connectors by the `data` field in the metadata file using a [simpleeval](https://github.com/danthedeckie/simpleeval) query. e.g. 'data.ab_internal.ql == 200' |
| `--use-local-cdk` | False | False | Build with the airbyte-cdk from the local repository. " "This is useful for testing changes to the CDK. |
| `--language` | True | | Select connectors with a specific language: `python`, `low-code`, `java`. Can be used multiple times to select multiple languages. |
| `--modified` | False | False | Run the pipeline on only the modified connectors on the branch or previous commit (depends on the pipeline implementation). |
| `--concurrency` | False | 5 | Control the number of connector pipelines that can run in parallel. Useful to speed up pipelines or control their resource usage. |
| `--metadata-change-only/--not-metadata-change-only` | False | `--not-metadata-change-only` | Only run the pipeline on connectors with changes on their metadata.yaml file. |
| `--enable-dependency-scanning / --disable-dependency-scanning` | False | ` --disable-dependency-scanning` | When enabled the dependency scanning will be performed to detect the connectors to select according to a dependency change. |
| Option | Multiple | Default value | Mapped Environment Variable | Description |
| -------------------------------------------------------------- | -------- | -------------------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--use-remote-secrets` | False | True | | If True, connectors configuration will be pulled from Google Secret Manager. Requires the GCP_GSM_CREDENTIALS environment variable to be set with a service account with permission to read GSM secrets. If False the connector configuration will be read from the local connector `secrets` folder. |
| `--name` | True | | | Select a specific connector for which the pipeline will run. Can be used multiple times to select multiple connectors. The expected name is the connector technical name. e.g. `source-pokeapi` |
| `--support-level` | True | | | Select connectors with a specific support level: `community`, `certified`. Can be used multiple times to select multiple support levels. |
| `--metadata-query` | False | | | Filter connectors by the `data` field in the metadata file using a [simpleeval](https://github.com/danthedeckie/simpleeval) query. e.g. 'data.ab_internal.ql == 200' |
| `--use-local-cdk` | False | False | | Build with the airbyte-cdk from the local repository. " "This is useful for testing changes to the CDK. |
| `--language` | True | | | Select connectors with a specific language: `python`, `low-code`, `java`. Can be used multiple times to select multiple languages. |
| `--modified` | False | False | | Run the pipeline on only the modified connectors on the branch or previous commit (depends on the pipeline implementation). |
| `--concurrency` | False | 5 | | Control the number of connector pipelines that can run in parallel. Useful to speed up pipelines or control their resource usage. |
| `--metadata-change-only/--not-metadata-change-only` | False | `--not-metadata-change-only` | | Only run the pipeline on connectors with changes on their metadata.yaml file. |
| `--enable-dependency-scanning / --disable-dependency-scanning` | False | ` --disable-dependency-scanning` | | When enabled the dependency scanning will be performed to detect the connectors to select according to a dependency change. |
| `--docker-hub-username` | | | DOCKER_HUB_USERNAME | Your username to connect to DockerHub. Required for the publish subcommand. |
| `--docker-hub-password` | | | DOCKER_HUB_PASSWORD | Your password to connect to DockerHub. Required for the publish subcommand. |

### <a id="connectors-list-command"></a>`connectors list` command
Retrieve the list of connectors satisfying the provided filters.
Expand Down Expand Up @@ -293,8 +296,6 @@ Publish all connectors modified in the head commit: `airbyte-ci connectors --mod
| Option | Required | Default | Mapped environment variable | Description |
| ------------------------------------ | -------- | --------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--pre-release/--main-release` | False | `--pre-release` | | Whether to publish the pre-release or the main release version of a connector. Defaults to pre-release. For main release you have to set the credentials to interact with the GCS bucket. |
| `--docker-hub-username` | True | | `DOCKER_HUB_USERNAME` | Your username to connect to DockerHub. |
| `--docker-hub-password` | True | | `DOCKER_HUB_PASSWORD` | Your password to connect to DockerHub. |
| `--spec-cache-gcs-credentials` | False | | `SPEC_CACHE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting spec cache. |
| `--spec-cache-bucket-name` | False | | `SPEC_CACHE_BUCKET_NAME` | The name of the GCS bucket where specs will be cached. |
| `--metadata-service-gcs-credentials` | False | | `METADATA_SERVICE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting the metadata files. |
Expand Down Expand Up @@ -398,6 +399,7 @@ This command runs the Python tests for a airbyte-ci poetry package.
## Changelog
| Version | PR | Description |
| ------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| 2.2.5 | [#31718](https://github.com/airbytehq/airbyte/pull/31718) | Authenticate the sidecar docker daemon to DockerHub. |
| 2.2.4 | [#31535](https://github.com/airbytehq/airbyte/pull/31535) | Improve gradle caching when building java connectors. |
| 2.2.3 | [#31688](https://github.com/airbytehq/airbyte/pull/31688) | Fix failing `CheckBaseImageUse` step when not running on PR. |
| 2.2.2 | [#31659](https://github.com/airbytehq/airbyte/pull/31659) | Support builds on x86_64 platform |
Expand Down
18 changes: 18 additions & 0 deletions airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/commands.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,20 @@ def validate_environment(is_local: bool, use_remote_secrets: bool):
default=True,
type=bool,
)
@click.option(
"--docker-hub-username",
help="Your username to connect to DockerHub.",
type=click.STRING,
required=False,
envvar="DOCKER_HUB_USERNAME",
)
@click.option(
"--docker-hub-password",
help="Your password to connect to DockerHub.",
type=click.STRING,
required=False,
envvar="DOCKER_HUB_PASSWORD",
)
@click.pass_context
def connectors(
ctx: click.Context,
Expand All @@ -192,6 +206,8 @@ def connectors(
enable_dependency_scanning: bool,
use_local_cdk: bool,
enable_report_auto_open: bool,
docker_hub_username: str,
docker_hub_password: str,
):
"""Group all the connectors-ci command."""
validate_environment(ctx.obj["is_local"], use_remote_secrets)
Expand All @@ -202,6 +218,8 @@ def connectors(
ctx.obj["execute_timeout"] = execute_timeout
ctx.obj["use_local_cdk"] = use_local_cdk
ctx.obj["open_report_in_browser"] = enable_report_auto_open
ctx.obj["docker_hub_username"] = docker_hub_username
ctx.obj["docker_hub_password"] = docker_hub_password
ctx.obj["selected_connectors_with_modified_files"] = get_selected_connectors_with_modified_files(
names,
support_levels,
Expand Down
12 changes: 12 additions & 0 deletions airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/context.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,18 @@ def docker_image_tag(self) -> str:
def docker_image(self) -> str:
return f"{self.docker_repository}:{self.docker_image_tag}"

@property
def docker_hub_username_secret(self) -> Optional[Secret]:
if self.docker_hub_username is None:
return None
return self.dagger_client.set_secret("docker_hub_username", self.docker_hub_username)

@property
def docker_hub_password_secret(self) -> Optional[Secret]:
if self.docker_hub_password is None:
return None
return self.dagger_client.set_secret("docker_hub_password", self.docker_hub_password)

async def get_connector_dir(self, exclude=None, include=None) -> Directory:
"""Get the connector under test source code directory.
Expand Down
23 changes: 5 additions & 18 deletions ...ci/connectors/pipelines/pipelines/airbyte_ci/connectors/migrate_to_base_image/commands.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,36 +8,23 @@
from pipelines.airbyte_ci.connectors.migrate_to_base_image.pipeline import run_connector_migration_to_base_image_pipeline
from pipelines.airbyte_ci.connectors.pipeline import run_connectors_pipelines
from pipelines.cli.dagger_pipeline_command import DaggerPipelineCommand
from pipelines.helpers.utils import fail_if_missing_docker_hub_creds


@click.command(
cls=DaggerPipelineCommand,
short_help="Make the selected connectors use our base image: remove dockerfile, update metadata.yaml and update documentation.",
)
@click.argument("pull-request-number", type=str)
@click.option(
"--docker-hub-username",
help="Your username to connect to DockerHub to read the registries.",
type=click.STRING,
required=True,
envvar="DOCKER_HUB_USERNAME",
)
@click.option(
"--docker-hub-password",
help="Your password to connect to DockerHub to read the registries.",
type=click.STRING,
required=True,
envvar="DOCKER_HUB_PASSWORD",
)
@click.pass_context
def migrate_to_base_image(
ctx: click.Context,
pull_request_number: str,
docker_hub_username: str,
docker_hub_password: str,
) -> bool:
"""Bump a connector version: update metadata.yaml, changelog and delete legacy files."""

fail_if_missing_docker_hub_creds(ctx)

connectors_contexts = [
ConnectorContext(
pipeline_name=f"Upgrade base image versions of connector {connector.technical_name}",
Expand All @@ -56,8 +43,8 @@ def migrate_to_base_image(
ci_git_user=ctx.obj["ci_git_user"],
ci_github_access_token=ctx.obj["ci_github_access_token"],
open_report_in_browser=False,
docker_hub_username=docker_hub_username,
docker_hub_password=docker_hub_password,
docker_hub_username=ctx.obj.get("docker_hub_username"),
docker_hub_password=ctx.obj.get("docker_hub_password"),
s3_build_cache_access_key_id=ctx.obj.get("s3_build_cache_access_key_id"),
s3_build_cache_secret_key=ctx.obj.get("s3_build_cache_secret_key"),
)
Expand Down
5 changes: 4 additions & 1 deletion airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/pipeline.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,10 @@ async def run_connectors_pipelines(
# HACK: This is to get a long running dockerd service to be shared across all the connectors pipelines
# Using the "normal" service binding leads to restart of dockerd during pipeline run that can cause corrupted docker state
# See https://github.com/airbytehq/airbyte/issues/27233
dockerd_service = docker.with_global_dockerd_service(dagger_client)
docker_hub_username_secret = dagger_client.set_secret("DOCKER_HUB_USERNAME", contexts[0].docker_hub_username)
docker_hub_password_secret = dagger_client.set_secret("DOCKER_HUB_PASSWORD", contexts[0].docker_hub_password)

dockerd_service = docker.with_global_dockerd_service(dagger_client, docker_hub_username_secret, docker_hub_password_secret)
async with anyio.create_task_group() as tg_main:
tg_main.start_soon(dockerd_service.sync)
await ( # Wait for the docker service to be ready
Expand Down

0 comments on commit f8ae6f3

Please sign in to comment.