Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

airbyte-ci: add /var/lib/docker cachemount #32114

Merged
merged 1 commit into from
Nov 3, 2023
Merged

airbyte-ci: add /var/lib/docker cachemount #32114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion airbyte-ci/connectors/pipelines/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,7 +408,8 @@ This command runs the Python tests for a airbyte-ci poetry package.
## Changelog
| Version | PR | Description |
| ------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| 2.5.3 | [#32090](https://github.com/airbytehq/airbyte/pull/32090) | Do not cache `docker login`. |
| 2.5.5 | [#32114](https://github.com/airbytehq/airbyte/pull/32114) | Create cache mount for `/var/lib/docker` to store images in `dind` context. |
| 2.5.4 | [#32090](https://github.com/airbytehq/airbyte/pull/32090) | Do not cache `docker login`. |
| 2.5.3 | [#31974](https://github.com/airbytehq/airbyte/pull/31974) | Fix latest CDK install and pip cache mount on connector install. |
| 2.5.2 | [#31871](https://github.com/airbytehq/airbyte/pull/31871) | Deactivate PR comments, add HTML report links to the PR status when its ready. |
| 2.5.1 | [#31774](https://github.com/airbytehq/airbyte/pull/31774) | Add a docker configuration check on `airbyte-ci` startup. |
Expand Down
1 change: 1 addition & 0 deletions airbyte-ci/connectors/pipelines/pipelines/consts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
DOCKER_HOST_NAME = "global-docker-host"
DOCKER_HOST_PORT = 2375
DOCKER_TMP_VOLUME_NAME = "shared-tmp"
DOCKER_VAR_LIB_VOLUME_NAME = "docker-cache"
REPO = git.Repo(search_parent_directories=True)
REPO_PATH = REPO.working_tree_dir
STATIC_REPORT_PREFIX = "airbyte-ci"
Expand Down
6 changes: 4 additions & 2 deletions airbyte-ci/connectors/pipelines/pipelines/dagger/actions/system/docker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from dagger import Client, Container, File, Secret
from pipelines import consts
from pipelines.airbyte_ci.connectors.context import ConnectorContext, PipelineContext
from pipelines.consts import DOCKER_HOST_NAME, DOCKER_HOST_PORT, DOCKER_TMP_VOLUME_NAME
from pipelines.consts import DOCKER_HOST_NAME, DOCKER_HOST_PORT, DOCKER_TMP_VOLUME_NAME, DOCKER_VAR_LIB_VOLUME_NAME
from pipelines.helpers.utils import sh_dash_c


Expand All @@ -26,7 +26,8 @@ def with_global_dockerd_service(
Container: The container running dockerd as a service
"""
dockerd_container = (
dagger_client.container().from_(consts.DOCKER_DIND_IMAGE)
dagger_client.container()
.from_(consts.DOCKER_DIND_IMAGE)
# We set this env var because we need to use a non-default zombie reaper setting.
# The reason for this is that by default it will want to set its parent process ID to 1 when reaping.
# This won't be possible because of container-ception: dind is running inside the dagger engine.
Expand All @@ -49,6 +50,7 @@ def with_global_dockerd_service(
# Expose the docker host port.
.with_exposed_port(DOCKER_HOST_PORT)
# Mount the docker cache volumes.
.with_mounted_cache("/var/lib/docker", dagger_client.cache_volume(DOCKER_VAR_LIB_VOLUME_NAME))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth a try.

.with_mounted_cache("/tmp", dagger_client.cache_volume(DOCKER_TMP_VOLUME_NAME))
)
if docker_hub_username_secret and docker_hub_password_secret:
Expand Down
2 changes: 1 addition & 1 deletion airbyte-ci/connectors/pipelines/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "pipelines"
version = "2.5.4"
version = "2.5.5"
description = "Packaged maintained by the connector operations team to perform CI for connectors' pipelines"
authors = ["Airbyte <contact@airbyte.io>"]

Expand Down
Loading