feat(ci): add CPU architecture suffixes to the Docker build cache #4826

youtalk · 2024-06-05T21:10:07Z

Description

This PR adds CPU architecture suffixes to the tag name of the Docker build cache which introduced by #4730.
Currently, the CI of the build-main for the x86_64 architecture and the build-main-self-hosted for the aarch64 architecture are running daily. Since the build cache itself is not platform-compatible, it seems that the effect of the build cache is lost each time.

The Docker build cache has been enabled, resulting in a speedup of about 10 minutes. However, since colcon build, which takes up 2 hours of build time, is not being cached and is being re-executed, significant acceleration has not been achieved. This is currently being investigated in a separate PR.

Tests performed

docker-build-and-push-main
- first https://github.com/autowarefoundation/autoware/actions/runs/9391390328/attempts/1
- second https://github.com/autowarefoundation/autoware/actions/runs/9391390328
docker-build-and-push-main-self-hosted
- first https://github.com/autowarefoundation/autoware/actions/runs/9391396527/attempts/1
- second https://github.com/autowarefoundation/autoware/actions/runs/9391396527 (ongoing)

Effects on system behavior

Not applicable.

Interface changes

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

I've confirmed the contribution guidelines.
The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

The PR follows the pull request guidelines.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

.github/workflows/build-main-self-hosted.yaml

Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

mitsudome-r

LGTM!

…towarefoundation#4826) * add arch prefix Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * rename Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * use lib_dir Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * change suffix Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

…towarefoundation#4826) (#32) * add arch prefix * rename * use lib_dir * change suffix --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

* feat(ci): add CPU architecture suffixes to the Docker build cache (autowarefoundation#4826) * add arch prefix Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * rename Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * use lib_dir Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * change suffix Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * use buildjet to self-hosted Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

* chore(docker): separate `rosdep install` and `colcon build` RUN commands (autowarefoundation#4719) * refactor(ci): reuse `load-env.yaml` to load env (autowarefoundation#4734) * reuse load-env.yaml to load env * update base_image Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * fix base_image Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * refactor(docker): introduce `src-imported` stage (autowarefoundation#4712) * introduce src-imported stage Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * empty Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * generate install package lists * append || true Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * cat packages Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * chore(docker): separate `rosdep install` and `colcon build` RUN commands (autowarefoundation#4719) * wip Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * wip Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * Update Dockerfile * Update Dockerfile * install pacmod * rename * Revert "install pacmod" This reverts commit ebff10f. * install pacmod3 * register pamod3 repos * ignore hadolint * revert * run set-dev-env.sh * pip install * copy * rosdep update --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * feat(docker): install minimum packages in `base` stage (autowarefoundation#4751) * install ssh only and copy only once Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * install gosu Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * fix dir Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * revert ansible Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * chore: add discussion templates for general category (autowarefoundation#4764) Signed-off-by: Ryohsuke Mitsudome <ryohsuke.mitsudome@tier4.jp> * feat(ci,docker): change `vcs import` to `COPY src` (autowarefoundation#4738) * introduce src-imported stage Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * empty Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * generate install package lists * append || true Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * cat packages Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * chore(docker): separate `rosdep install` and `colcon build` RUN commands (autowarefoundation#4719) * wip Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * wip Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * Update Dockerfile * Update Dockerfile * install pacmod * rename * Revert "install pacmod" This reverts commit ebff10f. * install pacmod3 * register pamod3 repos * ignore hadolint * revert * run set-dev-env.sh * pip install * copy * rosdep update * vcs import outside dockerfile Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * refine order and remove /autoware/src at last * fix hadolint * remove tar files Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * shell: bash Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * feat(docker,ansible): download artifacts at runtime (autowarefoundation#4771) * feat(ci): enable Docker build cache to 5 times faster container build (autowarefoundation#4730) * introduce src-imported stage Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * empty Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * generate install package lists * append || true Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * cat packages Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * chore(docker): separate `rosdep install` and `colcon build` RUN commands (autowarefoundation#4719) * wip Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * wip Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * Update Dockerfile * Update Dockerfile * install pacmod * rename * Revert "install pacmod" This reverts commit ebff10f. * install pacmod3 * register pamod3 repos * ignore hadolint * revert * run set-dev-env.sh * pip install * copy * rosdep update * vcs import outside dockerfile Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * enable docker build cache Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * empty Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * add no-cuda/cuda suffix Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * remove --download-artifacts Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * remove --download-artifacts Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * add --data-path option Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * wip Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * run download_artifacts Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * remove todo Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * revert run Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * feat(docker): cache `ccache` to the Docker build cache (autowarefoundation#4772) * fix: add download url of yolox with semantic segmentation header (autowarefoundation#4012) Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: add error handling for held packages during rmw installation (autowarefoundation#4781) * chore: add error handling for held packages during apt installation Signed-off-by: Naophis <nao1288stusj@gmail.com> * feat: Add check for held ROS packages during installation Signed-off-by: Naophis <nao1288stusj@gmail.com> --------- Signed-off-by: Naophis <nao1288stusj@gmail.com> * refactor(docker,ci): rename `autoware-openadk` to `autoware` and `adkit` to `openadkit` (autowarefoundation#4785) * rename adkit to openadkit Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * rename autoware-openadkit to autoware Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * fix hcl path Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * feat(ci): disable `Upload Artifacts` on `docker-build-and-push` action (autowarefoundation#4789) * remove upload artifact Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * remove build and push Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * feat(docker): cache apt archives to the Docker build cache (autowarefoundation#4778) * mount apt cache Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * reorder Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * remove upload artifact Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * remove build and push Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * chore: update code owners (autowarefoundation#4797) Signed-off-by: M. Fatih Cırıt <mfc@leodrive.ai> * ci(pre-commit-ansible): autoupdate (autowarefoundation#4676) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> * ci: pre-commit autoupdate (autowarefoundation#4670) [pre-commit.ci] pre-commit autoupdate updates: - [github.com/igorshubovych/markdownlint-cli: v0.39.0 → v0.41.0](igorshubovych/markdownlint-cli@v0.39.0...v0.41.0) - [github.com/AleksaC/hadolint-py: v2.12.1-beta → v2.12.1b3](AleksaC/hadolint-py@v2.12.1-beta...v2.12.1b3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * ci(pre-commit): autoupdate (autowarefoundation#4662) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> * ci(pre-commit-optional): autoupdate (autowarefoundation#4766) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> * fix(ci): run `vsc import` on `build-main` workflows (autowarefoundation#4799) Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * feat(docker): cache `ccache` to the Docker build cache (autowarefoundation#4800) cache ccache again Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * chore: hold check of plotjuggler-ros package and display warning if held (autowarefoundation#4805) * chore: Hold check of plotjuggler-ros package and display warning if held Signed-off-by: Naophis <nao1288stusj@gmail.com> * chore: fix condition for checking held ROS package in dev_tools role Signed-off-by: Naophis <nao1288stusj@gmail.com> * style(pre-commit): autofix * chore: Update dev_tools role to fix condition for checking held ROS package Signed-off-by: Naophis <nao1288stusj@gmail.com> --------- Signed-off-by: Naophis <nao1288stusj@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat(docker): revert "cache apt archives to the Docker build cache" (autowarefoundation#4793) Revert "feat(docker): cache apt archives to the Docker build cache (autowarefoundation#4778)" Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * chore(ci): add a workflow for DCO check to avoid DCO bot system failure (autowarefoundation#4825) Signed-off-by: Ryohsuke Mitsudome <ryohsuke.mitsudome@tier4.jp> * feat(ci): add CPU architecture suffixes to the Docker build cache (autowarefoundation#4826) * add arch prefix Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * rename Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * use lib_dir Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * change suffix Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * feat(simulator.repos): use autoware_msgs_support branch for scenario_simulator (autowarefoundation#4823) Signed-off-by: Ryohsuke Mitsudome <ryohsuke.mitsudome@tier4.jp> Co-authored-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * feat(ansible): add lidar_transfusion artifacts (autowarefoundation#4835) * refactor(docker): fix colcon `--cmake-args` (autowarefoundation#4828) fix colcon cmake-args Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * refactor(ci): use platform arg to replace raw string (autowarefoundation#4844) use platform arg to replace raw string Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * refactor(ci): install `vcstool` and run `vcs import` on `docker-build-and-push.yaml` (autowarefoundation#4845) * refactor docker-build-and-push Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> * fix shell Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> Co-authored-by: oguzkaganozt <oguzkaganozt@gmail.com> * fix platform Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> --------- Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp> Signed-off-by: Ryohsuke Mitsudome <ryohsuke.mitsudome@tier4.jp> Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> Signed-off-by: Naophis <nao1288stusj@gmail.com> Signed-off-by: M. Fatih Cırıt <mfc@leodrive.ai> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ryohsuke Mitsudome <43976834+mitsudome-r@users.noreply.github.com> Co-authored-by: badai nguyen <94814556+badai-nguyen@users.noreply.github.com> Co-authored-by: Naophis <nao1288stusj@gmail.com> Co-authored-by: M. Fatih Cırıt <xmfcx@users.noreply.github.com> Co-authored-by: awf-autoware-bot[bot] <94889083+awf-autoware-bot[bot]@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Amadeusz Szymko <amadeuszszymko@gmail.com> Co-authored-by: oguzkaganozt <oguzkaganozt@gmail.com>

youtalk added 2 commits June 6, 2024 05:56

add arch prefix

fee0f76

Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

rename

002372e

Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

youtalk self-assigned this Jun 5, 2024

use lib_dir

65b988b

Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

youtalk changed the title ~~feat(ci): add platform suffix to the Docker build cache~~ feat(ci): add CPU archtecture suffixes to the Docker build cache Jun 5, 2024

youtalk changed the title ~~feat(ci): add CPU archtecture suffixes to the Docker build cache~~ feat(ci): add CPU architecture suffixes to the Docker build cache Jun 5, 2024

youtalk marked this pull request as ready for review June 6, 2024 04:36

youtalk requested a review from oguzkaganozt as a code owner June 6, 2024 04:36

youtalk requested review from xmfcx and mitsudome-r June 6, 2024 04:36

mitsudome-r reviewed Jun 6, 2024

View reviewed changes

.github/workflows/build-main-self-hosted.yaml Outdated Show resolved Hide resolved

youtalk mentioned this pull request Jun 6, 2024

feat(docker,ci): mount .ccache by bind mount instead of cache mount #4829

Closed

6 tasks

change suffix

29e5193

Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

youtalk requested a review from mitsudome-r June 6, 2024 10:01

mitsudome-r approved these changes Jun 6, 2024

View reviewed changes

youtalk enabled auto-merge (squash) June 6, 2024 10:31

youtalk merged commit c73aaf0 into main Jun 6, 2024
14 checks passed

youtalk deleted the upstream-buildcache-platform branch June 6, 2024 13:26

tier4-autoware-public-bot bot mentioned this pull request Jun 7, 2024

chore: sync upstream tier4/autoware#5

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ci): add CPU architecture suffixes to the Docker build cache #4826

feat(ci): add CPU architecture suffixes to the Docker build cache #4826

youtalk commented Jun 5, 2024 •

edited

mitsudome-r left a comment

feat(ci): add CPU architecture suffixes to the Docker build cache #4826

feat(ci): add CPU architecture suffixes to the Docker build cache #4826

Conversation

youtalk commented Jun 5, 2024 • edited

Description

Tests performed

Effects on system behavior

Interface changes

Pre-review checklist for the PR author

In-review checklist for the PR reviewers

Post-review checklist for the PR author

mitsudome-r left a comment

Choose a reason for hiding this comment

youtalk commented Jun 5, 2024 •

edited