Skip to content

CI: Use ARC runners for container image build jobs #922

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 9, 2024
Merged

CI: Use ARC runners for container image build jobs #922

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dbf3508
CI: Run container image build job on ARC cluster
markgoddard Jan 23, 2024
05e1995
Add package repository credentials to container image build
markgoddard Feb 8, 2024
f8947a9
Don't verify Apt repo CA initially when using HTTPS in container build
markgoddard Feb 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions .github/workflows/stackhpc-all-in-one.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,8 +290,3 @@ jobs:
OS_APPLICATION_CREDENTIAL_ID: ${{ secrets.OS_APPLICATION_CREDENTIAL_ID }}
OS_APPLICATION_CREDENTIAL_SECRET: ${{ secrets.OS_APPLICATION_CREDENTIAL_SECRET }}
if: always()

- name: Prune Docker images over 1 week old
# May fail if another prune is running
run: docker image prune --all --force --filter until=168h || true
if: always()
66 changes: 22 additions & 44 deletions .github/workflows/stackhpc-container-image-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
generate-tag:
name: Generate container image tag
if: github.repository == 'stackhpc/stackhpc-kayobe-config'
runs-on: [self-hosted, stackhpc-kayobe-config-kolla-builder]
runs-on: ubuntu-latest
permissions: {}
outputs:
kolla_tag: ${{ steps.kolla_tag.outputs.kolla_tag }}
Expand Down Expand Up @@ -97,7 +97,7 @@ jobs:
container-image-build:
name: Build Kolla container images
if: github.repository == 'stackhpc/stackhpc-kayobe-config'
runs-on: [self-hosted, stackhpc-kayobe-config-kolla-builder]
runs-on: arc-skc-container-image-builder-runner
timeout-minutes: 720
permissions: {}
strategy:
Expand All @@ -106,7 +106,13 @@ jobs:
needs:
- generate-tag
steps:
- uses: actions/checkout@v4
- name: Install package dependencies
run: |
sudo apt update
sudo apt install -y build-essential git unzip nodejs python3-wheel python3-pip python3-venv

- name: Checkout
uses: actions/checkout@v4
with:
path: src/kayobe-config

Expand All @@ -117,33 +123,9 @@ jobs:
ref: refs/heads/stackhpc/${{ needs.generate-tag.outputs.openstack_release }}
path: src/kayobe

# FIXME: Failed in kolla-ansible : Ensure the latest version of pip is installed
- name: Install dependencies
- name: Make sure dockerd is running and test Docker
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this always required or just for testing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not required, but has been useful for checking that the dind configuration for the runner is working. It seems to run very quickly - it's a tiny image

run: |
sudo dnf -y install python3-virtualenv

- name: Setup networking
run: |
if ! ip l show breth1 >/dev/null 2>&1; then
sudo ip l add breth1 type bridge
fi
sudo ip l set breth1 up
if ! ip a show breth1 | grep 192.168.33.3/24; then
sudo ip a add 192.168.33.3/24 dev breth1
fi
if ! ip l show dummy1 >/dev/null 2>&1; then
sudo ip l add dummy1 type dummy
fi
sudo ip l set dummy1 up
sudo ip l set dummy1 master breth1

# FIXME: Without this workaround we see the following issue after the runner is power cycled:
# TASK [MichaelRigart.interfaces : RedHat | ensure network service is started and enabled] ***
# Unable to start service network: Job for network.service failed because the control process exited with error code.
# See \"systemctl status network.service\" and \"journalctl -xe\" for details.
- name: Kill dhclient (workaround)
run: |
(sudo killall dhclient || true) && sudo systemctl restart network
docker run --rm hello-world

- name: Install Kayobe
run: |
Expand All @@ -154,23 +136,19 @@ jobs:
pip install -U pip &&
pip install ../src/kayobe

- name: Bootstrap the control host
run: |
source venvs/kayobe/bin/activate &&
source src/kayobe-config/kayobe-env --environment ci-builder &&
kayobe control host bootstrap

- name: Configure the seed host
# Required for Docker registry login. Normally installed during host configure.
- name: Install Docker Python SDK
run: |
source venvs/kayobe/bin/activate &&
source src/kayobe-config/kayobe-env --environment ci-builder &&
kayobe seed host configure
env:
KAYOBE_VAULT_PASSWORD: ${{ secrets.KAYOBE_VAULT_PASSWORD }}
pip install --user docker

- name: Prune local Kolla container images over 1 week old
- name: Configure localhost as a seed
run: |
sudo docker image prune --all --force --filter until=168h --filter="label=kolla_version"
cat > src/kayobe-config/etc/kayobe/environments/ci-builder/inventory/hosts << EOF
# A 'seed' host used for building images.
# Use localhost for container image builds.
[seed]
localhost ansible_connection=local ansible_python_interpreter=/usr/bin/python3
EOF

- name: Build and push kolla overcloud images
run: |
Expand Down Expand Up @@ -203,7 +181,7 @@ jobs:

- name: Get built container images
run: |
sudo docker image ls --filter "reference=ark.stackhpc.com/stackhpc-dev/${{ matrix.distro }}-*:${{ needs.generate-tag.outputs.kolla_tag }}" > ${{ matrix.distro }}-container-images
docker image ls --filter "reference=ark.stackhpc.com/stackhpc-dev/${{ matrix.distro }}-*:${{ needs.generate-tag.outputs.kolla_tag }}" > ${{ matrix.distro }}-container-images

- name: Fail if no images have been built
run: if [ $(wc -l < ${{ matrix.distro }}-container-images) -le 1 ]; then exit 1; fi
Expand Down
25 changes: 25 additions & 0 deletions etc/kayobe/kolla.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,12 +317,26 @@ kolla_build_blocks:
sed -i -e '/\[{{ repo.tag }}\]/,/^\[/ s/^\(mirrorlist *=.*\)/#\1/g' \
-e '/\[{{ repo.tag }}\]/,/^\[/ s/^[# ]*\(baseurl *=.*\)/#\1/g' \
-e '/\[{{ repo.tag }}\]/,/^\[/ s/^[# ]*\(metalink *=.*\)/#\1/g' \
{% if stackhpc_repo_mirror_username is truthy %}
-e '/\[{{ repo.tag }}\]/,/^\[/ s|^\(name.*\)|\1\nusername={{ stackhpc_repo_mirror_username }}|' \
-e '/\[{{ repo.tag }}\]/,/^\[/ s|^\(name.*\)|\1\npassword={{ stackhpc_repo_mirror_password }}|' \
{% endif %}
-e '/\[{{ repo.tag }}\]/,/^\[/ s|^\(name.*\)|\1\nbaseurl={{ repo.url }}|' /etc/yum.repos.d/{{ repo.file }}{% if not loop.last %} && \
{% endif %}
{% endfor %}
{% else %}
RUN \
rm /etc/apt/sources.list && \
rm -f /etc/apt/auth.conf && \
{% if stackhpc_repo_mirror_url | urlsplit('scheme') == 'https' %}
{# We lack the ca-certificates package at this stage, so don't verify the CA #}
echo 'Acquire::https::Verify-Peer "false";' > /etc/apt/apt.conf.d/90no-verify-peer && \
{% endif %}
{% if stackhpc_repo_mirror_username is truthy %}
echo 'machine {{ stackhpc_repo_mirror_url }}' >> /etc/apt/auth.conf && \
echo 'login {{ stackhpc_repo_mirror_username }}' >> /etc/apt/auth.conf && \
echo 'password {{ stackhpc_repo_mirror_password }}' >> /etc/apt/auth.conf && \
{% endif %}
{% for repo in stackhpc_ubuntu_focal_base_repos %}
echo '{{ repo }}' >> /etc/apt/sources.list {% if not loop.last %} && \
{% endif %}
Expand All @@ -340,6 +354,10 @@ kolla_build_blocks:
sed -i -e '/\[{{ repo.tag }}\]/,/^\[/ s/^\(mirrorlist *=.*\)/#\1/g' \
-e '/\[{{ repo.tag }}\]/,/^\[/ s/^[# ]*\(baseurl *=.*\)/#\1/g' \
-e '/\[{{ repo.tag }}\]/,/^\[/ s/^[# ]*\(metalink *=.*\)/#\1/g' \
{% if stackhpc_repo_mirror_username is truthy %}
-e '/\[{{ repo.tag }}\]/,/^\[/ s|^\(name.*\)|\1\nusername={{ stackhpc_repo_mirror_username }}|' \
-e '/\[{{ repo.tag }}\]/,/^\[/ s|^\(name.*\)|\1\npassword={{ stackhpc_repo_mirror_password }}|' \
{% endif %}
-e '/\[{{ repo.tag }}\]/,/^\[/ s|^\(name.*\)|\1\nbaseurl={{ repo.url }}|' /etc/yum.repos.d/{{ repo.file }}{% if not loop.last %} &&{% endif %} \
{% endfor %}
{% endif %}
Expand All @@ -350,6 +368,13 @@ kolla_build_blocks:
{% endif %}
RUN \
rm /etc/apt/sources.list && \
rm -f /etc/apt/auth.conf && \
rm -f /etc/apt/apt.conf.d/90no-verify-peer && \
{% if stackhpc_repo_mirror_username is truthy %}
echo 'machine {{ stackhpc_repo_mirror_url }}' >> /etc/apt/auth.conf && \
echo 'login {{ stackhpc_repo_mirror_username }}' >> /etc/apt/auth.conf && \
echo 'password {{ stackhpc_repo_mirror_password }}' >> /etc/apt/auth.conf && \
{% endif %}
{% for repo in stackhpc_ubuntu_focal_repos %}
echo '{{ repo }}' >> /etc/apt/sources.list {% if not loop.last %} && \
{% endif %}
Expand Down