Skip to content

Commit

Permalink
add auto-commit capability to build base images GH workflow
Browse files Browse the repository at this point in the history
This commit adds the ability to have automatic commits being pushed for
the base images github workflow. This simplifies the build process of the
base images as it will no longer require developers to wait and re-push
changes.

Resume of the changes:
- Re-write documentation with the new process
- Moved scripts to update tags in the git tree to the respective
  directories of the images. This is to prevent accidental or malicious
  changes by third-parties in the script files that could compromise the
  build process. If the files are modified, it will require an approval
  from the cilium/build team before the GH action is executed.
- Changed CODEOWNERS file to request for a review from the
  cilium/github-sec team as the scripts are directly used in the GH
  workflow.
- For future reference, this GH auto-committer workflow is using a token
  created by the machine-learning-apps/actions-app-token GH action and
  that GH action is using the auto-committer app [1] created exclusively
  for this purpose.

[1] https://github.com/organizations/cilium/settings/apps/auto-committer/permissions

Signed-off-by: André Martins <andre@cilium.io>
  • Loading branch information
aanm committed Nov 17, 2022
1 parent 2046ce1 commit c5a7787
Show file tree
Hide file tree
Showing 7 changed files with 120 additions and 70 deletions.
43 changes: 43 additions & 0 deletions .github/workflows/build-images-base.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ jobs:
persist-credentials: false
ref: ${{ github.event.pull_request.head.sha }}

- name: Set-up git
run: |
git config user.name "Cilium Imagebot"
git config user.email "noreply@cilium.io"
- name: Generating image tag for Cilium-Runtime
id: runtime-tag
run: |
Expand Down Expand Up @@ -112,6 +117,12 @@ jobs:
path: image-digest
retention-days: 1

- name: Update Runtime Image
if: ${{ steps.cilium-runtime-tag-in-repositories.outputs.exists == 'false' }}
run: |
images/runtime/update-cilium-runtime-image.sh "quay.io/${{ env.QUAY_ORGANIZATION_DEV }}/cilium-runtime:${{ steps.runtime-tag.outputs.tag }}@${{ steps.docker_build_release_runtime.outputs.digest }}"
git commit -sam "images: update cilium-{runtime,builder}"
- name: Generating image tag for Cilium-Builder
id: builder-tag
run: |
Expand Down Expand Up @@ -172,6 +183,38 @@ jobs:
path: image-digest
retention-days: 1

- name: Update Runtime and Builder Images
if: ${{ steps.cilium-builder-tag-in-repositories.outputs.exists == 'false' || steps.cilium-runtime-tag-in-repositories.outputs.exists == 'false' }}
run: |
images/runtime/update-cilium-runtime-image.sh "quay.io/${{ env.QUAY_ORGANIZATION_DEV }}/cilium-runtime:${{ steps.runtime-tag.outputs.tag }}@${{ steps.docker_build_release_runtime.outputs.digest }}"
images/builder/update-cilium-builder-image.sh "quay.io/${{ env.QUAY_ORGANIZATION_DEV }}/cilium-builder:${{ steps.builder-tag.outputs.tag }}@${{ steps.docker_build_release_builder.outputs.digest }}"
- name: Commit changes by amending previous commit
# Run this step in case we have committed the cilium-runtime changes before
if: ${{ steps.cilium-runtime-tag-in-repositories.outputs.exists == 'false' }}
run: |
git commit --amend -sam "images: update cilium-{runtime,builder}"
- name: Commit changes
# Run this step in case we have NOT committed the cilium-runtime changes before
if: ${{ steps.cilium-runtime-tag-in-repositories.outputs.exists != 'false' && steps.cilium-builder-tag-in-repositories.outputs.exists == 'false' }}
run: |
git commit -sam "images: update cilium-{runtime,builder}"
- name: Get token
if: ${{ steps.cilium-builder-tag-in-repositories.outputs.exists == 'false' || steps.cilium-runtime-tag-in-repositories.outputs.exists == 'false' }}
id: get_token
uses: cilium/actions-app-token@350a14155dd9be79227f081310f4d77fdf792e76
with:
APP_PEM: ${{ secrets.AUTO_COMMITTER_PEM }}
APP_ID: ${{ secrets.AUTO_COMMITTER_APP_ID }}

- name: Push changes into PR
if: ${{ steps.cilium-runtime-tag-in-repositories.outputs.exists == 'false' || steps.cilium-builder-tag-in-repositories.outputs.exists == 'false' }}
run: |
git diff HEAD^
git push https://x-access-token:${{ steps.get_token.outputs.app_token }}@github.com/${{ env.QUAY_ORGANIZATION }}/cilium.git HEAD:${{ github.event.pull_request.head.ref }}
image-digests:
name: Display Digests
runs-on: ubuntu-20.04
Expand Down
2 changes: 2 additions & 0 deletions CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,8 @@ Makefile* @cilium/build
/images @cilium/build
/images/builder/install-protoc.sh @cilium/sig-hubble-api
/images/builder/install-protoplugins.sh @cilium/sig-hubble-api
/images/builder/update-cilium-builder-image.sh @cilium/github-sec
/images/runtime/update-cilium-runtime-image.sh @cilium/github-sec
/install/kubernetes/ @cilium/sig-k8s @cilium/helm
/install/kubernetes/cilium/templates/hubble* @cilium/sig-k8s @cilium/helm @cilium/sig-hubble
jenkinsfiles @cilium/ci-structure
Expand Down
37 changes: 3 additions & 34 deletions Documentation/contributing/development/images.rst
Original file line number Diff line number Diff line change
Expand Up @@ -211,14 +211,6 @@ If you only wish to update the packages in these images, then you can manually
update the ``FORCE_BUILD`` variable in ``images/runtime/Dockerfile`` to have a
different value and then proceed with the steps below.

#. cilium-builder depends on cilium-runtime so one needs to update
cilium-runtime first. Steps 4 and 7 will fetch the digest of the image built
by GitHub actions.

.. code-block:: shell-session
$ make -C images/ update-runtime-image
#. Commit your changes and create a PR in cilium/cilium.

.. code-block:: shell-session
Expand All @@ -229,32 +221,9 @@ different value and then proceed with the steps below.
to approve the build that was created by GitHub Actions `here <https://github.com/cilium/cilium/actions?query=workflow:%22Base+Image+Release+Build%22>`__.
Note that at this step cilium-builder build failure is expected since we have yet to update the runtime digest.

#. Wait for cilium-runtime build to complete. Only after the image is available run:

.. code-block:: shell-session
$ make -C images/ update-runtime-image update-builder-image
#. Commit your changes and re-push to the PR in cilium/cilium.

.. code-block:: shell-session
$ git commit --amend -sa
#. Ping one of the members of `team/build <https://github.com/orgs/cilium/teams/build/members>`__
to approve the build that was created by GitHub Actions `here <https://github.com/cilium/cilium/actions?query=workflow:%22Base+Image+Release+Build%22>`__.

#. Wait for the build to complete. Only after the image is available run:

.. code-block:: shell-session
$ make -C images/ update-runtime-image update-builder-image
#. Commit your changes and re-push to the PR in cilium/cilium.

.. code-block:: shell-session
$ git commit --amend -sa
#. Wait for build to complete. The build will automatically generate one commit
and push it to your branch with all the necessary changes across files in the
repository. Once this is done the CI can be executed.

#. Update the versions of the images that are pulled into the CI VMs.

Expand Down
28 changes: 28 additions & 0 deletions images/builder/update-cilium-builder-image.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/bash

# Copyright Authors of Cilium
# SPDX-License-Identifier: Apache-2.0

set -o xtrace
set -o errexit
set -o pipefail
set -o nounset

image_full=${1}
root_dir="$(git rev-parse --show-toplevel)"

cd "${root_dir}"

image="quay.io/cilium/cilium-builder"

# shellcheck disable=SC2207
used_by=($(git grep -l CILIUM_BUILDER_IMAGE= images/*/Dockerfile) "test/k8s/manifests/demo-customcalls.yaml" "api/v1/Makefile")

for i in "${used_by[@]}" ; do
sed -E "s#(CILIUM_BUILDER_IMAGE=|image: )${image}:.*\$#\1${image_full}#" "${i}" > "${i}.sedtmp" && mv "${i}.sedtmp" "${i}"
done

do_check="${CHECK:-false}"
if [ "${do_check}" = "true" ] ; then
git diff --exit-code "${used_by[@]}"
fi
42 changes: 42 additions & 0 deletions images/runtime/update-cilium-runtime-image.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/bin/bash

# Copyright Authors of Cilium
# SPDX-License-Identifier: Apache-2.0

set -o xtrace
set -o errexit
set -o pipefail
set -o nounset

image_full=${1}
root_dir="$(git rev-parse --show-toplevel)"

cd "${root_dir}"

image="quay.io/cilium/cilium-runtime"

# shellcheck disable=SC2207
used_by=($(git grep -l CILIUM_RUNTIME_IMAGE= images/*/Dockerfile))

for i in "${used_by[@]}" ; do
sed -E "s#((CILIUM_RUNTIME|BASE)_IMAGE=)${image}:.*\$#\1${image_full}#" "${i}" > "${i}.sedtmp" && mv "${i}.sedtmp" "${i}"
done

# shellcheck disable=SC2207
jenkins_used_by=($(git grep -l "${image}:" jenkinsfiles/))

for i in "${jenkins_used_by[@]}" ; do
sed -E "s#\"${image}:.*\"#\"${image_full}\"#" "${i}" > "${i}.sedtmp" && mv "${i}.sedtmp" "${i}"
done

# shellcheck disable=SC2207
github_used_by=($(git grep -l "${image}:" .github/workflows/))

for i in "${github_used_by[@]}" ; do
sed -E "s#${image}:.*#${image_full}#" "${i}" > "${i}.sedtmp" && mv "${i}.sedtmp" "${i}"
done

do_check="${CHECK:-false}"
if [ "${do_check}" = "true" ] ; then
git diff --exit-code "${used_by[@]}"
fi
12 changes: 1 addition & 11 deletions images/scripts/update-cilium-builder-image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,4 @@ if [ -n "${sha256}" ]; then
image_full="${image_full}@${sha256}"
fi

# shellcheck disable=SC2207
used_by=($(git grep -l CILIUM_BUILDER_IMAGE= images/*/Dockerfile) "test/k8s/manifests/demo-customcalls.yaml" "api/v1/Makefile")

for i in "${used_by[@]}" ; do
sed -E "s#(CILIUM_BUILDER_IMAGE=|image: )${image}:.*\$#\1${image_full}#" "${i}" > "${i}.sedtmp" && mv "${i}.sedtmp" "${i}"
done

do_check="${CHECK:-false}"
if [ "${do_check}" = "true" ] ; then
git diff --exit-code "${used_by[@]}"
fi
"${script_dir}/../builder/update-cilium-builder-image.sh" "${image_full}"
26 changes: 1 addition & 25 deletions images/scripts/update-cilium-runtime-image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,28 +24,4 @@ if [ -n "${sha256}" ]; then
image_full="${image_full}@${sha256}"
fi

# shellcheck disable=SC2207
used_by=($(git grep -l CILIUM_RUNTIME_IMAGE= images/*/Dockerfile))

for i in "${used_by[@]}" ; do
sed -E "s#((CILIUM_RUNTIME|BASE)_IMAGE=)${image}:.*\$#\1${image_full}#" "${i}" > "${i}.sedtmp" && mv "${i}.sedtmp" "${i}"
done

# shellcheck disable=SC2207
jenkins_used_by=($(git grep -l "${image}:" jenkinsfiles/))

for i in "${jenkins_used_by[@]}" ; do
sed -E "s#\"${image}:.*\"#\"${image_full}\"#" "${i}" > "${i}.sedtmp" && mv "${i}.sedtmp" "${i}"
done

# shellcheck disable=SC2207
github_used_by=($(git grep -l "${image}:" .github/workflows/))

for i in "${github_used_by[@]}" ; do
sed -E "s#${image}:.*#${image_full}#" "${i}" > "${i}.sedtmp" && mv "${i}.sedtmp" "${i}"
done

do_check="${CHECK:-false}"
if [ "${do_check}" = "true" ] ; then
git diff --exit-code "${used_by[@]}"
fi
"${script_dir}/../runtime/update-cilium-runtime-image.sh" "${image_full}"

0 comments on commit c5a7787

Please sign in to comment.