[workflows] Build a container for running CI on github actions

[tstellar]

Using a container will allow us to have similar testing environments on both the GitHub hosted runners and the self-hosted runners.

[llvmbot]

@llvm/pr-subscribers-github-workflow

Author: Tom Stellard (tstellar)

Changes

Using a container will allow us to have similar testing environments on both the GitHub hosted runners and the self-hosted runners.

---

Full diff: https://github.com/llvm/llvm-project/pull/75286.diff

2 Files Affected:

• (added) .github/workflows/build-ci-container.yml (+51)
• (added) .github/workflows/containers/github-action-ci/Dockerfile (+26)

diff --git a/.github/workflows/build-ci-container.yml b/.github/workflows/build-ci-container.yml
new file mode 100644
index 00000000000000..9d2d18f7679734
--- /dev/null
+++ b/.github/workflows/build-ci-container.yml
@@ -0,0 +1,51 @@
+
+name: Build CI Container
+
+permissions:
+  contents: read
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - .github/workflows/build-ci-container.yml
+      - '.github/workflows/containers/github-action-ci/**'
+  pull_request:
+    branches:
+      - main
+    paths:
+      - .github/workflows/build-ci-container.yml
+      - '.github/workflows/containers/github-action-ci/**'
+
+jobs:
+  build-ci-container:
+    if: github.repository_owner == 'llvm'
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+    steps:
+      - name: Write Variables
+        id: vars
+        run: |
+          echo "container-name=ghcr.io/$GITHUB_REPOSITORY_OWNER/ci-ubuntu-22.04" >> $GITHUB_OUTPUT
+
+      - name: Checkout LLVM
+        uses: actions/checkout@v4
+
+      - name: Build Container
+        working-directory: ./.github/workflows/containers/github-action-ci/
+        run: |
+          # TODO: Ideally, we would give each container build a unique name,
+          # but we have limited space in the container registry, so we can't
+          # do this until we implement some kind of garbage collection for
+          # old containers.
+          podman build -t ${{ steps.vars.outputs.container-name }} .
+
+      - name: Push Container
+        if: github.event_name == 'push'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          podman login -u ${{ github.actor }} -p $GITHUB_TOKEN ghcr.io
+          podman push ${{ steps.vars.outputs.container-name }}
diff --git a/.github/workflows/containers/github-action-ci/Dockerfile b/.github/workflows/containers/github-action-ci/Dockerfile
new file mode 100644
index 00000000000000..5bde8e5e6b6aa2
--- /dev/null
+++ b/.github/workflows/containers/github-action-ci/Dockerfile
@@ -0,0 +1,26 @@
+FROM docker.io/library/ubuntu:22.04
+
+# swig is needed for lldb docs.
+
+# abi-dumper, autoconf, and pkg-config are required for the abi-dump workflow.
+# abi-compliance-checker is needed for the abi-compare workflow.
+# python3-setuptools and python3-psutil are required for the release-lit task.
+# Need to install curl for hendrikmuhs/ccache-action
+# Need nodejs for some of the GitHub actions.
+RUN apt-get update && \
+    apt-get install -y \
+    abi-compliance-checker \
+    abi-dumper \
+    autoconf \
+    cmake \
+    curl \
+    doxygen \
+    graphviz \
+    nodejs \
+    ninja-build \
+    pkg-config \
+    python3-github \
+    python3-psutil \
+    python3-setuptools \
+    swig \
+    texlive-font-utils

[boomanaiden154]

Minor comments.

I'm assuming the plan is to migrate the key existing workflows to use the container image after this lands? We might also want to add a couple more dependencies or sub-containers (that use this one as their base image) for specific workflows as some of them (like the code formatting action) install packages through apt-get (or could).

Installing python packages into a container would be nice as well for some of the jobs and would shave off a bit of setup time, but that's already mostly negated by the caching.

For building LLVM/running tests, we'll probably need to make some changes to take more explicit control of the toolchain that we're using. For a build-once run many times container, I think a ThinLTO+PGO+BOLTed clang based toolchain would make a lot of sense, but that is irrelevant for this patch.

[tstellar]

Minor comments.

I'm assuming the plan is to migrate the key existing workflows to use the container image after this lands? We might also want to add a couple more dependencies or sub-containers (that use this one as their base image) for specific workflows as some of them (like the code formatting action) install packages through apt-get (or could).

I thought about creating sub-containers, but I wasn't sure if all the extra complexity was actually worth it. What's the advantage of having separate containers for different types of jobs?

Installing python packages into a container would be nice as well for some of the jobs and would shave off a bit of setup time, but that's already mostly negated by the caching.

I would like to install the python packages too, I wasn't sure the best way to implement this. Should I copy all the requirements.txt files into the container directory and then add them to the container?

For building LLVM/running tests, we'll probably need to make some changes to take more explicit control of the toolchain that we're using. For a build-once run many times container, I think a ThinLTO+PGO+BOLTed clang based toolchain would make a lot of sense, but that is irrelevant for this patch.

This is a good idea. I've also thought about creating a container that has a pre-filled ccache cache to help speed up builds.

[boomanaiden154]

I thought about creating sub-containers, but I wasn't sure if all the extra complexity was actually worth it. What's the advantage of having separate containers for different types of jobs?

The main advantage I can think of is that it keeps the image size smaller as only the dependencies needed for the job are actually pulled in. For python packages, it also allows us to install everything at the system level in the container rather than having to worry about maintaining multiple virtual environments/trying to consolidate python dependencies.

I would like to install the python packages too, I wasn't sure the best way to implement this. Should I copy all the requirements.txt files into the container directory and then add them to the container?

I would try and keep each set of dependencies in a separate container. Otherwise the best way to do it (that I can think of) is to consolidate into a single requirements.txt/pipenv/something for all the CI jobs. Ideally we want to install the exact transitive closure of the dependencies used for development. Installing all the requirements.txt wouldn't guarantee this currently.

This is a good idea. I've also thought about creating a container that has a pre-filled ccache cache to help speed up builds.

Shouldn't Github's actions/cache (or some other caching mechanism) automatically download the most recent cache even if the image changes? We don't want to be rebuilding container images that frequently, and I think the pre-filled ccache would quickly become out of date (pretty much as soon as someone touches a major header).

[tstellar]

@boomanaiden154 A few updates based on your suggestions:

• Rather than having one container that can be used for every job, I created a container that only works with the llvm-project-tests workflows (and probably the release-binaries workflow too). I'm still not convinced that having different containers for each job is the right approach, but these workflow are the only jobs that actually need a container right now, because I want to move them to the self-hosted runner. The rest of the jobs are fine on the GitHub runners, so they don't really need to use containers.
• I installed the most recent clang release (with binaries from the release page) into the container and added its install directory to PATH. I'm planning on enabling PGO/llvm bolt optimizations for the release builds in the next cycle, so using the release binaries will eventually give us a highly optimized clang for the container as you suggested.
• I added unique tags for each new container build.

TODOs:

• Choose a better container name?
• Add workflow_dispatch.

[boomanaiden154]

Rather than having one container that can be used for every job, I created a container that only works with the llvm-project-tests workflows (and probably the release-binaries workflow too). I'm still not convinced that having different containers for each job is the right approach, but these workflow are the only jobs that actually need a container right now, because I want to move them to the self-hosted runner. The rest of the jobs are fine on the GitHub runners, so they don't really need to use containers.

That works. My point was mainly for jobs with extensive python dependencies, but we can look at that later, and that's also only if we want to move those jobs to containers given that they are cheap enough to run on the github hosted runners.

I installed the most recent clang release (with binaries from the release page) into the container and added its install directory to PATH. I'm planning on enabling PGO/llvm bolt optimizations for the release builds in the next cycle, so using the release binaries will eventually give us a highly optimized clang for the container as you suggested.

Sounds good.

I added unique tags for each new container build.

Awesome!

Everything in this patch pretty much LGTM. Just minor nits and I don't want to keep anything blocked for future work.