[Workflows] Re-write release-binaries workflow #89521

tstellar · 2024-04-20T23:35:55Z

This updates the release-binaries workflow so that the different build stages are split across multiple jobs. This saves money by reducing the time spent on the larger github runners and also makes it easier to debug, because now it's possible to build a smaller release package (with clang and lld) using only the free GitHub runners.

The workflow no longer uses the test-release.sh script but instead uses the Release.cmake cache. This gives the workflow more flexibility and ensures that the binary package will always be created even if the tests fail.

This idea to split the stages comes from the "LLVM Precommit CI through Github Actions" RFC:
https://discourse.llvm.org/t/rfc-llvm-precommit-ci-through-github-actions/76456

llvmbot · 2024-04-20T23:36:25Z

@llvm/pr-subscribers-github-workflow

Author: Tom Stellard (tstellar)

Changes

This updates the release-binaries workflow so that the different build stages are split across multiple jobs. This saves money by reducing the time spent on the larger github runners and also makes it easier to debug, because now it's possible to build a smaller release package (with clang and lld) using only the free GitHub runners.

The workflow no longer uses the test-release.sh script but instead uses the Release.cmake cache. This gives the workflow more flexibility and ensures that the binary package will always be created even if the tests fail.

This idea to split the stages comes from the "LLVM Precommit CI through Github Actions" RFC:
https://discourse.llvm.org/t/rfc-llvm-precommit-ci-through-github-actions/76456

Full diff: https://github.com/llvm/llvm-project/pull/89521.diff

2 Files Affected:

(modified) .github/workflows/release-binaries.yml (+192-66)
(modified) .github/workflows/set-release-binary-outputs.sh (-7)

diff --git a/.github/workflows/release-binaries.yml b/.github/workflows/release-binaries.yml
index 131ad3004f4577..5f864cb34b3fcc 100644
--- a/.github/workflows/release-binaries.yml
+++ b/.github/workflows/release-binaries.yml
@@ -38,9 +38,6 @@ jobs:
     if: github.repository == 'llvm/llvm-project'
     outputs:
       release-version: ${{ steps.vars.outputs.release-version }}
-      flags: ${{ steps.vars.outputs.flags }}
-      build-dir: ${{ steps.vars.outputs.build-dir }}
-      rc-flags: ${{ steps.vars.outputs.rc-flags }}
       ref: ${{ steps.vars.outputs.ref }}
       upload: ${{ steps.vars.outputs.upload }}
 
@@ -85,17 +82,11 @@ jobs:
         fi
         bash .github/workflows/set-release-binary-outputs.sh "$tag" "$upload"
 
-  # Try to get around the 6 hour timeout by first running a job to fill
-  # the build cache.
-  fill-cache:
-    name: "Fill Cache ${{ matrix.os }}"
+  build-stage1-linux:
+    name: "Build Stage 1 Linux"
     needs: prepare
-    runs-on: ${{ matrix.os }}
+    runs-on: ubuntu-22.04
     if: github.repository == 'llvm/llvm-project'
-    strategy:
-      matrix:
-        os:
-          - ubuntu-22.04
     steps:
     - name: Checkout LLVM
       uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
@@ -109,81 +100,216 @@ jobs:
       uses: hendrikmuhs/ccache-action@ca3acd2731eef11f1572ccb126356c2f9298d35e # v1.2.9
       with:
         max-size: 250M
-        key: sccache-${{ matrix.os }}-release
+        key: sccache-${{ runner.os }}-release
         variant: sccache
 
-    - name: Build Clang
+    - name: Build Stage 1 Clang
       run: |
-        cmake -G Ninja -C clang/cmake/caches/Release.cmake -DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache -DCMAKE_POSITION_INDEPENDENT_CODE=ON -S llvm -B build
-        ninja -v -C build clang
+        sudo chown $USER:$USER /mnt/
+        mkdir -p /mnt/build
+        cmake -G Ninja -C clang/cmake/caches/Release.cmake -DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache -DCMAKE_POSITION_INDEPENDENT_CODE=ON -S llvm -B /mnt/build
+        ninja -v -C /mnt/build
 
+    # We need to create an archive of the build directory, because it has too
+    # many files to upload.
+    - name: Package Build and Source Directories
+      run: |
+        tar -czf ${{ github.workspace }}/../llvm-project.tar.gz .
+        mv ../llvm-project.tar.gz ${{ github.workspace }}
+        tar -C /mnt/ -czf ${{ github.workspace }}/build.tar.gz build/
 
-  build-binaries:
-    name: ${{ matrix.target.triple }}
-    permissions:
-      contents: write # To upload assets to release.
+    - name: Upload Stage 1 Source
+      uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0
+      with:
+        name: stage1-source
+        path: ${{ github.workspace }}/llvm-project.tar.gz
+        retention-days: 2
+
+    - name: Upload Stage 1 Build Dir
+      uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0
+      with:
+        name: stage1-build
+        path: ${{ github.workspace }}/build.tar.gz
+        retention-days: 2
+
+  build-stage2-linux:
+    name: "Build Stage 2 Linux"
     needs:
       - prepare
-      - fill-cache
-    runs-on: ${{ matrix.target.runs-on }}
+      - build-stage1-linux
+    runs-on: ubuntu-22.04
     if: github.repository == 'llvm/llvm-project'
-    strategy:
-      fail-fast: false
-      matrix:
-        target:
-          - triple: x86_64-linux-gnu-ubuntu-22.04
-            os: ubuntu-22.04
-            runs-on: ubuntu-22.04-16x64
-            debian-build-deps: >
-              chrpath
-              gcc-multilib
-              ninja-build
-
     steps:
-    - name: Checkout LLVM
-      uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
+    - name: Install Ninja
+      uses: llvm/actions/install-ninja@22e9f909d35b50bd1181709564bfe816eaeaae81 # main
+
+    - name: Download Stage 1 Artifacts
+      uses: actions/download-artifact@6b208ae046db98c579e8a3aa621ab581ff575935 # v4.1.1
       with:
-        ref: ${{ needs.prepare.outputs.ref }}
-        path: ${{ needs.prepare.outputs.build-dir }}/llvm-project
+        pattern: stage1-*
+        merge-multiple: true
 
-    - name: Setup sccache
-      uses: hendrikmuhs/ccache-action@ca3acd2731eef11f1572ccb126356c2f9298d35e # v1.2.9
+    - name: Unpack Artifacts
+      run: |
+        tar -xzf llvm-project.tar.gz
+        rm llvm-project.tar.gz
+        sudo chown $USER:$USER /mnt/
+        tar -C /mnt -xzf build.tar.gz
+        rm build.tar.gz
+
+    - name: Build Stage 2
+      run: |
+        ninja -C /mnt/build stage2-instrumented
+
+    # We need to create an archive of the build directory, because it has too
+    # many files to upload.
+    - name: Save Build and Source Directories
+      run: |
+        tar -czf ${{ github.workspace }}/../llvm-project.tar.gz .
+        mv ../llvm-project.tar.gz ${{ github.workspace }}
+        tar -C /mnt/ -czf ${{ github.workspace }}/build.tar.gz build/
+
+    - name: Upload Stage 2 Source
+      uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0
       with:
-        max-size: 250M
-        key: sccache-${{ matrix.target.os }}-release
-        save: false
-        variant: sccache
+        name: stage2-source
+        path: ${{ github.workspace }}/llvm-project.tar.gz
+        retention-days: 2
+
+    - name: Upload Stage 2 Build Dir
+      uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0
+      with:
+        name: stage2-build
+        path: ${{ github.workspace }}/build.tar.gz
+        retention-days: 2
 
-    - name: Install Brew build dependencies
-      if: matrix.target.brew-build-deps != ''
-      run: brew install ${{ matrix.target.brew-build-deps }}
 
-    - name: Install Debian build dependencies
-      if: matrix.target.debian-build-deps != ''
-      run: sudo apt install ${{ matrix.target.debian-build-deps }}
+  build-stage3-linux:
+    name: "Build Stage 3 Linux"
+    needs:
+      - prepare
+      - build-stage2-linux
+    outputs:
+      filename: ${{ steps.package-info.outputs.release-filename }}
+    runs-on: ubuntu-22.04-16x64
+    if: github.repository == 'llvm/llvm-project'
+    steps:
+    - name: Install Ninja
+      uses: llvm/actions/install-ninja@22e9f909d35b50bd1181709564bfe816eaeaae81 # main
+
+    - name: 'Download artifact'
+      uses: actions/download-artifact@6b208ae046db98c579e8a3aa621ab581ff575935 # v4.1.1
+      with:
+        pattern: stage2-*
+        merge-multiple: true
 
-    - name: Set macOS build env variables
-      if: runner.os == 'macOS'
+    - name: Unpack Artifact
       run: |
-        echo "MACOSX_DEPLOYMENT_TARGET=10.9" >> "$GITHUB_ENV"
+        tar -xzf llvm-project.tar.gz
+        rm llvm-project.tar.gz
+        sudo chown $USER:$USER /mnt/
+        tar -C /mnt -xzf build.tar.gz
+        rm build.tar.gz
 
-    - name: Build and test release
+    - name: Build Release Package
       run: |
-        ${{ needs.prepare.outputs.build-dir }}/llvm-project/llvm/utils/release/test-release.sh \
-        ${{ needs.prepare.outputs.flags }} \
-        -triple ${{ matrix.target.triple }} \
-        -use-ninja \
-        -no-checkout \
-        -use-cmake-cache \
-        -no-test-suite \
-        -configure-flags "-DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache"
+        ninja -C /mnt/build stage2-package
 
-    - name: Upload binaries
-      if: ${{ always() && needs.prepare.outputs.upload == 'true' }}
+    - id: package-info
+      run: |
+        filename="LLVM-${{ needs.prepare.outputs.release-version }}-Linux.tar.Z"
+        echo "filename=$filename" >> $GITHUB_OUTPUT
+        echo "path=/mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/$filename" >> $GITHUB_OUTPUT
+
+    - uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0
+      if: always()
+      with:
+        name: release-binary
+        path: ${{ steps.package-info.outputs.path }}
+
+    # Clean up some build files to reduce size of artifact.
+    - name: Clean Up Build Directory
+      run: |
+        find /mnt/build -iname ${{ steps.package-info.outputs.filename }} -delete
+
+    # We need to create an archive of the build directory, because it has too
+    # many files to upload.
+    - name: Save Build and Source Directories
+      run: |
+        sudo apt-get update
+        sudo apt-get install pbzip2
+        tar -czf ${{ github.workspace }}/../llvm-project.tar.gz .
+        mv ../llvm-project.tar.gz ${{ github.workspace }}
+        # Use bzip2 for better compression since this file will be huge.
+        tar -C /mnt/ -c build/ | pbzip2 -c > ${{ github.workspace }}/build.tar.bz2
+
+    - name: Upload Stage 3 Source
+      uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0
+      with:
+        name: stage3-source
+        path: ${{ github.workspace }}/llvm-project.tar.gz
+        retention-days: 2
+
+    - name: Upload Stage 3 Build Dir
+      uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0
+      with:
+        name: stage3-build
+        path: ${{ github.workspace }}/build.tar.bz2
+        retention-days: 2
+
+  upload-release-binaries-linux:
+    name: "Upload Linux Release Binaries"
+    needs:
+      - prepare
+      - build-stage3-linux
+    if : ${{ needs.prepare.outputs.upload == 'true' }}
+    runs-on: ubuntu-22.04
+    permissions:
+      contents: write # For release uploads
+
+    steps:
+    - name: 'Download artifact'
+      uses: actions/download-artifact@6b208ae046db98c579e8a3aa621ab581ff575935 # v4.1.1
+      with:
+        name: release-binary
+
+    - name: Upload Release
       run: |
         sudo apt install python3-github
-        ${{ needs.prepare.outputs.build-dir }}/llvm-project/llvm/utils/release/github-upload-release.py \
+        ./llvm-project/llvm/utils/release/github-upload-release.py \
         --token ${{ github.token }} \
         --release ${{ needs.prepare.outputs.release-version }} \
         upload \
-        --files ${{ needs.prepare.outputs.build-dir }}/clang+llvm-${{ needs.prepare.outputs.release-version }}-${{ matrix.target.triple }}.tar.xz
+        --files ${{ needs.build-stage3-linux.outputs.release-filename }}
+
+
+  test-stage3-linux:
+    name: "Test Stage 3 Linux"
+    needs:
+      - prepare
+      - build-stage3-linux
+    runs-on: ubuntu-22.04
+    if: github.repository == 'llvm/llvm-project'
+    steps:
+    - name: Install Ninja
+      uses: llvm/actions/install-ninja@22e9f909d35b50bd1181709564bfe816eaeaae81 # main
+
+    - name: 'Download artifact'
+      uses: actions/download-artifact@6b208ae046db98c579e8a3aa621ab581ff575935 # v4.1.1
+      with:
+        pattern: stage3-*
+        merge-multiple: true
+
+    - name: Unpack Artifact
+      run: |
+        sudo apt-get update
+        sudo apt-get install bzip2
+        tar -xzf llvm-project.tar.gz
+        rm llvm-project.tar.gz
+        sudo chown $USER:$USER /mnt/
+        tar -C /mnt -xjf build.tar.bz2
+        rm build.tar.bz2
+
+    - name: Run Tests
+      run: |
+        ninja -C /mnt/build stage2-check-all
diff --git a/.github/workflows/set-release-binary-outputs.sh b/.github/workflows/set-release-binary-outputs.sh
index 59470cf83ba755..14d0798364e914 100644
--- a/.github/workflows/set-release-binary-outputs.sh
+++ b/.github/workflows/set-release-binary-outputs.sh
@@ -15,10 +15,8 @@ if echo $tag | grep -e '^[0-9a-f]\+$'; then
   # This is a plain commit.
   # TODO: Don't hardcode this.
   release_version="18"
-  build_dir="$tag"
   upload='false'
   ref="$tag"
-  flags="-git-ref $tag -test-asserts"
 
 else
 
@@ -30,12 +28,7 @@ else
   fi
   release_version=`echo "$tag" | sed 's/llvmorg-//g'`
   release=`echo "$release_version" | sed 's/-.*//g'`
-  build_dir=`echo "$release_version" | sed 's,^[^-]\+,final,' | sed 's,[^-]\+-rc\(.\+\),rc\1,'`
-  rc_flags=`echo "$release_version" | sed 's,^[^-]\+,-final,' | sed 's,[^-]\+-rc\(.\+\),-rc \1 -test-asserts,' | sed 's,--,-,'`
-  flags="-release $release $rc_flags"
 fi
 echo "release-version=$release_version" >> $GITHUB_OUTPUT
-echo "build-dir=$build_dir" >> $GITHUB_OUTPUT
-echo "flags=$flags" >> $GITHUB_OUTPUT
 echo "upload=$upload" >> $GITHUB_OUTPUT
 echo "ref=$tag" >> $GITHUB_OUTPUT

github-actions · 2024-04-20T23:37:18Z

⚠️ We detected that you are using a GitHub private e-mail address to contribute to the repo.
Please turn off Keep my email addresses private setting in your account.
See LLVM Discourse for more information.

This updates the release-binaries workflow so that the different build stages are split across multiple jobs. This saves money by reducing the time spent on the larger github runners and also makes it easier to debug, because now it's possible to build a smaller release package (with clang and lld) using only the free GitHub runners. The workflow no longer uses the test-release.sh script but instead uses the Release.cmake cache. This gives the workflow more flexibility and ensures that the binary package will always be created even if the tests fail. This idea to split the stages comes from the "LLVM Precommit CI through Github Actions" RFC: https://discourse.llvm.org/t/rfc-llvm-precommit-ci-through-github-actions/76456

tstellar · 2024-04-20T23:39:53Z

Here is an example of a workflow run using a slightly older version of this patch:
https://github.com/tstellar/llvm-project/actions/runs/8765505041

tru · 2024-04-24T13:19:45Z

Here is an example of a workflow run using a slightly older version of this patch: https://github.com/tstellar/llvm-project/actions/runs/8765505041

Do you have a run that's using the latest version somewhere as well?

tru

LGTM, but I would like to see the latest run.

tru · 2024-04-24T13:20:29Z

.github/workflows/release-binaries.yml

-        ninja -v -C build clang
+        sudo chown $USER:$USER /mnt/
+        mkdir -p /mnt/build
+        cmake -G Ninja -C clang/cmake/caches/Release.cmake -DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache -DCMAKE_POSITION_INDEPENDENT_CODE=ON -S llvm -B /mnt/build


do we need to pass the position independent option here? I thought on was the default?

I've removed this. I think it's better to add this to the cache file anyway.

tru · 2024-04-24T13:22:57Z

.github/workflows/release-binaries.yml

+      run: |
+        find /mnt/build -iname ${{ steps.package-info.outputs.filename }} -delete
+
+    # We need to create an archive of the build directory, because it has too


Did you try to use zstd for this? Just curious, I have found that when I need to pack and unpack a large temporary tree where speed is more important than smallest size it is really useful.

I switched to using zstd for packaging just this large binary. Do you think we should use zstd everywhere?

The benefit of zstd is that it's very fast at packing/unpacking, but has the same compression as .gz/bz2 more or less. XZ has better compression but it's slower. So I think zstd makes sense for temporary/transitionary packages between ci jobs etc. But XZ is better for long term storage.

I've updated the job to use zstd for all stages now.

tstellar · 2024-04-25T22:16:16Z

This is the result from using the latest changes in this PR: https://github.com/tstellar/llvm-project/actions/runs/8832350283

boomanaiden154

Overall LGTM.

Do you have numbers on how much time this ends up saving on the larger runners?

Hopefully soonish (might be a while) we have sufficient self-hosted runners where we can run these jobs so we don't have to add the complexity of splitting. Nice to see that this idea seems to work out in practice though!

tstellar · 2024-05-04T20:32:52Z

@boomanaiden154 I think this will save about an hour of run-time on the paid builders. I may try to incorporate some of the strategies from #82093 to see if I can get this to run completely on free runners.

This updates the release-binaries workflow so that the different build stages are split across multiple jobs. This saves money by reducing the time spent on the larger github runners and also makes it easier to debug, because now it's possible to build a smaller release package (with clang and lld) using only the free GitHub runners. The workflow no longer uses the test-release.sh script but instead uses the Release.cmake cache. This gives the workflow more flexibility and ensures that the binary package will always be created even if the tests fail. This idea to split the stages comes from the "LLVM Precommit CI through Github Actions" RFC: https://discourse.llvm.org/t/rfc-llvm-precommit-ci-through-github-actions/76456 (cherry picked from commit abac984)

tstellar requested review from tru and boomanaiden154 April 20, 2024 23:35

llvmbot added the github:workflow label Apr 20, 2024

tstellar force-pushed the release-binaries-rewrite branch from 85aef93 to 0564af5 Compare April 20, 2024 23:39

tru reviewed Apr 24, 2024

View reviewed changes

tstellar mentioned this pull request Apr 25, 2024

[CMake][Release] Refactor cache file and use two stages for non-PGO builds #89812

Merged

tstellar added 2 commits April 25, 2024 15:12

Review fixups

fea99b5

Remove CMAKE_POSITION_INDEPENDENT_CODE

ee8dbdc

tru approved these changes Apr 26, 2024

View reviewed changes

tstellar added 2 commits April 27, 2024 14:43

Use zst for everything

8341348

Fix path to release tarball

7cf7a9b

boomanaiden154 approved these changes Apr 30, 2024

View reviewed changes

tstellar merged commit abac984 into llvm:main May 4, 2024
4 checks passed

aemerson mentioned this pull request May 9, 2024

release/18.x: [AArc64][GlobalISel] Fix legalizer assert for G_INSERT_VECTOR_ELT - manual merge #91672

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Workflows] Re-write release-binaries workflow #89521

[Workflows] Re-write release-binaries workflow #89521

tstellar commented Apr 20, 2024

llvmbot commented Apr 20, 2024

github-actions bot commented Apr 20, 2024

tstellar commented Apr 20, 2024

tru commented Apr 24, 2024

tru left a comment

tru Apr 24, 2024

tstellar Apr 25, 2024

tru Apr 24, 2024

tstellar Apr 25, 2024

tru Apr 26, 2024

tstellar Apr 27, 2024

tstellar commented Apr 25, 2024

boomanaiden154 left a comment

tstellar commented May 4, 2024

[Workflows] Re-write release-binaries workflow #89521

[Workflows] Re-write release-binaries workflow #89521

Conversation

tstellar commented Apr 20, 2024

llvmbot commented Apr 20, 2024

github-actions bot commented Apr 20, 2024

tstellar commented Apr 20, 2024

tru commented Apr 24, 2024

tru left a comment

Choose a reason for hiding this comment

tru Apr 24, 2024

Choose a reason for hiding this comment

tstellar Apr 25, 2024

Choose a reason for hiding this comment

tru Apr 24, 2024

Choose a reason for hiding this comment

tstellar Apr 25, 2024

Choose a reason for hiding this comment

tru Apr 26, 2024

Choose a reason for hiding this comment

tstellar Apr 27, 2024

Choose a reason for hiding this comment

tstellar commented Apr 25, 2024

boomanaiden154 left a comment

Choose a reason for hiding this comment

tstellar commented May 4, 2024