Skip to content

ci: unfreeze the build caches and build the bindings centrally #619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andiwand merged 3 commits into from
Jul 26, 2026
Merged

ci: unfreeze the build caches and build the bindings centrally #619

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3d6cba5
ci: stop freezing the conan and ccache caches
andiwand Jul 26, 2026
bd85f22
ci: build the bindings in the central build job
andiwand Jul 26, 2026
ad2a2cc
ci: drop the dead NDK step from build-test-downstream
andiwand Jul 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 77 additions & 18 deletions .github/workflows/build_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,18 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true

# Cache keys, in short: `actions/cache` never overwrites an existing key, so a
# key that stays the same forever is written exactly once and then frozen —
# every later run restores that first snapshot and rebuilds whatever is missing
# from source. Both caches therefore vary their key:
# * conan — keyed on what determines its content (recipes + profile), so it is
# rewritten precisely when the dependencies change.
# * ccache — keyed per run and restored via prefix, so it keeps growing. Only
# the default branch writes it; branch-scoped entries are invisible to other
# branches and would just churn the 10 GB repo quota.
# `CACHE_FLAVOR` separates jobs that resolve a *different* dependency set for the
# same profile — without it they overwrite each other and both keep rebuilding.
# The `*_KEY_SUFFIX` variables stay as a manual "throw it all away" knob.
env:
CCACHE_DIR: ${{ github.workspace }}/.ccache
CCACHE_MAXSIZE: 1G
Expand All @@ -14,16 +26,22 @@ env:
CONAN_KEY_SUFFIX: r1

jobs:
# Builds the core once per profile and, where `bindings` is set, the JNI and
# Python bindings out of the same configure. They used to live in their own
# workflows, which meant compiling the core and resolving its dependencies
# three times over for the same two profiles.
build:
runs-on: ${{ matrix.os }}
env:
CACHE_FLAVOR: main${{ matrix.bindings && '-bindings' || '' }}
strategy:
fail-fast: false
matrix:
include:
- { os: ubuntu-24.04, build_profile: ubuntu-24.04-clang-18, host_profile: ubuntu-24.04-clang-18 }
- { os: ubuntu-24.04, build_profile: ubuntu-24.04-clang-18, host_profile: ubuntu-24.04-clang-18, bindings: true }
- { os: ubuntu-24.04, build_profile: ubuntu-24.04-gcc-14, host_profile: ubuntu-24.04-gcc-14 }
- { os: macos-15, build_profile: macos-15-armv8-clang-14, host_profile: macos-15-armv8-clang-14 }
- { os: macos-26, build_profile: macos-26-armv8-clang-14, host_profile: macos-26-armv8-clang-14 }
- { os: macos-26, build_profile: macos-26-armv8-clang-14, host_profile: macos-26-armv8-clang-14, bindings: true }
- { os: windows-2022, build_profile: windows-2022-msvc-1940, host_profile: windows-2022-msvc-1940 }
- { os: ubuntu-24.04, build_profile: ubuntu-24.04-clang-18, host_profile: android-35-x86_64, ndk_version: 28.1.13356709 }
steps:
Expand All @@ -43,41 +61,53 @@ jobs:
brew install ccache
ccache -V

- name: setup java
if: matrix.bindings
uses: actions/setup-java@f4f1212c880fdec8162ea9a6493f4495191887b4 # v5
with:
distribution: temurin
java-version: 21

- name: setup python 3.14
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: 3.14
- name: install python dependencies
run: pip install conan
run: pip install conan ${{ matrix.bindings && 'pytest' || '' }}

- name: install NDK
if: startsWith(matrix.host_profile, 'android')
run: yes | ${ANDROID_HOME}/cmdline-tools/latest/bin/sdkmanager --install 'ndk;${{ matrix.ndk_version }}'

- name: conan cache key
shell: bash
run: echo "CONAN_CACHE_KEY=$(git rev-parse HEAD:conan-odr-index)-${{ hashFiles('conanfile.py', '.github/config/conan/**') }}" >> "$GITHUB_ENV"

- name: cache conan
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
with:
path: ${{ env.CONAN_HOME }}
key: conan-${{ matrix.host_profile }}-${{ env.CONAN_KEY_SUFFIX }}
key: conan-${{ env.CACHE_FLAVOR }}-${{ matrix.host_profile }}-${{ env.CONAN_KEY_SUFFIX }}-${{ env.CONAN_CACHE_KEY }}
restore-keys: |
conan-${{ matrix.host_profile }}-
conan-${{ env.CACHE_FLAVOR }}-${{ matrix.host_profile }}-${{ env.CONAN_KEY_SUFFIX }}-

- name: export conan-odr-index
run: python conan-odr-index/scripts/conan_export_all_packages.py --selection-config conan-odr-index/defaults.yaml
- name: conan config
run: conan config install .github/config/conan

- name: cache ccache
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
- name: restore ccache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
with:
path: ${{ env.CCACHE_DIR }}
key: ccache-${{ matrix.host_profile }}-${{ env.CCACHE_KEY_SUFFIX }}
key: ccache-${{ env.CACHE_FLAVOR }}-${{ matrix.host_profile }}-${{ env.CCACHE_KEY_SUFFIX }}-${{ github.run_id }}
restore-keys: |
ccache-${{ matrix.host_profile }}-
ccache-${{ env.CACHE_FLAVOR }}-${{ matrix.host_profile }}-${{ env.CCACHE_KEY_SUFFIX }}-

- name: conan install
run: >
conan install .
${{ matrix.bindings && '-o "&:with_jni=True" -o "&:with_python=True"' || '' }}
--profile:host '${{ matrix.host_profile }}'
--profile:build '${{ matrix.build_profile }}'
--build missing
Expand All @@ -96,6 +126,7 @@ jobs:
-DODR_WITH_WVWARE=ON
-DODR_WITH_LIBMAGIC=ON
-DODR_BUNDLE_ASSETS=ON
${{ matrix.bindings && '-DODR_JNI=ON -DODR_PYTHON=ON' || '' }}

- name: cmake
if: runner.os == 'Windows'
Expand All @@ -113,6 +144,23 @@ jobs:
- name: build
run: cmake --build build --config Release

- name: save ccache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
with:
path: ${{ env.CCACHE_DIR }}
key: ccache-${{ env.CACHE_FLAVOR }}-${{ matrix.host_profile }}-${{ env.CCACHE_KEY_SUFFIX }}-${{ github.run_id }}

# Both suites run in seconds, so they stay in this job rather than paying
# an artifact round-trip. The heavy `odr_test` still runs in `test`, which
# needs firefox and the reference-output submodule.
- name: junit
if: matrix.bindings
run: ctest --test-dir build/jni --output-on-failure
- name: pytest
if: matrix.bindings
run: ctest --test-dir build/python --output-on-failure

- name: install
run: cmake --build build --target install --config Release

Expand Down Expand Up @@ -267,6 +315,10 @@ jobs:

build-test-downstream:
runs-on: ${{ matrix.os }}
# Exports the full odr-index (no `--selection-config`) and resolves through
# `conan.lock`, so its dependency set differs from the `build` job's.
env:
CACHE_FLAVOR: downstream
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -296,17 +348,17 @@ jobs:
- name: install python dependencies
run: pip install conan

- name: install NDK
if: startsWith(matrix.host_profile, 'android')
run: yes | ${ANDROID_HOME}/cmdline-tools/latest/bin/sdkmanager --install 'ndk;${{ matrix.ndk_version }}'
- name: conan cache key
shell: bash
run: echo "CONAN_CACHE_KEY=$(git rev-parse HEAD:conan-odr-index)-${{ hashFiles('conanfile.py', 'conan.lock', '.github/config/conan/**') }}" >> "$GITHUB_ENV"

- name: cache conan
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
with:
path: ${{ env.CONAN_HOME }}
key: conan-${{ matrix.host_profile }}-${{ env.CONAN_KEY_SUFFIX }}
key: conan-${{ env.CACHE_FLAVOR }}-${{ matrix.host_profile }}-${{ env.CONAN_KEY_SUFFIX }}-${{ env.CONAN_CACHE_KEY }}
restore-keys: |
conan-${{ matrix.host_profile }}-
conan-${{ env.CACHE_FLAVOR }}-${{ matrix.host_profile }}-${{ env.CONAN_KEY_SUFFIX }}-

- name: export conan-odr-index
run: python conan-odr-index/scripts/conan_export_all_packages.py
Expand All @@ -316,13 +368,13 @@ jobs:
- name: conan odrcore
run: conan export . --name odrcore --version 0.0.0

- name: cache ccache
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
- name: restore ccache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
with:
path: ${{ env.CCACHE_DIR }}
key: ccache-${{ matrix.host_profile }}-${{ env.CCACHE_KEY_SUFFIX }}
key: ccache-${{ env.CACHE_FLAVOR }}-${{ matrix.host_profile }}-${{ env.CCACHE_KEY_SUFFIX }}-${{ github.run_id }}
restore-keys: |
ccache-${{ matrix.host_profile }}-
ccache-${{ env.CACHE_FLAVOR }}-${{ matrix.host_profile }}-${{ env.CCACHE_KEY_SUFFIX }}-

- name: conan install
run: >
Expand Down Expand Up @@ -350,6 +402,13 @@ jobs:
- name: build
run: cmake --build test/downstream/build --config Release

- name: save ccache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
with:
path: ${{ env.CCACHE_DIR }}
key: ccache-${{ env.CACHE_FLAVOR }}-${{ matrix.host_profile }}-${{ env.CCACHE_KEY_SUFFIX }}-${{ github.run_id }}

- name: run
if: matrix.build_profile == matrix.host_profile && (runner.os == 'Linux' || runner.os == 'macOS')
run: test/downstream/build/odr-test-downstream
Expand Down
102 changes: 0 additions & 102 deletions .github/workflows/jni.yml
View file
Open in desktop

This file was deleted.

Loading
Loading