ci: unfreeze the build caches and build the bindings centrally - #619
Merged
Conversation
`actions/cache` never overwrites an existing key, so a key that never varies is written once and then frozen: every later run restores that first snapshot and rebuilds whatever it is missing from source. Measured on run 30175816272, the ubuntu-clang build spends 13 of its 22 minutes in `conan install`, rebuilding 14 dependencies (pdf2htmlex, poppler, cairo, fontforge, wvware, ...) from source — on every single run. No job in that run logs a cache save. The jni workflow, whose key happens to be young enough to hold a complete cache, does the same `conan install` in under a minute. Two jobs sharing one key made it worse: `build` and `build-test-downstream` both wrote `conan-ubuntu-24.04-clang-18-r1` with different dependency sets, so whichever landed first left the other rebuilding forever. `CACHE_FLAVOR` now separates them. Key the conan cache on what determines its content (odr-index revision + conanfile + profile) so it is rewritten exactly when dependencies change, and give ccache a per-run key restored by prefix so it can grow. ccache is saved only on the default branch — branch-scoped entries are invisible to other branches and would only churn the 10 GB repo quota, which is already full (10.74 GB across 24 entries). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LDqWDNJQNroJPH9j5BfaPv
The JNI and Python bindings each had their own workflow that reconfigured and rebuilt the core from scratch for ubuntu-clang and macos-26 — the two profiles `build_test` already covers. Per push that was three independent resolutions of the same dependency graph and three compiles of the same library, plus three sets of caches competing for the repo quota. `ODR_CLI`/`ODR_TEST`/`ODR_JNI`/`ODR_PYTHON` are independent subdirectory guards and `with_jni`/`with_python` only widen the conan graph, so one configure produces all of it. The matrix gains a `bindings` flag; the two suites run in-job via ctest because they take seconds and an artifact round-trip would cost more than it saves. `wheels` stays where it is: it deliberately builds the reduced dependency set that `pyproject.toml` pins (no pdf2htmlEX/wvWare/libmagic, whose runtime data cannot ship inside a wheel), so its binary cannot be the one `build_test` produces. `maven` is unaffected — the jar is classes-only and needs no native artifact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LDqWDNJQNroJPH9j5BfaPv
That job's matrix only ever contains ubuntu-24.04-clang-18, so the step's `startsWith(matrix.host_profile, 'android')` guard never fires and the `matrix.ndk_version` it interpolates is not defined anywhere — actionlint flags the reference as an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LDqWDNJQNroJPH9j5BfaPv
andiwand
force-pushed
the
ci-consolidate-builds
branch
from
July 26, 2026 07:29
62aa7af to
ad2a2cc
Compare
andiwand
enabled auto-merge (squash)
July 26, 2026 08:09
andiwand
disabled auto-merge
July 26, 2026 08:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Generated with Claude Code
Stacked on #618.
Two changes, independent of each other.
1. Stop freezing the conan and ccache caches
actions/cachenever overwrites an existing key, so a key that never varies is written exactly once and then frozen — every later run restores that first snapshot and rebuilds whatever is missing from source.Measured on run 30175816272, the
ubuntu-clangbuild spends 13 of its 22 minutes inconan install, rebuilding 14 dependencies from source (pdf2htmlex, poppler, cairo, fontforge, wvware, fontconfig, libgsf, …) — on every single run. Every job in that run logsCache restored from key: …; none logs a cache save. The control case is already in the repo: the jni workflow's key was young enough to hold a complete cache, and it did the sameconan installin under a minute.Two jobs sharing one key made it worse —
buildandbuild-test-downstreamboth wroteconan-ubuntu-24.04-clang-18-r1with different dependency sets, so whichever landed first left the other rebuilding forever.conanfile.py+ conan config + profile), so it is rewritten precisely when the dependencies change.CACHE_FLAVORseparates jobs that resolve a different dependency set for the same profile.CCACHE_KEY_SUFFIX/CONAN_KEY_SUFFIXstay as the manual "throw it all away" knob.2. Build the bindings in the central build job
jni.ymlandpython.yml'sbuild-testeach reconfigured and rebuilt the core from scratch forubuntu-clangandmacos-26— the two profilesbuild_testalready covers. Per push: three independent resolutions of the same dependency graph, three compiles of the same library, three sets of caches competing for quota.ODR_CLI/ODR_TEST/ODR_JNI/ODR_PYTHONare independentadd_subdirectoryguards andwith_jni/with_pythononly widen the conan graph, so a single configure produces all of it. The matrix gains abindingsflag; both suites run in-job via ctest since they take seconds and an artifact round-trip would cost more than it saves.jni.ymlis removed andpython.ymlkeepsformat/wheels/sdist/publish/pypi.What deliberately did not move
wheelsbuilds the reduced dependency setpyproject.tomlpins (no pdf2htmlEX/wvWare/libmagic — their runtime data cannot ship inside a wheel), so its binary cannot be the onebuild_testproduces. It is also already cheap at 4–5 min.mavenis untouched: the jar is classes-only and needs no native artifact. It only becomes a consumer of this job if the jar ever bundles per-platformlibodr_jni.Trade-off
Binding compile time now sits on the critical path ahead of
dockerand the reference-outputtestjob, and a binding failure blocks them. Limited to the two profiles that need it.Verification
actionlintclean apart from pre-existing warnings.ctest --test-dir build/pythonis a new invocation (the old workflow called pytest directly) — verified locally against a build withODR_JNI/ODR_PYTHON/ODR_TESTall on: 32 passed. The JNI ctest path is unchanged fromjni.yml.