Skip to content

ci: Refactor build-related nix / docker / workflows#7408

Merged
bthomee merged 18 commits into
XRPLF:developfrom
mathbunnyru:more_nix
Jun 5, 2026
Merged

ci: Refactor build-related nix / docker / workflows#7408
bthomee merged 18 commits into
XRPLF:developfrom
mathbunnyru:more_nix

Conversation

@mathbunnyru
Copy link
Copy Markdown
Contributor

@mathbunnyru mathbunnyru commented Jun 5, 2026

High Level Overview of Change

This:

  • Updates most images to use nix-based versions
  • Moves files around, so it's easier to tell what they belong to
  • Creates a workflow to build packaging images (the images themselves will only be used after this PR is merged and images are pushed)
  • Creates a reusable workflow, which would build images for 2 archs and then merge it to a single one

What images are left, will be done separately:

  • pre-commit - has some libatomic issues
  • clang-tidy, it updates from 21 to 22, and there are lots of changes

Context of Change

API Impact

  • Public API: New feature (new methods and/or new fields)
  • Public API: Breaking change (in general, breaking changes should only impact the next api_version)
  • libxrpl change (any change that may affect libxrpl or dependents of libxrpl)
  • Peer protocol change (must be backward compatible or bump the peer protocol version)

Comment thread .github/workflows/reusable-build-merge-docker-images.yml Outdated
@mathbunnyru mathbunnyru changed the title ci: Use Nix for all workflows ci: Refactor build-related nix / docker / workflows Jun 5, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 5, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.3%. Comparing base (8abe82e) to head (f78cbdb).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           develop   #7408     +/-   ##
=========================================
- Coverage     82.4%   82.3%   -0.0%     
=========================================
  Files         1011    1011             
  Lines        76911   76911             
  Branches      8965    8965             
=========================================
- Hits         63343   63332     -11     
- Misses       13559   13570     +11     
  Partials         9       9             

see 5 files with indirect coverage changes

Impacted file tree graph

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mathbunnyru mathbunnyru requested a review from bthomee June 5, 2026 10:12
@mathbunnyru mathbunnyru added Ready to merge *PR author* thinks it's ready to merge. Has passed code review. Perf sign-off may still be required. Trivial Simple change with minimal effect, or already tested. Only needs one approval. labels Jun 5, 2026
@mathbunnyru mathbunnyru removed the Ready to merge *PR author* thinks it's ready to merge. Has passed code review. Perf sign-off may still be required. label Jun 5, 2026
@mathbunnyru mathbunnyru added the Ready to merge *PR author* thinks it's ready to merge. Has passed code review. Perf sign-off may still be required. label Jun 5, 2026
Comment thread .github/workflows/reusable-build-merge-docker-images.yml
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors CI-related Docker/Nix assets and GitHub Actions workflows to standardize on Nix-based CI images, add multi-arch image building/merging, and introduce dedicated workflows for building/publishing the Nix and packaging images.

Changes:

  • Switch multiple workflows to new ghcr.io/xrplf/xrpld/nix-* container images and bump the Linux CI image tag.
  • Add reusable workflows and top-level workflows to build per-arch Docker images and merge them into multi-arch manifests (Nix images + packaging images).
  • Add/relocate Docker test harness scripts and C++ sources used to validate toolchains/sanitizer runtime behavior inside the Nix Docker images.

Reviewed changes

Copilot reviewed 12 out of 21 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
package/README.md Updates packaging documentation for new matrix/image approach (but currently contains inconsistencies vs linux.json).
package/install-packaging-tools.sh New script to install deb/rpm tooling in packaging base images.
package/Dockerfile New packaging image Dockerfile that installs packaging tools into a vanilla distro base.
nix/docker/test_files/run-test-binaries.sh New runtime test runner to validate sanitizer binaries emit expected diagnostics.
nix/docker/test_files/cpp_sources/ubsan.cpp New UBSan-triggering test program source.
nix/docker/test_files/cpp_sources/tsan.cpp New TSan-triggering test program source.
nix/docker/test_files/cpp_sources/regular.cpp New non-sanitized “regular” test program source.
nix/docker/test_files/cpp_sources/asan.cpp New ASan-triggering test program source (also checks sanitizer headers).
nix/docker/test_files/compile-cpp-sources.sh New build-time compiler script to compile and patchelf the test binaries.
nix/docker/loader-path.sh New helper script to compute target PT_INTERP loader path by arch.
nix/docker/install-sanitizer-libs.sh New script to install sanitizer runtime libs into non-Nix base images for runtime validation.
nix/docker/Dockerfile Updates Nix CI Dockerfile to use relocated scripts and add build/run sanity checks.
nix/docker/check-tools.sh New script to verify expected tools are present and that git HTTPS cloning works.
.github/workflows/reusable-upload-recipe.yml Switches recipe upload workflow container to the new Nix-based image.
.github/workflows/reusable-build-merge-docker-images.yml New reusable workflow that builds per-arch images and merges them into multi-arch tags.
.github/workflows/reusable-build-docker-image.yml Small naming tweak; builds single-platform image with arch-suffixed tags for later merging.
.github/workflows/publish-docs.yml Switches docs build workflow container to the new Nix-based image and simplifies env printing.
.github/workflows/build-packaging-images.yml New workflow to build/push packaging images for Debian + RHEL base images.
.github/workflows/build-nix-images.yml New workflow to build/push Nix CI images for multiple base distros, multi-arch.
.github/workflows/build-nix-image.yml Removes the prior single-workflow implementation in favor of the new reusable/multi-arch flow.
.github/scripts/strategy-matrix/linux.json Bumps image_tag used for Nix-based Linux CI images.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread package/README.md
Comment thread package/README.md
Comment thread package/install-packaging-tools.sh
Comment thread .github/workflows/reusable-build-merge-docker-images.yml Outdated
Copy link
Copy Markdown
Collaborator

@bthomee bthomee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not for this PR, but should the publish-docs workflow be renamed to build-publish-docs? Or, should other workflows be renamed to not include build in their names? Building is usually a prerequisite for the action we're really interested in, whether pushing, publishing, or merging.

Separately, the reusable-build-docker-image workflow file also pushes, which is not reflected in the name. Similarly, the reusable-build-merge-docker-images does what it advertised, but the workflow name inside the file doesn't contain "merge".

There's some inconsistency in the naming overall.

Comment thread package/Dockerfile
Comment on lines +5 to +10
# Packaging runs in a vanilla distro image, so the tooling has to come
# from the distro's archive: debhelper for deb, rpm-build (and the
# systemd / find-debuginfo macros it depends on) for rpm.
# The container also uses git (real history) for
# build_pkg.sh's SOURCE_DATE_EPOCH; otherwise it falls back to a tarball
# download and the timestamp comes from wall-clock time.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see variable line lengths here. While this is just cosmetic and not in a need of fixing right now, it would be nice if we could enforce filling up the full 100 line length to the extent possible in the near future.

Do any such tools exist? I recall you mentioned markdownlint a while back, but even that can fix it for .md files, it would presumably not work for source code, Dockerfiles, etc. We'd need something generic.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think tools are changing comments that much, because sometimes there is custom formatting, like:

# To achieve this:
# - do this
# - and that
``

It looks nice, even though it's short.
I have no idea how to write a static tool which would make all these cases work nicely

@mathbunnyru
Copy link
Copy Markdown
Contributor Author

Not for this PR, but should the publish-docs workflow be renamed to build-publish-docs? Or, should other workflows be renamed to not include build in their names? Building is usually a prerequisite for the action we're really interested in, whether pushing, publishing, or merging.

Separately, the reusable-build-docker-image workflow file also pushes, which is not reflected in the name. Similarly, the reusable-build-merge-docker-images does what it advertised, but the workflow name inside the file doesn't contain "merge".

There's some inconsistency in the naming overall.

I am not sure we'll be able to achieve 100% consistency.
On one hand, we want to use all verbs what workflow does, on the other - we don't want to have 3 verbs in the name.
Tbh, for me a bit more important thing is clarity - whether it's easy to quickly understand what's going on in this workflow (both in file, and in UI). And from that point it's not too bad, I would say.

I fixed name for reusable-build-merge-docker-images.

@bthomee bthomee added this pull request to the merge queue Jun 5, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jun 5, 2026
@bthomee bthomee added this pull request to the merge queue Jun 5, 2026
Merged via the queue into XRPLF:develop with commit 63ffdc3 Jun 5, 2026
65 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready to merge *PR author* thinks it's ready to merge. Has passed code review. Perf sign-off may still be required. Trivial Simple change with minimal effect, or already tested. Only needs one approval.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants