Skip to content

WIP: Isolate ARMBUILD-Ubuntu-24.04-arm CI for failure diagnosis#5948

Closed
hjmjohnson wants to merge 3 commits intoInsightSoftwareConsortium:mainfrom
hjmjohnson:isolate-arm-ci-failures
Closed

WIP: Isolate ARMBUILD-Ubuntu-24.04-arm CI for failure diagnosis#5948
hjmjohnson wants to merge 3 commits intoInsightSoftwareConsortium:mainfrom
hjmjohnson:isolate-arm-ci-failures

Conversation

@hjmjohnson
Copy link
Member

@hjmjohnson hjmjohnson commented Mar 14, 2026

Summary

  • Diagnose and fix ARM CI dashboard failures caused by false-positive warnings and errors
  • Suppress GCC -Wmaybe-uninitialized warnings from ThirdParty Eigen3 via CTEST_CUSTOM_WARNING_EXCEPTION
  • Add false-positive error exceptions to CTEST_CUSTOM_ERROR_EXCEPTION for expected test output (e.g., "--- ERROR ---" banners from caught exceptions, itk::ExceptionObject traces)
  • Temporarily disable non-ARM CI pipelines to isolate the issue (WIP commit, will be reverted before merge)

Root Cause

  1. Warnings: GCC on ARM produces -Wmaybe-uninitialized warnings from Eigen3's SelfadjointMatrixVector.h. The itk_common.cmake dashboard script treats build_warnings > 0 as CI failure. These warnings are fixed upstream (Eigen commit 662d5c21f).
  2. Errors: CDash flags test output containing error-like strings (e.g., "--- ERROR ---" from expected exception handling) as build errors despite tests passing.

Commits

  • WIP: Disable all CI builds except ARMBUILD-Ubuntu-24.04-arm — CI isolation (to be reverted)
  • COMP: suppress some third party warnings on cdash — Add ThirdParty warning exceptions
  • COMP: Suppress warnings from ThirdParty Eigen module — Add Eigen-specific warning suppression
  • COMP: Add false-positive error exceptions for expected ITK test output — Suppress false-positive CDash errors

Test plan

  • Only ARMBUILD-Ubuntu-24.04-arm CI runs on this PR
  • Verify dashboard no longer reports false warnings/errors
  • Revert WIP CI isolation commit before merge

🤖 Generated with Claude Code

@github-actions github-actions bot added type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct labels Mar 14, 2026
@hjmjohnson
Copy link
Member Author

@blowekamp

FYI: The following build warnings only show up under the linux arm builds:

└─[0] rg "warning: " 1_ARMBUILD-Ubuntu-24.04-arm.txt|fgrep -v libpng                             (isolate-arm-ci-failures|…1⚑9)
2026-03-13T23:54:34.6885757Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-13T23:54:34.6934207Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-13T23:54:34.6951147Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:06:13.5784560Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:06:56.2395410Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:08:56.5683796Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:24:19.7212980Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:24:19.7420150Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]

and there were some changes made to EIgen yesterday.

@hjmjohnson
Copy link
Member Author

@blowekamp

A little AI evaluation of the differences for the passing and failing logs is narrowing down the issue with what look to be very plausible:

⏺ The ThirdParty patterns on lines 80-92 use a format like ".*[/\\][Mm]odules[/\\][Tt]hird[Pp]arty[/\\][Specific]..." but there's no Eigen3 entry. Lines 127 and 130 add GoogleTest and DoubleConversion entries. But the Eigen3 warnings have this path format:

../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/...

Let me check if the existing ThirdParty warning exceptions match this. The patterns on lines 80-92 are for specific subdirectories (DICOMParser, Expat, JPEG, etc.) — there's no catch-all for all of ThirdParty.

The warning is: Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]

This only appears on GCC (Ubuntu ARM) and not on Clang (macOS Rosetta), which is typical for -Wmaybe-uninitialized — it's a GCC-only warning that Clang doesn't produce.

Root cause: The ARMBUILD-Ubuntu-24.04-arm CI fails because GCC on ARM produces 8 -Wmaybe-uninitialized warnings from Eigen3's SelfadjointMatrixVector.h, which are counted by ctest_build(NUMBER_WARNINGS build_warnings). The itk_common.cmake script (line 495-498) treats any
build_warnings > 0 as a CI failure. These Eigen3 warnings are not suppressed by CTEST_CUSTOM_WARNING_EXCEPTION because there's no pattern matching Eigen3 in the exception list — unlike other ThirdParty subdirectories (JPEG, TIFF, VNL, etc.) that each have their own exception
entry.

The Rosetta build passes because Clang doesn't emit -Wmaybe-uninitialized.

@hjmjohnson
Copy link
Member Author

@blowekamp FYI:

⏺ Based on the analysis, the GCC -Wmaybe-uninitialized warnings from Eigen3's SelfadjointMatrixVector.h have been fixed upstream through a substantial rewrite of the algorithm. The warning-triggering code no longer exists in the upstream version.

@github-actions github-actions bot added the area:ThirdParty Issues affecting the ThirdParty module label Mar 14, 2026
@blowekamp
Copy link
Member

@seanm ref#5876

There are more warnings suppression here for the third party than I realized:
https://github.com/InsightSoftwareConsortium/ITK/blob/main/CMake/CTestCustom.cmake.in#L80

Adding Eigen3 here seem reasonable if that is the convention. I am not sure how the changed yesterday caused this warning to be emitted, maybe it was already being suppressed somehow.

@seanm
Copy link
Contributor

seanm commented Mar 14, 2026

how about this: #5949

@hjmjohnson hjmjohnson force-pushed the isolate-arm-ci-failures branch 2 times, most recently from 131f614 to 06fa5c6 Compare March 14, 2026 23:28
@github-actions github-actions bot removed the area:ThirdParty Issues affecting the ThirdParty module label Mar 14, 2026
@hjmjohnson hjmjohnson force-pushed the isolate-arm-ci-failures branch from 6b8797b to 6151b8c Compare March 16, 2026 14:13
hjmjohnson and others added 2 commits March 16, 2026 09:17
Improve the ccache caching strategy in pixi.yml and arm.yml:

- Key cache on OS + build config + build-system file hashes rather
  than github.sha. This gives reuse when source files change but
  the toolchain/build-config is still compatible, which is exactly
  where ccache pays off. ccache itself handles source-level reuse.
- Add tiered restore-keys fallback (OS+config → OS-only) so PR
  branches inherit the best available cache via GitHub's built-in
  branch scoping (PRs can read default branch caches)
- Save cache on all branches (if: always()), not just main, so
  PR branches benefit from incremental caches across pushes
- Bump cache key version to v3 to avoid restoring old caches
- Remove CCACHE_NODIRECT=1 to enable direct mode for better hit
  rates (CCACHE_COMPILERCHECK=content already ensures correctness)
- Increase CCACHE_MAXSIZE from 2.4G to 5G to leverage larger
  storage limits
- Use separate restore/save actions to support saving even on
  build failure (if: always())

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Temporarily disable all CI pipelines except the ARM Ubuntu build
to isolate dashboard failures for debugging.

- arm.yml: Remove x86_64-rosetta and Python matrix entries
- pixi.yml: Comment out PR trigger
- Azure pipelines: Set pr: none for Linux, LinuxPython, MacOS,
  MacOSPython, Windows, WindowsPython

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@hjmjohnson hjmjohnson force-pushed the isolate-arm-ci-failures branch 2 times, most recently from 8e144dd to 2642b9b Compare March 16, 2026 14:18
@github-actions github-actions bot added the area:ThirdParty Issues affecting the ThirdParty module label Mar 16, 2026
…ector

GCC on aarch64 (ARM) emits a false-positive -Wmaybe-uninitialized
warning at line 224 of SelfadjointMatrixVector.h. The dashboard script
(itk_common.cmake) treats any build warning as CI failure.

Add targeted GCC diagnostic pragmas to suppress the warning in this
file only, scoped to GCC (not Clang) via __GNUC__ / __clang__ guards.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@blowekamp
Copy link
Member

I think this is mostly a agent PR, and I'm not sure it's all going in the right direction.

Regarding adding the disabling of the warnings for Eigen. Please avoid directly modifying the third party libraries. For the warning disable there is already some done here And additional ones for GCC could be added here.

I also did work trying to get the Eigen library to follow the convention of other ITK ThirdParty headers here: #5952 But because Eigen is an all header library there is not a single header file for an interface, so the include path needs access to all the Eigen header files. However, such an approach would allow for wrapping headers with push/pop to disable warnings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ThirdParty Issues affecting the ThirdParty module type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants