Trilinos refactoring and updating #424

dcodoni · 2025-08-25T23:22:55Z

The Trilinos implementation is updated to use the latest packages available in the library. (address #422 )

Current situation

Currently the Trilinos implementation relies on archived, not supported or deprecated packages. In particular, it is based on the following packages:

Epetra: legacy linear algebra library in Trilinos for dense and sparse matrices and vectors. It has limited flexibility, making it harder to maintain and extend, for example if a custom preconditioner needs to be implemented in the trilinos environment (Planned Archival/Deprecation)
AztecOO: iterative linear solver package in Trilinos, it lacks some linear solvers such as FGMRES (Planned Archival/Deprecation)
Ifpack: legacy preconditioner package for Epetra-based matrices (Planned Archival/Deprecation)
ML: multilevel preconditioner package (Planned Archival/Deprecation)
Since Ifpack and ML are deprecated, they are not developed anymore and new optimized preconditioners will not be available in our solver.

Release Notes

Packages updated:

Epetra -> Tpetra: actively developed, templated, and Kokkos-enabled for GPU/CPU portability.
AztecOO -> Belos: supported modern iterative solver framework with flexible operator support.
Ifpack -> Ifpack2: supported templated preconditioner package compatible with Tpetra.
ML -> MueLu: actively developed multigrid preconditioner framework with advanced scalability (parameters are tuned for working on a general fsi problem).
The core package Teuchos is used for memory management, communication utilities, parameter lists, and interoperability across Trilinos packages.

All updated packages are wrapped under a Kokkos Node abstraction, enabling the same code to target different execution spaces. This provides the possibility of running for example on GPUs in the future, while still supporting efficient CPU execution when GPU usage is not required.

The workflow files are updated to use the latest docker image which contains the updated Trilinos built.
The dockerfiles are updated also for reference.

Documentation

Documentation about Trilinos linear algebra should be added.

Testing

existing tests using Trilinos are updated.
a new test is added for the fluid iliac artery to use Trilinos ML preconditioner

Code of Conduct & Contributing Guidelines

I agree to follow the Code of Conduct and Contributing Guidelines.

* main packages updates: Epetra->Tpetra; AztecOO->Belos; Ifpack->Ifpack2; ML->MueLu * refactor linear algebra backend to use Teuchos and Kokkos: - wrap Tpetra vectors and matrices with Teuchos::RCP for safer memory management and automatic reference counting. - use Kokkos (via Tpetra::Node/KokkosDeviceWrapperNode) to abstract parallel execution (same code to run efficiently on CPU and GPU theoretically) * removed all static objects * change of IC/ICT preconditioners in favor of RILUK(0)/RILUK(1)

* removed ubuntu20 and ubuntu22 folders. Created ubunut/ folder that contains dockerfile based on ubuntu-latest * modified the solver/dockerfile to use the ubuntu-latest docker container as a base

…ner: * simvascular/libraries:latest is the image containing the latest trilinos built

* increased the maximum iteration number for linear solver * add the fluid case iliac artery with ml preconditioner

Code/Source/solver/TrilinosLinearAlgebra.cpp

ktbolt

@dcodoni Nice work !

Make sure to update the Trilinos build instructions in the README.

* kokkos::finalize has been moved inside the TrilinosImpl class, fixing the compilation errors whenever trilinos is not used * trilinos linear solver residual test has been properly set, fixing first non-linear iteration problem of maxing out the linear iterations reducing excessively and unnecessarily the residual

ktbolt

Looks good !

* string length was computed incorrectly, leading to silent buffer overflow. * this only surfaced as a critical memory crash when linking with the latest Trilinos libraries.

…ics into trilinos-refactoring

- Added --ignore-errors mismatch to avoid geninfo line mismatch errors - Added --rc geninfo_unexecuted_blocks=1 to silence GCC warnings

ktbolt · 2025-09-16T19:15:00Z

@dcodoni What's the plan here ?

dcodoni · 2025-09-16T19:21:11Z

@ktbolt macos now is failing the ustruct tests even though they were succeeding before. The coverage problem I honestly have no idea yet!

ktbolt · 2025-09-16T21:53:38Z

@dcodoni MacOS tests failed because of an MPI error related (maybe) to a network time out, they now complete.

Ubuntu is failing because of a make error related (I think) to computing coverage.

dcodoni · 2025-09-16T22:07:29Z

@ktbolt the ubuntu test fails during generation of code coverage with error: geninfo: ERROR: mismatched end line. I tried then to ignore this type of errors by including in the CmakeLists the following flags: --ignore-errors mismatch --rc geninfo_unexecuted_blocks=1. This made it go further but at the end I got another make error, which I haven't investigated still.

mrp089 · 2025-09-16T22:36:47Z

Should we also exclude the tests/unitTests folder from coverage? That's currently included in computing overall coverage.

dcodoni · 2025-09-16T22:39:18Z

@mrp089 that is actually a good idea, and I think it may solve the problem I have been having

codecov · 2025-09-16T23:58:43Z

Codecov Report

❌ Patch coverage is 73.29545% with 94 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.35%. Comparing base (44fc5ec) to head (3b34dd5).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
Code/Source/solver/trilinos_impl.cpp	73.80%	88 Missing ⚠️
Code/Source/solver/trilinos_impl.h	33.33%	4 Missing ⚠️
Code/Source/solver/main.cpp	60.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #424      +/-   ##
==========================================
- Coverage   67.47%   67.35%   -0.13%     
==========================================
  Files         170      169       -1     
  Lines       32622    32620       -2     
  Branches     5740     5718      -22     
==========================================
- Hits        22012    21970      -42     
- Misses      10472    10513      +41     
+ Partials      138      137       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dcodoni · 2025-09-17T00:18:45Z

@ktbolt please review the changes in Code/Source/solver/CMakeLists.txt. In particular this flag: --ignore-errors mismatch avoided the line mismatch type of error I was having. The alternative is to keep the original CMake and exclude the unit tests from coverage as @mrp089 was saying.

ktbolt · 2025-09-17T00:29:44Z

@dcodoni I think it is cleaner to exclude the unit tests from the coverage step, the macros used by the unit tests are probably causing problems.

* Built solver sources into a shared object library (solver_objs) so they can be reused by both the main executable and unit tests. This avoids recompiling solver sources twice when ENABLE_UNIT_TEST is enabled. * Excluded unit tests from code coverage by removing coverage flags from the run_all_unit_tests target (-fno-profile-arcs, -fno-test-coverage). This prevents .gcda files from being generated for test code, ensuring coverage only reflects solver sources.

from the coverage.

…ics into trilinos-refactoring

…e section: * --ignore-errors mismatch --rc geninfo_unexecuted_blocks=1 * --ignore-errors unused These flags allowed to save a coverage file even in presence of errors of mismatch type. Fix some parametres setting in the Trilinos GMRES solver: krylov space, max iteration and max restarts.

Modified tests.yml to use the latest libraries container that has been built with latest Ubuntu and contains updated Trilinos library.

* add timer for the linear solver in Trilinos * add the possibility to read the MueLu Options for the ML preconditioner from file mueluOptions.xml if file is not present built-in parameters are used

dcodoni · 2025-10-24T21:58:34Z

@ktbolt I found some allocation problems in the Trilinos implementation that showed up only at high number of cores and for CFD problems. I am trying to investigate it. Let's not merge it now.

ktbolt · 2025-10-24T22:54:47Z

@dcodoni Yes I was wondering if something like that might show up.

* the localt to global map was created in sorted order, while the graph allocated in unsorted order * the graph is now created using the sorted order like the map

dcodoni · 2025-10-28T20:52:56Z

@ktbolt I fixed the bug in the LHS matrix graph creation. The problem was in the allocation of the non-zero elements per row during graph creation. The local to global map of the matrix used the sorted ordering while the graph was using information in unsorted ordering. This mismatch caused the wrong allocation problem, since Tpetra does not allow for dynamic allocation unlike Epetra.

ktbolt · 2025-10-28T21:26:50Z

@dcodoni Tricky bug to fix, good job !

* Trilinos updated: * main packages updates: Epetra->Tpetra; AztecOO->Belos; Ifpack->Ifpack2; ML->MueLu * refactor linear algebra backend to use Teuchos and Kokkos: - wrap Tpetra vectors and matrices with Teuchos::RCP for safer memory management and automatic reference counting. - use Kokkos (via Tpetra::Node/KokkosDeviceWrapperNode) to abstract parallel execution (same code to run efficiently on CPU and GPU theoretically) * removed all static objects * change of IC/ICT preconditioners in favor of RILUK(0)/RILUK(1) * Updating dockerfiles: * removed ubuntu20 and ubuntu22 folders. Created ubunut/ folder that contains dockerfile based on ubuntu-latest * modified the solver/dockerfile to use the ubuntu-latest docker container as a base * Updated dockerfiles and workflow file to use the latest docker container: * simvascular/libraries:latest is the image containing the latest trilinos built * Update the input files of trilinos related test cases for fluid and fsi * increased the maximum iteration number for linear solver * add the fluid case iliac artery with ml preconditioner * Modifications and improvements: * kokkos::finalize has been moved inside the TrilinosImpl class, fixing the compilation errors whenever trilinos is not used * trilinos linear solver residual test has been properly set, fixing first non-linear iteration problem of maxing out the linear iterations reducing excessively and unnecessarily the residual * Updating Trilinos packages list in README.md * Final update on dockerfile * Fix buffer overflow in genBC_Integ_X (set_bc.cpp): * string length was computed incorrectly, leading to silent buffer overflow. * this only surfaced as a critical memory crash when linking with the latest Trilinos libraries. * Changes to CMake to fix coverage and unit test mismatch * Modifying cmake file t ofix the coverage error * Fix coverage target for GCC (lcov/geninfo): - Added --ignore-errors mismatch to avoid geninfo line mismatch errors - Added --rc geninfo_unexecuted_blocks=1 to silence GCC warnings * fix coverage: correct path expansion in 'lcov --remove ' command * Removing the unit tests from coverage * Refactor CMake: unify solver build, exclude unit tests from coverage * Built solver sources into a shared object library (solver_objs) so they can be reused by both the main executable and unit tests. This avoids recompiling solver sources twice when ENABLE_UNIT_TEST is enabled. * Excluded unit tests from code coverage by removing coverage flags from the run_all_unit_tests target (-fno-profile-arcs, -fno-test-coverage). This prevents .gcda files from being generated for test code, ensuring coverage only reflects solver sources. * Removing the unit tests folder '*/CMakeFiles/run_all_unit_tests.dir/*' from the coverage. * Remove '*/CMakeFiles/run_all_unit_tests.dir/*' from the coverage list * Adding the following flags in the Cmake file under the enable_coverage section: * --ignore-errors mismatch --rc geninfo_unexecuted_blocks=1 * --ignore-errors unused These flags allowed to save a coverage file even in presence of errors of mismatch type. Fix some parametres setting in the Trilinos GMRES solver: krylov space, max iteration and max restarts. * Updates in new trilinos implementation: * add timer for the linear solver in Trilinos * add the possibility to read the MueLu Options for the ML preconditioner from file mueluOptions.xml if file is not present built-in parameters are used * Using fsils for the mesh equation in FSI test cases * Fixed allocation problem in the graph creation: * the localt to global map was created in sorted order, while the graph allocated in unsorted order * the graph is now created using the sorted order like the map

dcodoni added 4 commits August 21, 2025 17:16

Updating dockerfiles:

21cc5ce

* removed ubuntu20 and ubuntu22 folders. Created ubunut/ folder that contains dockerfile based on ubuntu-latest * modified the solver/dockerfile to use the ubuntu-latest docker container as a base

Updated dockerfiles and workflow file to use the latest docker contai…

a3d6c80

…ner: * simvascular/libraries:latest is the image containing the latest trilinos built

Update the input files of trilinos related test cases for fluid and fsi

2feec3c

* increased the maximum iteration number for linear solver * add the fluid case iliac artery with ml preconditioner

dcodoni requested a review from ktbolt August 25, 2025 23:23

ktbolt reviewed Aug 26, 2025

View reviewed changes

Code/Source/solver/TrilinosLinearAlgebra.cpp Show resolved Hide resolved

ktbolt requested changes Aug 26, 2025

View reviewed changes

dcodoni added 2 commits August 26, 2025 12:41

Updating Trilinos packages list in README.md

d1a8535

ktbolt approved these changes Aug 26, 2025

View reviewed changes

dcodoni and others added 7 commits August 26, 2025 15:57

Final update on dockerfile

30c8604

Fix buffer overflow in genBC_Integ_X (set_bc.cpp):

9a00ccd

* string length was computed incorrectly, leading to silent buffer overflow. * this only surfaced as a critical memory crash when linking with the latest Trilinos libraries.

Merge branch 'main' into trilinos-refactoring

f0da41c

Changes to CMake to fix coverage and unit test mismatch

5dc55e5

Merge branch 'trilinos-refactoring' of github.com:dcodoni/svMultiPhys…

c9f91a6

…ics into trilinos-refactoring

Modifying cmake file t ofix the coverage error

4882d86

Fix coverage target for GCC (lcov/geninfo):

353a30e

- Added --ignore-errors mismatch to avoid geninfo line mismatch errors - Added --rc geninfo_unexecuted_blocks=1 to silence GCC warnings

fix coverage: correct path expansion in 'lcov --remove ' command

1312bbb

dcodoni added 3 commits September 17, 2025 16:40

Removing the unit tests from coverage

0109e52

Removing the unit tests folder '*/CMakeFiles/run_all_unit_tests.dir/*'

9d97878

from the coverage.

dcodoni added 6 commits September 22, 2025 17:40

Merge branch 'trilinos-refactoring' of github.com:dcodoni/svMultiPhys…

5d792f6

…ics into trilinos-refactoring

Remove '*/CMakeFiles/run_all_unit_tests.dir/*' from the coverage list

364f8da

Merge branch 'main' into trilinos-refactoring

80d2430

Modified tests.yml to use the latest libraries container that has been built with latest Ubuntu and contains updated Trilinos library.

Updates in new trilinos implementation:

d9c9f71

* add timer for the linear solver in Trilinos * add the possibility to read the MueLu Options for the ML preconditioner from file mueluOptions.xml if file is not present built-in parameters are used

Using fsils for the mesh equation in FSI test cases

8df67be

Fixed allocation problem in the graph creation:

a15df0a

* the localt to global map was created in sorted order, while the graph allocated in unsorted order * the graph is now created using the sorted order like the map

Merge branch 'main' into trilinos-refactoring

3b34dd5

ktbolt merged commit 27bda0f into SimVascular:main Oct 28, 2025
8 checks passed

Trilinos refactoring and updating #424

Trilinos refactoring and updating #424

Uh oh!

Conversation

dcodoni commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current situation

Release Notes

Documentation

Testing

Code of Conduct & Contributing Guidelines

Uh oh!

Uh oh!

ktbolt left a comment

Choose a reason for hiding this comment

Uh oh!

ktbolt left a comment

Choose a reason for hiding this comment

Uh oh!

ktbolt commented Sep 16, 2025

Uh oh!

dcodoni commented Sep 16, 2025

Uh oh!

ktbolt commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dcodoni commented Sep 16, 2025

Uh oh!

mrp089 commented Sep 16, 2025

Uh oh!

dcodoni commented Sep 16, 2025

Uh oh!

codecov bot commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dcodoni commented Sep 17, 2025

Uh oh!

ktbolt commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dcodoni commented Oct 24, 2025

Uh oh!

ktbolt commented Oct 24, 2025

Uh oh!

dcodoni commented Oct 28, 2025

Uh oh!

ktbolt commented Oct 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dcodoni commented Aug 25, 2025 •

edited

Loading

ktbolt commented Sep 16, 2025 •

edited

Loading

codecov bot commented Sep 16, 2025 •

edited

Loading

ktbolt commented Sep 17, 2025 •

edited

Loading