Failing tests in M1/M2 macOS

[cdeil]

I get the following failing CPP tests:

(reaktoro) reaktoro $ cmake --build build --target tests-cpp
[ 29%] Built target ReaktoroEmbedded
[ 67%] Built target Reaktoro
[100%] Built target reaktoro-cpptests
[100%] Running C++ tests...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
reaktoro-cpptests is a Catch v2.13.9 host application.
Run with -? for options

-------------------------------------------------------------------------------
Testing ChemicalState class
-------------------------------------------------------------------------------
/Users/cdeil/code/oss/reaktoro/Reaktoro/Core/ChemicalState.test.cxx:34
...............................................................................

/Users/cdeil/code/oss/reaktoro/Reaktoro/Core/ChemicalState.test.cxx:164: FAILED:
  CHECK( state.speciesMass(idx("CaCO3(s)")) == 0.007 )
with expansion:
  0.007 == 0.007

***WARNING***
Species with same aggregate state should have unique names in Database, but species `H2O(aq)` with aggregate state `Aqueous` violates this rule. The unique name H2O(aq)! has been assigned instead.
-------------------------------------------------------------------------------
Testing EquilibriumSolver
  There is only pure water but there are other elements besides H and O with
  zero amounts
       When: using epsilon 1e-40
-------------------------------------------------------------------------------
/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:205
...............................................................................

/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:213: FAILED:
  CHECK( result.iterations() == 29 )
with expansion:
  27 == 29

-------------------------------------------------------------------------------
Testing EquilibriumSolver
  There is only pure water but there are other elements besides H and O with
  zero amounts
       When: using epsilon 1e-16
-------------------------------------------------------------------------------
/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:223
...............................................................................

/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:231: FAILED:
  CHECK( result.iterations() == 17 )
with expansion:
  16 == 17

-------------------------------------------------------------------------------
Testing EquilibriumSolver
  There is an aqueous solution, gaseous solution, several minerals
       When: reactivity restrictions are not imposed
       When: using epsilon 1e-40
-------------------------------------------------------------------------------
/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:379
...............................................................................

/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:387: FAILED:
  CHECK( result.iterations() <= 54 )
with expansion:
  56 <= 54

-------------------------------------------------------------------------------
Testing EquilibriumSolver
  There is an aqueous solution, gaseous solution, several minerals
       When: reactivity restrictions are not imposed
       When: using epsilon 1e-16
-------------------------------------------------------------------------------
/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:397
...............................................................................

/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:405: FAILED:
  CHECK( result.iterations() == 35 )
with expansion:
  34 == 35

-------------------------------------------------------------------------------
Testing EquilibriumSolver
  There is an aqueous solution with given pH in equilibrium with a gaseous
  solution
       When: using epsilon 1e-40
-------------------------------------------------------------------------------
/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:614
...............................................................................

/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:622: FAILED:
  CHECK( result.iterations() <= 57 )
with expansion:
  59 <= 57

-------------------------------------------------------------------------------
Testing EquilibriumSolver
  There is an aqueous solution in equilibrium with one or another mineral
-------------------------------------------------------------------------------
/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:659
...............................................................................

/Users/cdeil/code/oss/reaktoro/Reaktoro/Equilibrium/EquilibriumSolver.test.cxx:682: FAILED:
  CHECK( result.iterations() == 32 )
with expansion:
  28 == 32

===============================================================================
test cases:   169 |   167 passed | 2 failed
assertions: 13757 | 13750 passed | 7 failed

make[3]: *** [tests/CMakeFiles/tests-cpp.dir/build.make:71: tests/CMakeFiles/tests-cpp] Error 7
make[2]: *** [CMakeFiles/Makefile2:1365: tests/CMakeFiles/tests-cpp.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:1372: tests/CMakeFiles/tests-cpp.dir/rule] Error 2
make: *** [Makefile:670: tests-cpp] Error 2

This is on MacOS with system clang:

(reaktoro) reaktoro $ clang --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

@allanleal - how to fix them? Asserting on float equality or number of iterations probably isn't stable and should be relaxed? Or is there some better way or assertions overall that could be used?

[allanleal]

Hi Christoph, the current CI system is not building for macOS ARM64. For some calculations, there are some differences in rounding that leads to slight deviations in numerical behavior (thus causing calculations to converge in small differing number of iterations). For the float checks, they should have been compared with Approx:

CHECK(value == Approx(expected_value));

The link below indicates that macOS ARM64 is now supported in GitHub Actions (since Oct 2023). I'll create an issue to consider it as a target for CI.

https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/