Complex multiply issue in GCC15

[WardBrian]

Caught by https://jenkins.flatironinstitute.org/blue/organizations/jenkins/Stan%2FBleedingEdgeCompilersMonthly/detail/BleedingEdgeCompilersMonthly/259/pipeline

The issue seems to be specific to fvar<fvar<double>>.

c.f. #3006, though note that this is a runtime failure, not a compile failure.

cc @SteveBronder @andrjohns

Details

[----------] 1 test from mathMix
[ RUN      ] mathMix.complexMultiply
./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(1) = x2(1): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(2) = x2(2): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(3) = x2(3): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(4) = x2(4): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(5) = x2(5): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(6) = x2(6): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(7) = x2(7): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(8) = x2(8): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(2) = x2(2): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(3) = x2(3): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(4) = x2(4): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(5) = x2(5): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(6) = x2(6): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(7) = x2(7): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(8) = x2(8): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(9) = x2(9): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(0) = x2(0): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(1) = x2(1): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(2) = x2(2): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(3) = x2(3): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(4) = x2(4): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(5) = x2(5): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(6) = x2(6): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(7) = x2(7): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(0) = x2(0): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(1) = x2(1): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(2) = x2(2): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(3) = x2(3): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(4) = x2(4): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(5) = x2(5): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(6) = x2(6): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(7) = x2(7): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

[  FAILED  ] mathMix.complexMultiply (238 ms)

[andrjohns]

Passes for me under the fedora:42 docker image with gcc 15.1:

[root@a2b32ce531e0 math]# python ./runTests.py -j4 test/unit/math/mix/fun/multiply_complex_test.cpp 
------------------------------------------------------------
make -j4 test/unit/math/mix/fun/multiply_complex_test
g++ -std=c++17 -pthread -D_REENTRANT -ffp-contract=off -Wno-sign-compare -Wno-ignored-attributes -Wno-class-memaccess      -I lib/tbb_2020.3/include    -O3  -I . -I lib/eigen_3.4.0 -I lib/boost_1.84.0 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest   -w -Wno-psabi    -DBOOST_DISABLE_ASSERTS         -DGTEST_HAS_PTHREAD=0 -DGTEST_HAS_PTHREAD=0  -c -o test/unit/math/mix/fun/multiply_complex_test.o test/unit/math/mix/fun/multiply_complex_test.cpp
g++ -std=c++17 -pthread -D_REENTRANT -ffp-contract=off -Wno-sign-compare -Wno-ignored-attributes -Wno-class-memaccess      -I lib/tbb_2020.3/include    -O3  -I . -I lib/eigen_3.4.0 -I lib/boost_1.84.0 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest  -w -Wno-psabi    -DBOOST_DISABLE_ASSERTS         -DGTEST_HAS_PTHREAD=0       -Wl,-L,"/root/math/lib/tbb"   -Wl,-rpath,"/root/math/lib/tbb"      test/unit/math/mix/fun/multiply_complex_test.o lib/benchmark_1.5.1/googletest/googletest/src/gtest_main.cc lib/benchmark_1.5.1/googletest/googletest/src/gtest-all.o lib/tbb/libtbb.so.2       -ltbb   -o test/unit/math/mix/fun/multiply_complex_test
rm test/unit/math/mix/fun/multiply_complex_test.o
------------------------------------------------------------
test/unit/math/mix/fun/multiply_complex_test --gtest_output="xml:test/unit/math/mix/fun/multiply_complex_test.xml"
Running main() from lib/benchmark_1.5.1/googletest/googletest/src/gtest_main.cc
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from mathMix
[ RUN      ] mathMix.complexMultiply
[       OK ] mathMix.complexMultiply (207 ms)
[----------] 1 test from mathMix (207 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (207 ms total)
[  PASSED  ] 1 test.

[root@a2b32ce531e0 math]# g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/aarch64-redhat-linux/15/lto-wrapper
Target: aarch64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,objc,obj-c++,ada,go,d,m2,cobol,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --enable-libstdcxx-backtrace --with-libstdcxx-zoneinfo=/usr/share/zoneinfo --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-15.1.1-build/gcc-15.1.1-20250425/obj-aarch64-redhat-linux/isl-install --enable-gnu-indirect-function --build=aarch64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serialization=1
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 15.1.1 20250425 (Red Hat 15.1.1-1) (GCC)

[WardBrian]

I get the same failure as jenkins if I try in the fedora docker.

Could be an aarch vs x86 thing?

[root@99a10c8be2b6 math]# python runTests.py test/unit/math/mix/fun/multiply_complex_test.cpp -j4
------------------------------------------------------------
make -j4 test/unit/math/mix/fun/multiply_complex_test
make: 'test/unit/math/mix/fun/multiply_complex_test' is up to date.
------------------------------------------------------------
test/unit/math/mix/fun/multiply_complex_test --gtest_output="xml:test/unit/math/mix/fun/multiply_complex_test.xml"
Running main() from lib/benchmark_1.5.1/googletest/googletest/src/gtest_main.cc
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from mathMix
[ RUN      ] mathMix.complexMultiply
./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(1) = x2(1): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(2) = x2(2): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(3) = x2(3): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(4) = x2(4): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(5) = x2(5): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(6) = x2(6): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(7) = x2(7): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(8) = x2(8): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(0) = x2(0): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(1) = x2(1): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(2) = x2(2): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(3) = x2(3): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(4) = x2(4): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(5) = x2(5): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(6) = x2(6): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(7) = x2(7): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(0) = x2(0): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(1) = x2(1): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(2) = x2(2): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(3) = x2(3): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(4) = x2(4): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(5) = x2(5): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 0,
x2 evaluates to 1, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(6) = x2(6): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

./test/unit/math/expect_near_rel.hpp:34: Failure
The difference between x1 and x2 is 1, which exceeds tol_val, where
x1 evaluates to 1,
x2 evaluates to 0, and
tol_val evaluates to 5.0000000000000002e-05.
expect_near_rel_finite in: expect_near_rel; require items x1(7) = x2(7): hessian_fvar() grad for finite_diff vs fvar<fvar<double>>

[  FAILED  ] mathMix.complexMultiply (191 ms)
[----------] 1 test from mathMix (191 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (191 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] mathMix.complexMultiply

 1 FAILED TEST
test/unit/math/mix/fun/multiply_complex_test --gtest_output="xml:test/unit/math/mix/fun/multiply_complex_test.xml" failed
exit now (04/28/25 14:13:04 UTC)
1

[root@99a10c8be2b6 math]# g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/15/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,objc,obj-c++,ada,go,d,m2,cobol,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --enable-libstdcxx-backtrace --with-libstdcxx-zoneinfo=/usr/share/zoneinfo --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-15.1.1-build/gcc-15.1.1-20250425/obj-x86_64-redhat-linux/isl-install --enable-offload-targets=nvptx-none,amdgcn-amdhsa --enable-offload-defaulted --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serialization=1
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 15.1.1 20250425 (Red Hat 15.1.1-1) (GCC) 

[andrjohns]

Ahh yeah I can also reproduce the failures under the x64_64 container.

I also tried adding -ffp-contract=off, which (I think) is the only different compilation flag for aarch64, and there was no change.

Odd.

[andrjohns]

I think it's something optimisation-related, the test only fails with -O3, it's fine with -O{0,1,2}

[WardBrian]

In my experience those are the most annoying ones...

[andrjohns]

Narrowed it down a bit further, it only fails when mixing scalar & matrix arguments (not for both scalar or both matrix).

It's also definitely caused by the vectorisation of loops. The tests pass with -O2, but fail with -O2 -fvect-cost-model=dynamic (which is one of the flags added by -O3)

[andrjohns]

The plot continues to thicken. There's some interaction with the use of eval in the tests, where if I remove the eval call from the testing functor definitions then the tests also pass. Bizarre.

[andrjohns]

Here's a minimum test-case (essentially just extracting the relevant part of the test framework) to reproduce the error:

#include <stan/math.hpp>
#include <stan/math/fwd.hpp>

int main() {
    using stan::math::serialize_args;
    using stan::math::serialize_return;
    using stan::math::to_deserializer;
    using stan::math::eval;

    double d_scalar = 1.0;
    Eigen::VectorXcd cd_mat(2);
    cd_mat << 1, 2;

    Eigen::VectorXd args = serialize_args(cd_mat, d_scalar);

    auto g12 = [&](const auto& v) {
      using complex_t = std::complex<stan::scalar_type_t<decltype(v)>>;
      Eigen::Matrix<complex_t, -1, 1> vec(2);
      vec << complex_t(v(0), v(1)), complex_t(v(2), v(3));
      auto scal = v(4);

      auto ret = eval(stan::math::multiply(vec, scal));
      return ret(0).imag();
  };


   double fx_ad;
   Eigen::VectorXd grad_ad;
   Eigen::MatrixXd H_ad;
   stan::math::hessian<double>(g12, args, fx_ad, grad_ad, H_ad);

   double fx_fd;
   Eigen::VectorXd grad_fd;
   Eigen::MatrixXd H_fd;
   stan::math::internal::finite_diff_hessian_auto(g12, args, fx_fd, grad_fd, H_fd);

   std::cout << "grad_ad: " << grad_ad.transpose() << std::endl;
   std::cout << "grad_fd: " << grad_fd.transpose() << std::endl;
}

[WardBrian]

@andrjohns that reduced case is passing for me (the full test is still failing)

[andrjohns]

Ah yeah the reduced example doesn't trip the auto-vectorisation, you need to add CPPFLAGS += -O2 -fvect-cost-model=unlimited to the make/local

But you can verify the eval issue (without makeflag changes) by just removing the eval calls in the functors in expect_ad_vv, and all tests will pass.

Here's an even simpler reprex of the eval issue (compiled with the makeflags above):

#include <stan/math.hpp>
#include <stan/math/fwd.hpp>

int main() {
  using stan::math::fvar;
  using fd = fvar<double>;
  using ffd = fvar<fvar<double>>;

  ffd scal = 1.0;

  Eigen::Matrix<std::complex<ffd>, -1, 1> vec_fvar(2);
  // Estimating the gradient for the imaginary part of first element
  vec_fvar << std::complex<ffd>(1, ffd(fd(0, 1), fd(1, 0))),
              2;

  auto res = stan::math::multiply(vec_fvar, scal);
  auto res_eval = stan::math::eval(res);

  std::cout << "res: " << res(0).imag().d_.val_ << std::endl;
  std::cout << "res_eval: " << res_eval(0).imag().d_.val_ << std::endl;
}

Which prints:

res: 1
res_eval: 0

(where res is correct and res_eval is wrong)

[WardBrian]

Interesting. Manually assigning to an Eigen::Matrix rather than using auto works when done on its own, but calling eval on it then rewinds-time and breaks it....

  Eigen::Matrix<std::complex<ffd>, -1, 1> res_mat =  stan::math::multiply(vec_fvar, scal);
  auto res_eval = stan::math::eval(res_mat);

  std::cout << "res_mat: " << res_mat(0).imag().d_.val_ << std::endl;
  std::cout << "res_eval: " << res_eval(0).imag().d_.val_ << std::endl;

this prints 0, 0. Comment out res_eval and it prints 1

[WardBrian]

The definition of eval is so simple that I think it's probably just where some earlier UB is finally biting us

[WardBrian]

Maybe related? https://gcc.gnu.org/bugzilla/show_bug.cgi?id=116463

[WardBrian]

@andrjohns have you had a chance to look into this any more? I've talked to a few people who know a lot about compiler vectorization and they seemed to think that the compiler is probably assuming some maximum sizeof for complex types, which we're violating.

If that is indeed the case, I think we might be hosed unless we switch to using our own struct for complex numbers

[andrjohns]

I'm not sure if it's related somehow, but I found a different bug when initialising complex vector types: https://gitlab.com/libeigen/eigen/-/issues/2931

I'll see what the Eigen devs say, hopefully there's some central Eigen config for complex type handling that we just need to add

[WardBrian]

hopefully there's some central Eigen config for complex type handling that we just need to add

There is this, but it's unreleased.