Skip to content

Conversation

@FFY00
Copy link
Contributor

@FFY00 FFY00 commented Nov 5, 2025

Reference issue

Previous discussion in GH-23612.

What does this implement/fix?

This PR adds a Clang sanitizer ignore list (asan-ignore.txt; used with Clang's -fsanitize-ignorelist option) covering the existing issues, and adds a CI job that runs the test suite with the address sanitizer enabled.

Additional information

The -fsanitize-ignorelist option only seems to be able to match the first backtrace entry, so in certain situations, it doesn't seem to be possible to add well-scoped suppressions.

Consider the following example from test_sepfir2d_strided_3, where the error comes from a C++ type template method.

scipy/signal/tests/test_bsplines.py::TestSepfir2d::test_sepfir2d_strided_3[numpy-complex64] =================================================================
==2050131==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5110000233f8 at pc 0x7f065c78169a bp 0x7ffcf0e3e7a0 sp 0x7ffcf0e3e798
READ of size 8 at 0x5110000233f8 thread T0
    #0 0x7f065c781699 in std::complex<float>::__rep() const /usr/lib64/gcc/x86_64-pc-linux-gnu/15.2.1/../../../../include/c++/15.2.1/complex:1583:59
    #1 0x7f065c781699 in std::complex<float>& std::complex<float>::operator*=<float>(std::complex<float> const&) /usr/lib64/gcc/x86_64-pc-linux-gnu/15.2.1/../../../../include/c++/15.2.1/complex:1569:30
    #2 0x7f065c781699 in std::complex<float> std::operator*<float>(std::complex<float> const&, std::complex<float> const&) /usr/lib64/gcc/x86_64-pc-linux-gnu/15.2.1/../../../../include/c++/15.2.1/complex:437:11
    #3 0x7f065c7814ef in void _fir_mirror_symmetric<std::complex<float>>(std::complex<float>*, std::complex<float>*, int, std::complex<float>*, int, int, int) /home/anubis/git/scipy/build/../scipy/signal/_splinemodule.h:322:32
    #4 0x7f065c7800e2 in int _separable_2Dconvolve_mirror<std::complex<float>>(std::complex<float>*, std::complex<float>*, int, int, std::complex<float>*, std::complex<float>*, int, int, long*, long*) /home/anubis/git/scipy/build/../scipy/signal/_splinemodule.h:349:13
    #5 0x7f065c77d2ce in FIRsepsym2d(_object*, _object*) /home/anubis/git/scipy/build/../scipy/signal/_splinemodule.cc:105:11
    #6 0x7f06adbc3ae2 in cfunction_call /tmp/python-build.20251026041158.946242/Python-3.14.0/Objects/methodobject.c:575:18

It doesn't seem to be possible to add a suppression scoped only to _fir_mirror_symmetric.

To handle edge cases such as this one, I also added a fail_asan pytest marker, allowing us to easily skip these tests when running the test suite with the address sanitizer.

@github-actions github-actions bot added scipy.signal CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure Meson Items related to the introduction of Meson as the new build system for SciPy labels Nov 5, 2025
@FFY00 FFY00 force-pushed the add-asan branch 2 times, most recently from 4ad1dd5 to 713f23f Compare November 5, 2025 22:49
@lucascolley lucascolley changed the title Add ASAN suppressions and CI job BLD/CI: Add ASAN suppressions and CI job Nov 5, 2025
FFY00 and others added 3 commits November 6, 2025 06:02
Signed-off-by: Filipe Laíns <lains@riseup.net>
Signed-off-by: Filipe Laíns <lains@riseup.net>
Adapted from https://github.com/rgommers/scipy/tree/ci-asan

Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
Signed-off-by: Filipe Laíns <lains@riseup.net>
@rgommers
Copy link
Member

rgommers commented Nov 7, 2025

Thanks for working on this @FFY00. The job failure should be easy to resolve, it's just missing pooch and hypothesis.

I'd expect more suppressions are needed, which you may find if you re-run the job multiple times. E.g., #23612 (comment) identified a minpack function.

Also, it's be useful for the suppressions file to gain comments since from the symbol names it's very much non-obvious where the problematic code/component is.

Re the -fsanitize-ignorelist question in the PR description, maybe you have some suggestions @ngoldbaum?

@ngoldbaum
Copy link
Contributor

Re the -fsanitize-ignorelist question in the PR description, maybe you have some suggestions @ngoldbaum?

Sorry, no help here. My approach for fixing these in numpy has been to fix the issues rather than worrying about suppressions.

Signed-off-by: Filipe Laíns <lains@riseup.net>
@FFY00 FFY00 requested a review from DietBru as a code owner November 12, 2025 05:00
Signed-off-by: Filipe Laíns <lains@riseup.net>
Signed-off-by: Filipe Laíns <lains@riseup.net>
@FFY00
Copy link
Contributor Author

FFY00 commented Nov 23, 2025

Looks like faulthandler (and the free-threading GC, for that matter) changes the signal stack, so we might need to disable use_sigaltstack (see https://github.com/scipy/scipy/actions/runs/19614659518/job/56165518054?pr=23936).

@FFY00
Copy link
Contributor Author

FFY00 commented Nov 23, 2025

Seems like the CI is passing 😊

@FFY00
Copy link
Contributor Author

FFY00 commented Nov 23, 2025

I'd expect more suppressions are needed, which you may find if you re-run the job multiple times. E.g., #23612 (comment) identified a minpack function.

Also, it's be useful for the suppressions file to gain comments since from the symbol names it's very much non-obvious where the problematic code/component is.

That minpack function is already covered. I've added comments to the suppressions file to clarify where they come from, as you suggested 😊.

Signed-off-by: Filipe Laíns <lains@riseup.net>
Copy link
Member

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks good! Two minor requests:

  1. Can you move asan-ignore.txt under tools/?
  2. There is a serious amount of build log pollution, due to come we can't just update in this repo (e.g., f2py, SuperLU). Can you add -Wno-deprecated-declarations to CFLAGS in the CI job to patch that over?

Signed-off-by: Filipe Laíns <lains@riseup.net>
Signed-off-by: Filipe Laíns <lains@riseup.net>
@FFY00
Copy link
Contributor Author

FFY00 commented Nov 25, 2025

The CI is failing due to a flaky job. I am just gonna push a commit removing my accidental whitespace change to .spin/cmds.py to re-trigger it.

Signed-off-by: Filipe Laíns <lains@riseup.net>
# from scipy.optimize._minpack._lmdif
fun:enorm

# from scipy.interpolate._dierckx.qr_reduce_periodic
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one we just added, would be nice to investigate @czgdp1807 (does not block this PR of course).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Will look into this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will take this up tomorrow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A first try for a low-hanging fruit would be to apply Sebastian's fix from CuPy , cupy/cupy#9486

Short version: CuPy version of deBoor_D is a direct port from SciPY; the bug is a ~20 yr old use of unitialized memory; it never gave issues in SciPy but was causing segfaults in CuPy. I think Sebastian is right that in SciPy that unitialized memory is zero-initialized by std::vector<double>, but I did not gdb'd it myself.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NB: #24106 upstreamed the CuPy fix, but did not remove the ASAN suppression. We therefore don't know yet if the problem was in the deBoor_D kernel (== always been there, fixed now), or it is somewhere else, most likely in code added in #23233 ( == new in scipy 1.17.0, not fixed yet).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will remove it tomorrow or on Monday. Will test on my fork first and then send a PR to the main SciPy repository.

Signed-off-by: Filipe Laíns <lains@riseup.net>
@FFY00
Copy link
Contributor Author

FFY00 commented Nov 25, 2025

Seems like I forgot to update the paths when I moved the asan-ignore.txt to tools/. Just fixed it, so the PR should be ready 😊

@rgommers rgommers added this to the 1.17.0 milestone Nov 25, 2025
Copy link
Member

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great to get this in, thanks @FFY00!

@rgommers rgommers merged commit 33b1dcf into scipy:main Nov 25, 2025
53 checks passed
Comment on lines +157 to +158
fetch-tags: true
persist-credentials: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was there particular motivation for these options here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure Meson Items related to the introduction of Meson as the new build system for SciPy

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants