Skip to content

DOC: Add instructions for building with ASAN #2473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Jun 5, 2025
Merged
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
cd3e1ab
add instructions for building with ASAN
david-cortes-intel May 13, 2025
c816084
remove unneeded variable
david-cortes-intel May 15, 2025
f6911cc
Update INSTALL.md
david-cortes-intel May 16, 2025
1ec21ca
ENH: Enable model conversion from TreeLite (#2452)
david-cortes-intel May 14, 2025
03038f5
[enhancement, CI] add numpydoc validation to pre-commit (#2455)
icfaust May 14, 2025
9221ed4
[enhancement] add ```__dlpack__``` and ```__dlpack_device__``` attrib…
icfaust May 14, 2025
81a1478
chore(deps): update dependency xgboost to v3.0.1 (#2474)
renovate[bot] May 14, 2025
b60e4f6
chore(deps): update dependency cython to v3.1.0 (#2470)
renovate[bot] May 14, 2025
58329ce
chore(deps): update dependency cmake to v4.0.2 (#2468)
renovate[bot] May 14, 2025
1c2a5d4
DOC: Clarify that SPMD is linux-only (#2476)
david-cortes-intel May 16, 2025
a891203
DOC: More details for conda installs (#2475)
david-cortes-intel May 16, 2025
04f7762
chore(deps): update dependency tornado to v6.5 [security] (#2478)
renovate[bot] May 16, 2025
cf87404
Migrate to modern NumPy interface (#2479)
emmanuel-ferdman May 21, 2025
8c8f692
fix: add cstdint include to transceiver for public CI (#2484)
ethanglaser May 21, 2025
2d45c60
chore(deps): update dependency cython to v3.1.1 (#2483)
renovate[bot] May 21, 2025
217d19d
chore(deps): update dependency numpy to v2.2.6 (#2481)
renovate[bot] May 21, 2025
af91e83
chore(deps): update dependency array-api-compat to v1.12.0 (#2477)
renovate[bot] May 21, 2025
86f87c7
downgrade setuptools to v79 for develop mode setup (#2480)
ethanglaser May 21, 2025
095459b
chore(deps): update dependency xgboost to v3.0.2 (#2490)
renovate[bot] May 27, 2025
e46fd42
fix: use upload github action to public site (#2459)
yuejiaointel May 29, 2025
f744a60
ci(workflows): add `OpenSSF Scorecard` GitHub Action (#2499)
homksei May 30, 2025
ff3283a
[CI, bugfix] correct stability issues in CondaRecipes (#2493)
icfaust May 31, 2025
fa91076
[CI] temporarily deselect ```test_dbscan_params_validation``` for gre…
icfaust May 31, 2025
6cb98d7
[enhancement] unify policy creation in pybind11 (#2390)
icfaust May 31, 2025
4dde324
Merge branch 'main' into asan_instr
david-cortes-intel Jun 4, 2025
30dd68f
Update INSTALL.md
david-cortes-intel Jun 4, 2025
b0fa39b
Update INSTALL.md
david-cortes-intel Jun 4, 2025
e91d686
Update INSTALL.md
david-cortes-intel Jun 4, 2025
8c79933
Update INSTALL.md
david-cortes-intel Jun 4, 2025
87786f9
Update INSTALL.md
david-cortes-intel Jun 4, 2025
f3e07a4
Update INSTALL.md
david-cortes-intel Jun 5, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions INSTALL.md
Original file line number Diff line number Diff line change
@@ -214,6 +214,44 @@ python setup.py build --abs-rpath

**Note:** when building `scikit-learn-intelex` from source with this option, it will use the oneDAL library with which it was compiled. oneDAL has dependencies on other libraries such as TBB, which is also distributed as a python package through `pip` and as a `conda` package. By default, a conda environment will first try to load TBB from its own packages if it is installed in the environment, which might cause issues if oneDAL was compiled with a system TBB instead of a conda one. In such cases, it is advised to either uninstall TBB from pip/conda (it will be loaded from the oneDAL library which links to it), or modify the order of search paths in environment variables like `${LD_LIBRARY_PATH}`.

### Building with ASAN

In order to use AddressSanitizer (ASan) together with `scikit-learn-intelex`, it's necessary to:
* Build both oneDAL and scikit-learn-intelex with ASan (otherwise error traces will not be very informative).
* Preload the ASan runtime when executing the Python process that imports `scikit-learn-intelex`.
* Optionally, configure Python to use `malloc` as default allocator to reduce the number of false-positive leak reports.

See the instructions on the oneDAL repository for building the library from source with ASAN enabled:
https://github.com/uxlfoundation/oneDAL/blob/main/INSTALL.md

When building `scikit-learn-intelex`, the system's default compiler is used unless specified otherwise through variables such as `$CXX`. In order to avoid issues with incompatible runtimes of ASan, one might want to change the compiler to ICX if oneDAL was built with ICX (the default for it).

The compiler and flags to build with both ASan and debug symbols can be controlled through environment variables - **assuming a Linux system** (ASan on Windows has not been tested):
```shell
export CC="icx -fsanitize=address -g"
export CXX="icpx -fsanitize=address -g"
```

The ASAN runtime used by ICX is the same as the one by Clang. It's possible to preload the ASan runtime for GNU if that's the system's default through e.g. `LD_PRELOAD=libasan.so` or similar, but to get the same ASan runtime as for oneDAL, one might need to specifically pass the paths from Clang if that's not the system's default compiler:
```shell
export LD_PRELOAD="$(clang -print-file-name=libclang_rt.asan-x86_64.so)"
```

Then, the Python memory allocator can be set to `malloc` like this:
```shell
export PYTHONMALLOC=malloc
```

Putting it all together, the earlier examples building the library in-place and executing a python file with it become as follows:
```shell
source <path to ASan-enabled oneDAL env.sh>
CC="icx -fsanitize=address -g" CXX="icpx -fsanitize=address -g" python setup.py build_ext --inplace --force --abs-rpath
CC="icx -fsanitize=address -g" CXX="icpx -fsanitize=address -g" python setup.py build --abs-rpath
LD_PRELOAD="$(clang -print-file-name=libclang_rt.asan-x86_64.so)" PYTHONMALLOC=malloc PYTHONPATH=$(pwd) python <python file.py>
```

_Be aware that ASan is known to generate many false-positive reports of memory leaks when used with oneDAL, NumPy, and SciPy._

## Build from Sources with `conda-build`

Extension for Scikit-learn* is easily built from the sources using only one command and `conda-build` utility.
Loading
Oops, something went wrong.