Improve ASAN and TSAN handling in cmake #93147

cyyever · 2023-01-27T10:35:13Z

No description provided.

pytorch-bot · 2023-01-27T10:35:16Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/93147

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 51ea450:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

cyyever · 2023-01-27T10:38:57Z

Now we can test windows VS2022 with ASAN enabled but I don't know how to setup CI targets.

cyyever · 2023-01-27T12:13:46Z

The test failures are due to ASAN finally taking effect, and the old ASAN switch has no effect...:joy:
The failures can be solved by adding ASAN_OPTIONS=detect_container_overflow=0

cyyever · 2023-01-29T01:00:05Z

I have fix some leaks and there remains several JIT indirect leaks to digest. I think enable leak detection by default is not a bad idea now. But I need someone who is familiar with CI to help me setup the related config.

Skylion007 · 2023-01-31T16:28:20Z

@pytorchbot rebase

pytorchmergebot · 2023-01-31T16:30:28Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2023-01-31T16:30:35Z

Successfully rebased google_sanitizer onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout google_sanitizer && git pull --rebase)

Skylion007 · 2023-01-31T17:17:07Z

Hmm, @malfet Any idea what is going on with the build_docs error?

malfet · 2023-02-03T15:20:29Z

Hmm, @malfet Any idea what is going on with the build_docs error?

Just a hiccup, it could not download build artifacts from the previous step. Re-run should fix it (hopefully)

malfet · 2023-02-03T15:26:52Z

@pytorchbot rebase

pytorchmergebot · 2023-02-03T15:28:52Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2023-02-03T15:28:59Z

Successfully rebased google_sanitizer onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout google_sanitizer && git pull --rebase)

cmake/Modules/FindGoogleSanitizer.cmake

cyyever · 2023-02-04T15:46:20Z

@malfet asan tests fail with Indirect leak detected. A big success. The next step is to fix them. But JIT code is more complicated, it will take more time.

Skylion007 · 2023-02-04T16:59:26Z

@malfet asan tests fail with Indirect leak detected. A big success. The next step is to fix them. But JIT code is more complicated, it will take more time.

@cyyever Unfortunately, it looks like most of the leaks are from pybind11::enum which is known to be leaky pybind/pybind11#3865 . If you can find the root cause and fix it, we would gladly accept as a PR in pybind11 though. Otherwise, i suppose we have to add supressions.

cyyever · 2023-02-05T01:21:43Z

@Skylion007 It didn't report pybind11 leaks on 3.10, maybe fixed by official.

cyyever · 2023-03-07T14:08:13Z

@pytorchmergebot merge

pytorchmergebot · 2023-03-07T14:10:08Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Pull Request resolved: pytorch#93147 Approved by: https://github.com/malfet

Pull Request resolved: pytorch/pytorch#93147 Approved by: https://github.com/malfet

Pull Request resolved: pytorch#93147 Approved by: https://github.com/malfet

vfdev-5 · 2023-03-17T18:01:19Z

cmake/Modules/FindSanitizer.cmake

+
+  set(CMAKE_REQUIRED_QUIET ON)
+  set(_run_res 0)
+  include(CheckSourceRuns)


I may be wrong here, but this requires CMake version upper than 3.18 (minimum required specified in pytorch CMakeLists.txt):

pytorch/CMakeLists.txt

Line 1 in 57bb5b1

cmake_minimum_required(VERSION 3.18 FATAL_ERROR)

According to the logs when pytorch-linux-focal-py3-clang7-asan is built, CI is running CMake version 3.22.1.

Should we update it or at least specify cmake_minimum_required(VERSION 3.22 FATAL_ERROR) in this file ?

check_source_runs can be changed to check_cxx_source_runs which exists in 3.18

As indicated by the last comment from PR #93147, we should replace CheckSourceRuns in **cmake/Modules/FindSanitizer.cmake** with older versions to avoid dependency on CMake 3.19+ Pull Request resolved: #97073 Approved by: https://github.com/vfdev-5, https://github.com/Skylion007

If `USE_ASAN` is set, compile FBGEMM with ASAN as well, by setting `USE_SANITIZER` to `address,undefined` This fixes regression in sanitizer coverage introduced by #93147 that change effects of sanitizer from the entire project to just torch libraries

If `USE_ASAN` is set, compile FBGEMM with ASAN as well, by setting `USE_SANITIZER` to `address,undefined` This fixes regression in sanitizer coverage introduced by #93147 that change effects of sanitizer from the entire project to just torch libraries, and finally allows one to reliably catch regression reported in #111189 Pull Request resolved: #111266 Approved by: https://github.com/huydhn

If `USE_ASAN` is set, compile FBGEMM with ASAN as well, by setting `USE_SANITIZER` to `address,undefined` This fixes regression in sanitizer coverage introduced by pytorch#93147 that change effects of sanitizer from the entire project to just torch libraries, and finally allows one to reliably catch regression reported in pytorch#111189 Pull Request resolved: pytorch#111266 Approved by: https://github.com/huydhn

pytorchbot added the open source label Jan 27, 2023

Skylion007 requested a review from malfet January 27, 2023 17:50

ngimel added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jan 28, 2023

Skylion007 added module: infra Relates to CI infrastructure test-config/tsan labels Jan 28, 2023

cyyever force-pushed the google_sanitizer branch 2 times, most recently from b9bc807 to 9eb07d4 Compare January 29, 2023 00:51

cyyever force-pushed the google_sanitizer branch from 9eb07d4 to 9d41318 Compare January 29, 2023 04:58

pytorchmergebot force-pushed the google_sanitizer branch from 9d41318 to f90663f Compare January 31, 2023 16:30

malfet removed the test-config/tsan label Feb 3, 2023

pytorchmergebot force-pushed the google_sanitizer branch from f90663f to 049fe3b Compare February 3, 2023 15:29

malfet reviewed Feb 3, 2023

View reviewed changes

cmake/Modules/FindGoogleSanitizer.cmake Outdated Show resolved Hide resolved

malfet reviewed Feb 3, 2023

View reviewed changes

cmake/Modules/FindGoogleSanitizer.cmake Outdated Show resolved Hide resolved

cyyever force-pushed the google_sanitizer branch 2 times, most recently from f222675 to 0246128 Compare February 4, 2023 11:58

cyyever added 9 commits March 7, 2023 17:33

don't expose tsan to other tools

041bba1

add ubsan flags option

6129440

add leak suppression

b988509

link to ASAN as a shared library

af0eed8

enable some checks

660d0fe

remove unnecessary CMake checks

121f1db

disable LSAN

3e48962

rename option

08df819

rename module and remove leak suppresions

51ea450

cyyever force-pushed the google_sanitizer branch from 851207a to 51ea450 Compare March 7, 2023 09:34

pytorchmergebot added the Merged label Mar 7, 2023

pytorchmergebot closed this in 666efd8 Mar 7, 2023

cyyever mentioned this pull request Mar 7, 2023

enable LSAN #96196

Closed

ydwu4 pushed a commit to ydwu4/pytorch that referenced this pull request Mar 10, 2023

Improve ASAN and TSAN handling in cmake (pytorch#93147)

9d87867

Pull Request resolved: pytorch#93147 Approved by: https://github.com/malfet

cyyever added a commit to cyyever/pytorch_private that referenced this pull request Mar 12, 2023

Improve ASAN and TSAN handling in cmake (#93147)

6857f3e

Pull Request resolved: pytorch/pytorch#93147 Approved by: https://github.com/malfet

cyyever added a commit to cyyever/pytorch_private that referenced this pull request Mar 12, 2023

Improve ASAN and TSAN handling in cmake (#93147)

4cb0ba7

Pull Request resolved: pytorch/pytorch#93147 Approved by: https://github.com/malfet

ydwu4 added a commit to ydwu4/pytorch that referenced this pull request Mar 13, 2023

Improve ASAN and TSAN handling in cmake (pytorch#93147)

2d3c315

Pull Request resolved: pytorch#93147 Approved by: https://github.com/malfet

vfdev-5 reviewed Mar 17, 2023

View reviewed changes

cyyever mentioned this pull request Mar 18, 2023

lower cmake version requirement in FindSanitizer.cmake #97073

Closed

cyyever deleted the google_sanitizer branch August 8, 2023 05:08

malfet mentioned this pull request Oct 14, 2023

[BE] Compile FBGEMM with ASAN #111266

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve ASAN and TSAN handling in cmake #93147

Improve ASAN and TSAN handling in cmake #93147

cyyever commented Jan 27, 2023

pytorch-bot bot commented Jan 27, 2023 •

edited

Loading

cyyever commented Jan 27, 2023 •

edited

Loading

cyyever commented Jan 27, 2023 •

edited

Loading

cyyever commented Jan 29, 2023 •

edited

Loading

Skylion007 commented Jan 31, 2023

pytorchmergebot commented Jan 31, 2023

pytorchmergebot commented Jan 31, 2023

Skylion007 commented Jan 31, 2023

malfet commented Feb 3, 2023

malfet commented Feb 3, 2023

pytorchmergebot commented Feb 3, 2023

pytorchmergebot commented Feb 3, 2023

cyyever commented Feb 4, 2023 •

edited

Loading

Skylion007 commented Feb 4, 2023 •

edited

Loading

cyyever commented Feb 5, 2023

cyyever commented Mar 7, 2023

pytorchmergebot commented Mar 7, 2023

vfdev-5 Mar 17, 2023

cyyever Mar 18, 2023

Improve ASAN and TSAN handling in cmake #93147

Improve ASAN and TSAN handling in cmake #93147

Conversation

cyyever commented Jan 27, 2023

pytorch-bot bot commented Jan 27, 2023 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/93147

✅ No Failures

cyyever commented Jan 27, 2023 • edited Loading

cyyever commented Jan 27, 2023 • edited Loading

cyyever commented Jan 29, 2023 • edited Loading

Skylion007 commented Jan 31, 2023

pytorchmergebot commented Jan 31, 2023

pytorchmergebot commented Jan 31, 2023

Skylion007 commented Jan 31, 2023

malfet commented Feb 3, 2023

malfet commented Feb 3, 2023

pytorchmergebot commented Feb 3, 2023

pytorchmergebot commented Feb 3, 2023

cyyever commented Feb 4, 2023 • edited Loading

Skylion007 commented Feb 4, 2023 • edited Loading

cyyever commented Feb 5, 2023

cyyever commented Mar 7, 2023

pytorchmergebot commented Mar 7, 2023

Merge started

vfdev-5 Mar 17, 2023

Choose a reason for hiding this comment

cyyever Mar 18, 2023

Choose a reason for hiding this comment

pytorch-bot bot commented Jan 27, 2023 •

edited

Loading

cyyever commented Jan 27, 2023 •

edited

Loading

cyyever commented Jan 27, 2023 •

edited

Loading

cyyever commented Jan 29, 2023 •

edited

Loading

cyyever commented Feb 4, 2023 •

edited

Loading

Skylion007 commented Feb 4, 2023 •

edited

Loading