CI: add TSAN to the sanitizers pipelines #42444

tkf · 2021-10-01T03:53:46Z

This is built on top of #42440 and run make -C src with TSAN on CI. It is basically the same as #41530 (plus #41675) but for TSAN instead of ASAN.

Making a draft PR against #42440 so that it'd switch the base to master once #42440 is merged.

DilumAluthge · 2021-10-01T04:38:43Z

I'd like to "promote" asan from the experimental pipeline to the regular pipeline.

Should tsan be experimental, or can it go in the regular pipeline?

Keno · 2021-10-01T04:41:10Z

LGTM

tkf · 2021-10-01T05:01:29Z

I don't have a good sense of how stable TSAN is, since I've never actually made it work locally. That's why I was very eager to add CI, to keep it as close as working possible 😛

@Keno @vtjnash @vchuravy I guess you guys know more about if the experimental pipeline is more appropriate for this or not?

Keno · 2021-10-01T05:19:42Z

I don't think we've ever fully been able to run tsan. I think experimental is good for now. Alternatively, we could split it and just have the part that builds the C runtime with tsan in the regular pipeline to ensure that that doesn't regress. Anything beyond that (bootstrapping, maybe running the threads test) can go into the experimental pipeline.

Keno · 2021-10-01T05:21:40Z

contrib/tsan/Make.user.tsan

+override CC=$(BINDIR)/clang
+override CXX=$(TOOLDIR)/clang++
+
+USE_BINARYBUILDER_LLVM=1


This won't work. You need to build LLVM with tsan instrumentation since it uses locks.

Can we still use a normal-build LLVM with deadlock:llvm in the TSAN_OPTIONS=suppressions=... configuration? Or there's no way to work around this fundamentally?

If we need LLVM with TSAN, I guess we'd need to start adding it in BB before trying this in CI. But it'd be a much bigger project than this quick PR.

Yeah I don't want to build LLVM from source in the Base Julia CI. We'd need to have an "LLVM with TSAN" JLL available from Ygg.

Can we still use a normal-build LLVM with deadlock:llvm in the TSAN_OPTIONS=suppressions=... configuration? Or there's no way to work around this fundamentally?

It might be possible, but I'd probably recommend against it. I don't know if that can introduce false positives.

OK, so maybe it makes sense to close this PR until a JLL is ready.

Or, maybe we can still just run the build process (-C src), so that some rudimentary bugs can be caught?

I think a LLVM_tsan_jll makes the most sense. Then it only needs to be built very occasionally in Ygg, instead of building LLVM from source on every PR that is opened to JuliaLang/julia.

I don't really want to support "yet" another version of LLVM in Yggdrasil and slow the update cycle down even more.

Hmmm. That's a fair point. Let's see if @staticfloat has any ideas.

At least we can keep the first part, which doesn't require building LLVM from source.

What's the source of slowness in yet another LLVM JLL vs ccache? Aren't they equivalent in terms of machine time? Is it that, with the JLL approach, there need to be humans involved in the process waiting for the JLL and updating the URL/version/checksum? If that's the case, it can also potentially be solved by more automation?

I kinda liked the JLL approach since it'd make it easier for more casual users to try sanitizers. OTOH, I guess the ccache approach would be more hackable for core devs since it'd give us a "healthy" CI'ed build script that is easy to flip some LLVM build options. Maybe the upside of the ccache approach wins, since there's no single set of build options that covers all the needs when you debugging things.

Is it that, with the JLL approach, there need to be humans involved in the process waiting for the JLL and updating the URL/version/checksum? If that's the case, it can also potentially be solved by more automation?

Yes, the maintenance of the LLVM JLL is a lot of human time, some of it waiting, some of it making sure that all the version number bits line up. Sheparding things through to the registry.

I made attempts at trying to automate it, but currently it is pareto optimal in that I find is really annoying, but not annoying enough to descend to arcane wizardy. c.f JuliaPackaging/Yggdrasil#726 (comment)

.buildkite/pipelines/experimental/misc/sanitizers.yml

… default depot

.buildkite/pipelines/experimental/misc/sanitizers.yml

DilumAluthge · 2021-10-03T16:17:43Z

For now, let's just do the first part, which requires no LLVM build. This will need a rebase on the latest master in order to fix the merge conflicts.

DilumAluthge · 2021-10-03T21:07:55Z

We should remember to squash-merge when merging this.

DilumAluthge · 2021-10-03T21:08:35Z

Assuming that the tsan job passes on Buildkite, this looks good to me to merge.

DilumAluthge · 2021-10-03T21:10:41Z

@tkf What Julia versions will tsan work on? Should we do the same thing we did for asan, and only run it on 1.8 and later? See the asan configuration for details, but basically you add a conditional of the form:

    if: | # We only run the `asan` job on Julia 1.8 and later.
      (pipeline.slug != "julia-release-1-dot-6") && (pipeline.slug != "julia-release-1-dot-7")

tkf · 2021-10-03T21:47:39Z

Yeah, it's #42440 (so 1.8) and later.

(We might be able to find some ranges of commits that TSAN could be built but I think it'd require some serious archaeologies.)

tkf · 2021-10-04T05:54:16Z

I'm getting this error https://buildkite.com/julialang/julia-master/builds/4307#a956be25-a524-450d-b6f2-3c71b7cfb920/167-238

make[1]: Entering directory '/cache/build/amdci7-1/julialang/julia-master/src/support'
# potential confusion with debugging tools, when inspecting a process that has both `libjulia` and `libjulia-internal`
make -C /cache/build/amdci7-1/julialang/julia-master/src/support debug BUILDDIR='/cache/build/amdci7-1/julialang/julia-master/tmp/test-tsan/tsan/src/support'
make: *** No rule to make target '/cache/build/amdci7-1/julialang/julia-master/tmp/test-tsan/tsan/usr/lib/libuv.a', needed by '/cache/build/amdci7-1/julialang/julia-master/tmp/test-tsan/tsan/usr/lib/libjulia-internal-debug.so.1.8'.  Stop.

The error No rule to make target '/.../usr/lib/libuv.a, is a bit puzzling. I wounder if it's due to that I'm using make -C src on an out-of-source build?

DilumAluthge · 2021-10-04T12:45:31Z

Looks like you fixed it?

tkf · 2021-10-04T18:02:50Z

Yeah, it looks like julia-src-debug on the build root did the trick. (I'm guessing we need to pass around some states from the root Makefile to src/Makefile.)

DilumAluthge · 2021-10-04T18:04:57Z

(I'm guessing we need to pass around some states from the root Makefile to src/Makefile.)

Do you want to make those fixes in this PR, or a separate PR?

tkf · 2021-10-04T18:34:33Z

Oh, I was guessing the reason why make julia-src-debug worked in the out-of-tree build while make -C src debug broke something. AFAICT julia-src-debug just wraps make -C src debug

julia/Makefile

Lines 75 to 76 in 5a1b8ce

    
           julia-src-release julia-src-debug : julia-src-% : julia-deps julia_flisp.boot.inc.phony julia-cli-% 
        
           	@$(MAKE) $(QUIET_MAKE) -C $(BUILDROOT)/src $*

I think this PR is good to go, unless you want to double-check julia-src-debug is the correct make target.

DilumAluthge · 2021-10-04T18:47:30Z

unless you want to double-check julia-src-debug is the correct make target

@Keno or @vchuravy Can you confirm that this is the correct target to be building?

Once they've confirmed that, let's merge this.

vchuravy · 2021-10-04T18:55:22Z

Since we are setting override JULIA_BUILD_MODE=debug in Make.user yes that should be right.

(cherry picked from commit f985b47)

tkf requested review from DilumAluthge, Keno and vchuravy October 1, 2021 03:54

tkf mentioned this pull request Oct 1, 2021

Update references to tsan state #42440

Merged

tkf added the ci Continuous integration label Oct 1, 2021

Base automatically changed from kf/tsanstate to master October 1, 2021 04:44

tkf force-pushed the tkf/tsan-ci branch 2 times, most recently from be40313 to dcd5024 Compare October 1, 2021 04:51

Setup CI for TSAN

9913683

tkf force-pushed the tkf/tsan-ci branch from dcd5024 to 9913683 Compare October 1, 2021 04:52

tkf marked this pull request as ready for review October 1, 2021 04:52

tkf requested a review from a team as a code owner October 1, 2021 04:52

Keno reviewed Oct 1, 2021

View reviewed changes

Don't build sysimage with TSAN

6ed9b7f

tkf marked this pull request as draft October 1, 2021 06:13

DilumAluthge requested review from staticfloat and removed request for a team October 2, 2021 21:45

DilumAluthge reviewed Oct 2, 2021

View reviewed changes

.buildkite/pipelines/experimental/misc/sanitizers.yml Outdated Show resolved Hide resolved

in the Buildkite cache, don't cache the "registries" directory in the…

a09febb

… default depot

tkf commented Oct 2, 2021

View reviewed changes

.buildkite/pipelines/experimental/misc/sanitizers.yml Outdated Show resolved Hide resolved

tkf marked this pull request as ready for review October 3, 2021 16:59

Merge branch 'master' into tkf/tsan-ci

b92db8b

tkf requested a review from DilumAluthge October 3, 2021 21:05

DilumAluthge approved these changes Oct 3, 2021

View reviewed changes

We only run the tsan job on Julia 1.8 and later

f22d188

Use build target julia-src-debug

ee11eb4

tkf force-pushed the tkf/tsan-ci branch from 2e36e03 to ee11eb4 Compare October 4, 2021 06:41

DilumAluthge merged commit f985b47 into master Oct 4, 2021

DilumAluthge deleted the tkf/tsan-ci branch October 4, 2021 19:07

KristofferC added the backport 1.6 Change should be backported to release-1.6 label Oct 29, 2021

KristofferC pushed a commit that referenced this pull request Oct 29, 2021

CI: add TSAN to the sanitizers pipelines (#42444)

0b2d317

(cherry picked from commit f985b47)

KristofferC mentioned this pull request Oct 29, 2021

release-1.6: Backports for julia-1.6.4 #42147

Merged

95 tasks

DilumAluthge added the backport 1.7 label Oct 29, 2021

KristofferC pushed a commit that referenced this pull request Oct 31, 2021

CI: add TSAN to the sanitizers pipelines (#42444)

17e60a2

(cherry picked from commit f985b47)

KristofferC mentioned this pull request Nov 8, 2021

release-1.7: Backports for 1.7.0/1.7.0-rc3 #42765

Merged

66 tasks

KristofferC pushed a commit that referenced this pull request Nov 11, 2021

CI: add TSAN to the sanitizers pipelines (#42444)

ca3d9b0

(cherry picked from commit f985b47)

KristofferC pushed a commit that referenced this pull request Nov 12, 2021

CI: add TSAN to the sanitizers pipelines (#42444)

81312d9

(cherry picked from commit f985b47)

KristofferC removed backport 1.6 Change should be backported to release-1.6 backport 1.7 labels Nov 13, 2021

LilithHafner pushed a commit to LilithHafner/julia that referenced this pull request Feb 22, 2022

CI: add TSAN to the sanitizers pipelines (JuliaLang#42444)

1d76f3e

LilithHafner pushed a commit to LilithHafner/julia that referenced this pull request Mar 8, 2022

CI: add TSAN to the sanitizers pipelines (JuliaLang#42444)

8b061e5

staticfloat pushed a commit that referenced this pull request Dec 23, 2022

CI: add TSAN to the sanitizers pipelines (#42444)

bdec76a

(cherry picked from commit f985b47)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: add TSAN to the sanitizers pipelines #42444

CI: add TSAN to the sanitizers pipelines #42444

tkf commented Oct 1, 2021

DilumAluthge commented Oct 1, 2021

Keno commented Oct 1, 2021

tkf commented Oct 1, 2021

Keno commented Oct 1, 2021

Keno Oct 1, 2021

tkf Oct 1, 2021

DilumAluthge Oct 1, 2021

Keno Oct 1, 2021

tkf Oct 1, 2021

DilumAluthge Oct 2, 2021

vchuravy Oct 2, 2021

DilumAluthge Oct 2, 2021

tkf Oct 3, 2021

vchuravy Oct 3, 2021

DilumAluthge commented Oct 3, 2021 •

edited

DilumAluthge commented Oct 3, 2021

DilumAluthge commented Oct 3, 2021

DilumAluthge commented Oct 3, 2021

tkf commented Oct 3, 2021

tkf commented Oct 4, 2021

DilumAluthge commented Oct 4, 2021

tkf commented Oct 4, 2021

DilumAluthge commented Oct 4, 2021

tkf commented Oct 4, 2021

DilumAluthge commented Oct 4, 2021

vchuravy commented Oct 4, 2021

CI: add TSAN to the sanitizers pipelines #42444

CI: add TSAN to the sanitizers pipelines #42444

Conversation

tkf commented Oct 1, 2021

DilumAluthge commented Oct 1, 2021

Keno commented Oct 1, 2021

tkf commented Oct 1, 2021

Keno commented Oct 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DilumAluthge commented Oct 3, 2021 • edited

DilumAluthge commented Oct 3, 2021

DilumAluthge commented Oct 3, 2021

DilumAluthge commented Oct 3, 2021

tkf commented Oct 3, 2021

tkf commented Oct 4, 2021

DilumAluthge commented Oct 4, 2021

tkf commented Oct 4, 2021

DilumAluthge commented Oct 4, 2021

tkf commented Oct 4, 2021

DilumAluthge commented Oct 4, 2021

vchuravy commented Oct 4, 2021

DilumAluthge commented Oct 3, 2021 •

edited