Flaky or bugged async-hooks/test-zlib.zlib-binding.deflate #20907

apapirovski · 2018-05-23T05:40:42Z

For a good while now, this test has been flaky or perhaps broken on a number of the Linux platforms. Example failure:

00:25:23 not ok 57 async-hooks/test-zlib.zlib-binding.deflate
00:25:23   ---
00:25:23   duration_ms: 0.165
00:25:23   severity: crashed
00:25:23   exitcode: -11
00:25:23   stack: |-

but also the more thorough:

*** Error in `out/Release/node': malloc(): smallbin double linked list corrupted: 0x0000000011d104b0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x8c9d2)[0x3ff87a8c9d2]
/lib64/libc.so.6(+0x903b0)[0x3ff87a903b0]
/lib64/libc.so.6(__libc_malloc+0x70)[0x3ff87a92850]
/data/gcc-4.9/lib64/libstdc++.so.6(_Znwm+0x34)[0x3ff87e7d994]
out/Release/node(_ZNSt6vectorIN2v88internal10ParserBaseINS1_6ParserEE24DeclarationParsingResult11DeclarationESaIS6_EE19_M_emplace_back_auxIIRKS6_EEEvDpOT_+0x5e)[0x1d7e59e]
out/Release/node(_ZN2v88internal10ParserBaseINS0_6ParserEE25ParseVariableDeclarationsENS3_26VariableDeclarationContextEPNS3_24DeclarationParsingResultEPNS0_8ZoneListIPKNS0_12AstRawStringEEEPb+0xb60)[0x1d88788]
out/Release/node(_ZN2v88internal10ParserBaseINS0_6ParserEE17ParseForStatementEPNS0_8ZoneListIPKNS0_12AstRawStringEEEPb+0x42c)[0x1d9bac4]
out/Release/node(_ZN2v88internal10ParserBaseINS0_6ParserEE14ParseStatementEPNS0_8ZoneListIPKNS0_12AstRawStringEEENS0_30AllowLabelledFunctionStatementEPb+0x2b8)[0x1d83c48]
out/Release/node(_ZN2v88internal10ParserBaseINS0_6ParserEE18ParseStatementListEPNS0_8ZoneListIPNS0_9StatementEEENS0_5Token5ValueEbPb+0x9e)[0x1d83ffe]
out/Release/node(_ZN2v88internal10ParserBaseINS0_6ParserEE17ParseFunctionBodyEPNS0_8ZoneListIPNS0_9StatementEEEPKNS0_12AstRawStringEiRKNS0_22ParserFormalParametersENS0_12FunctionKindENS0_15FunctionLiteral12FunctionTypeEPb+0x10e)[0x1d85006]
out/Release/node(_ZN2v88internal6Parser13ParseFunctionEPKNS0_12AstRawStringEiNS0_12FunctionKindENS0_15FunctionLiteral12FunctionTypeEPNS0_16DeclarationScopeEPiSA_PbSA_SA_PNS0_8ZoneListIS4_EESB_+0x2c0)[0x1d89808]
out/Release/node(_ZN2v88internal6Parser20ParseFunctionLiteralEPKNS0_12AstRawStringENS0_7Scanner8LocationENS0_20FunctionNameValidityENS0_12FunctionKindEiNS0_15FunctionLiteral12FunctionTypeENS0_12LanguageModeEPNS0_8ZoneListIS4_EEPb+0x464)[0x1d8a224]
out/Release/node(_ZN2v88internal6Parser15DoParseFunctionEPNS0_9ParseInfoEPKNS0_12AstRawStringE+0x16c)[0x1d8ef64]
out/Release/node(_ZN2v88internal6Parser13ParseFunctionEPNS0_7IsolateEPNS0_9ParseInfoENS0_6HandleINS0_18SharedFunctionInfoEEE+0x252)[0x1d8f992]
out/Release/node(_ZN2v88internal7parsing13ParseFunctionEPNS0_9ParseInfoENS0_6HandleINS0_18SharedFunctionInfoEEEPNS0_7IsolateE+0x14a)[0x1d9cf7a]
out/Release/node(_ZN2v88internal8Compiler7CompileENS0_6HandleINS0_18SharedFunctionInfoEEENS1_18ClearExceptionFlagE+0x1f4)[0x1aac4ac]
out/Release/node(_ZN2v88internal8Compiler7CompileENS0_6HandleINS0_10JSFunctionEEENS1_18ClearExceptionFlagE+0x230)[0x1ab1d30]
out/Release/node(_ZN2v88internal19Runtime_CompileLazyEiPPNS0_6ObjectEPNS0_7IsolateE+0x9c)[0x1e6c634]
[0x9507e84474]
======= Memory map: ========
01000000-03086000 r-xp 00000000 5e:05 1057338                            /data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/out/Release/node
03087000-03089000 r--p 02086000 5e:05 1057338                            /data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/out/Release/node
03089000-030a0000 rw-p 02088000 5e:05 1057338                            /data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/out/Release/node
030a0000-030b6000 rw-p 00000000 00:00 0
11c80000-11da5000 rw-p 00000000 00:00 0                                  [heap]
17dfa80000-17dfb00000 rw-p 00000000 00:00 0
2516100000-2516180000 rw-p 00000000 00:00 0
4a2cd80000-4a2ce00000 rw-p 00000000 00:00 0
533a380000-533a400000 rw-p 00000000 00:00 0
6171080000-6171100000 rw-p 00000000 00:00 0
648e000000-648e080000 rw-p 00000000 00:00 0
6a4a280000-6a4a300000 rw-p 00000000 00:00 0
6af8280000-6af8300000 rw-p 00000000 00:00 0
6b60d80000-6b60e00000 rw-p 00000000 00:00 0
8bf1600000-8bf1680000 rw-p 00000000 00:00 0
8fb8380000-8fb8400000 rw-p 00000000 00:00 0
9507e61000-9507e80000 ---p 00000000 00:00 0
9507e80000-9507e83000 rw-p 00000000 00:00 0
9507e83000-9507e84000 ---p 00000000 00:00 0
9507e84000-9507eff000 rwxp 00000000 00:00 0
9507eff000-9507f00000 ---p 00000000 00:00 0
9507f00000-9507f03000 rw-p 00000000 00:00 0
9507f03000-9507f04000 ---p 00000000 00:00 0
9507f04000-9507f7f000 rwxp 00000000 00:00 0
9507f7f000-9507f80000 ---p 00000000 00:00 0
9507f80000-9507f83000 rw-p 00000000 00:00 0
9507f83000-9507f84000 ---p 00000000 00:00 0
9507f84000-9507fff000 rwxp 00000000 00:00 0
9507fff000-9527e61000 ---p 00000000 00:00 0
959d180000-959d185000 rw-p 00000000 00:00 0
a5c0500000-a5c054a000 rw-p 00000000 00:00 0
bdcfced000-bdcfcf0000 ---p 00000000 00:00 0
bdcfcf0000-bdcfcf8000 rw-p 00000000 00:00 0
bdcfcf8000-bdcfcf9000 ---p 00000000 00:00 0
c0a8880000-c0a8900000 rw-p 00000000 00:00 0
caccf80000-cacd000000 rw-p 00000000 00:00 0
ce0d480000-ce0d500000 rw-p 00000000 00:00 0
da4c200000-da4c280000 rw-p 00000000 00:00 0
f33e100000-f33e180000 rw-p 00000000 00:00 0
3ff6c000000-3ff6c021000 rw-p 00000000 00:00 0
3ff6c021000-3ff70000000 ---p 00000000 00:00 0
3ff70000000-3ff70021000 rw-p 00000000 00:00 0
3ff70021000-3ff74000000 ---p 00000000 00:00 0
3ff77800000-3ff77801000 ---p 00000000 00:00 0
3ff77801000-3ff78000000 rw-p 00000000 00:00 0
3ff78000000-3ff78021000 rw-p 00000000 00:00 0
3ff78021000-3ff7c000000 ---p 00000000 00:00 0
3ff7c000000-3ff7c021000 rw-p 00000000 00:00 0
3ff7c021000-3ff80000000 ---p 00000000 00:00 0
3ff80000000-3ff80021000 rw-p 00000000 00:00 0
3ff80021000-3ff84000000 ---p 00000000 00:00 0
3ff84200000-3ff84201000 ---p 00000000 00:00 0
3ff84201000-3ff84a00000 rw-p 00000000 00:00 0
3ff84a00000-3ff84a01000 ---p 00000000 00:00 0
3ff84a01000-3ff85200000 rw-p 00000000 00:00 0
3ff85200000-3ff85201000 ---p 00000000 00:00 0
3ff85201000-3ff85a00000 rw-p 00000000 00:00 0
3ff85a00000-3ff85a01000 ---p 00000000 00:00 0
3ff85a01000-3ff86200000 rw-p 00000000 00:00 0
3ff86200000-3ff86201000 ---p 00000000 00:00 0
3ff86201000-3ff86a00000 rw-p 00000000 00:00 0
3ff86a00000-3ff86a01000 ---p 00000000 00:00 0
3ff86a01000-3ff87200000 rw-p 00000000 00:00 0
3ff87200000-3ff87201000 ---p 00000000 00:00 0
3ff87201000-3ff87a00000 rw-p 00000000 00:00 0
3ff87a00000-3ff87ba6000 r-xp 00000000 5e:02 5039                         /usr/lib64/libc-2.17.so
3ff87ba6000-3ff87baa000 r--p 001a5000 5e:02 5039                         /usr/lib64/libc-2.17.so
3ff87baa000-3ff87bad000 rw-p 001a9000 5e:02 5039                         /usr/lib64/libc-2.17.so
3ff87bad000-3ff87bb0000 rw-p 00000000 00:00 0
3ff87c00000-3ff87c18000 r-xp 00000000 5e:02 5071                         /usr/lib64/libpthread-2.17.so
3ff87c18000-3ff87c19000 r--p 00017000 5e:02 5071                         /usr/lib64/libpthread-2.17.so
3ff87c19000-3ff87c1a000 rw-p 00018000 5e:02 5071                         /usr/lib64/libpthread-2.17.so
3ff87c1a000-3ff87c1e000 rw-p 00000000 00:00 0
3ff87c80000-3ff87c92000 r-xp 00000000 5e:05 6423382                      /data/gcc-4.9/lib64/libgcc_s.so.1
3ff87c92000-3ff87c93000 r--p 00011000 5e:05 6423382                      /data/gcc-4.9/lib64/libgcc_s.so.1
3ff87c93000-3ff87c94000 rw-p 00012000 5e:05 6423382                      /data/gcc-4.9/lib64/libgcc_s.so.1
3ff87d00000-3ff87da8000 r-xp 00000000 5e:02 29820                        /usr/lib64/libm-2.17.so
3ff87da8000-3ff87da9000 r--p 000a7000 5e:02 29820                        /usr/lib64/libm-2.17.so
3ff87da9000-3ff87daa000 rw-p 000a8000 5e:02 29820                        /usr/lib64/libm-2.17.so
3ff87e00000-3ff87f30000 r-xp 00000000 5e:05 6424618                      /data/gcc-4.9/lib64/libstdc++.so.6.0.20
3ff87f30000-3ff87f31000 ---p 00130000 5e:05 6424618                      /data/gcc-4.9/lib64/libstdc++.so.6.0.20
3ff87f31000-3ff87f37000 r--p 00130000 5e:05 6424618                      /data/gcc-4.9/lib64/libstdc++.so.6.0.20
3ff87f37000-3ff87f3a000 rw-p 00136000 5e:05 6424618                      /data/gcc-4.9/lib64/libstdc++.so.6.0.20
3ff87f3a000-3ff87f4f000 rw-p 00000000 00:00 0
3ff87f80000-3ff87f88000 r-xp 00000000 5e:02 29842                        /usr/lib64/librt-2.17.so
3ff87f88000-3ff87f89000 r--p 00007000 5e:02 29842                        /usr/lib64/librt-2.17.so
3ff87f89000-3ff87f8a000 rw-p 00008000 5e:02 29842                        /usr/lib64/librt-2.17.so
3ff88000000-3ff88003000 r-xp 00000000 5e:02 5049                         /usr/lib64/libdl-2.17.so
3ff88003000-3ff88004000 r--p 00002000 5e:02 5049                         /usr/lib64/libdl-2.17.so
3ff88004000-3ff88005000 rw-p 00003000 5e:02 5049                         /usr/lib64/libdl-2.17.so
3ff88100000-3ff88123000 r-xp 00000000 5e:02 2626                         /usr/lib64/ld-2.17.so
3ff88123000-3ff88124000 r--p 00022000 5e:02 2626                         /usr/lib64/ld-2.17.so
3ff88124000-3ff88125000 rw-p 00023000 5e:02 2626                         /usr/lib64/ld-2.17.so
3ff88125000-3ff88126000 rw-p 00000000 00:00 0
3ff8816f000-3ff88170000 rw-p 00000000 00:00 0
3ff88170000-3ff88171000 ---p 00000000 00:00 0
3ff88171000-3ff8817e000 rw-p 00000000 00:00 0
3ff8817e000-3ff88180000 r-xp 00000000 00:00 0                            [vdso]
3fffc1df000-3fffc200000 rw-p 00000000 00:00 0                            [stack]

The text was updated successfully, but these errors were encountered:

apapirovski · 2018-05-23T05:41:03Z

cc @nodejs/zlib @nodejs/async_hooks

addaleax · 2018-05-23T16:11:31Z

I think this is going to be nearly impossible to debug without having access to a machine on which this is reproducible. /cc @nodejs/build

Locally everything seems fine to me. As always with these kinds of issues, it would be great to have valgrind output (python tools/test.py --valgrind --repeat=100 -J async-hooks/test-zlib.zlib-binding.deflate) or core dumps for a crash available…

apapirovski · 2018-05-23T16:13:00Z

Yeah... I tried running my own machine with a similar setup but couldn't reproduce the crash or find anything with valgrind. Sounds like we will definitely need access.

addaleax · 2018-05-23T16:15:40Z

Maybe somebody like @Trott can ssh into one of these machines and run that command? That might be a huge help for finding a point to start. (Maybe first share the output privately though, if there is any, since memory corruption bugs sometimes can have security implications…)

maclover7 · 2018-05-23T21:20:01Z

@apapirovski @addaleax What would be the best way for build to help debug this? To run valgrind and then share the output privately?

addaleax · 2018-05-23T21:24:09Z

@maclover7 That is at least worth the attempt, yes

Trott · 2018-05-23T21:58:12Z

I'm doing a bisect to find the commit that caused this.

https://ci.nodejs.org/job/node-test-commit-linux/19012/ shows that 4f0ab76 is good. So it's a commit that landed after that.

maclover7 · 2018-05-23T22:15:19Z

Here's an interesting thing I found in Jenkins, not sure if helpful or not: https://ci.nodejs.org/job/node-test-commit-linuxone/1556/nodes=rhel72-s390x/testReport/(root)/test/async_hooks_test_zlib_zlib_binding_deflate/

Trott · 2018-05-23T22:28:45Z

According to @mcollina, this bug will be fixed with #19989.

mcollina · 2018-05-23T22:31:54Z

@Trott I think I have been misunderstood, I do not think this will be fixed by that update. I will test it.
It is very likely that it is: #20516.

Trott · 2018-05-23T22:48:50Z

OK, then, back to bisecting!

3654cd4 is good, so whatever caused this issue landed after that. https://ci.nodejs.org/job/node-test-commit-linux/19017/

mcollina · 2018-05-23T23:00:13Z

@Trott then that's definitely not the bug I'm talking about.

Trott · 2018-05-23T23:36:10Z

a5aad24 was green so it is a commit that landed after that.
https://ci.nodejs.org/job/node-test-commit-linux/19019/

Trott · 2018-05-23T23:44:12Z

23a56e0 is bad so it was either that commit or else 6f6f7f7. https://ci.nodejs.org/job/node-test-commit-linux/19021/

Trott · 2018-05-24T00:08:44Z

It's 23a56e0. 6f6f7f7 came up clean at: https://ci.nodejs.org/job/node-test-commit-linux/19023/

Trott · 2018-05-24T00:13:50Z

PR to revert the commit that seems to cause (or merely expose or whatever) the issue: #20919

apapirovski · 2018-05-24T06:10:50Z

~~FWIW I'm now roughly 80% certain it's tracing related... without access to the machine this has been a bit tough to debug but I'm almost there.~~

Ok, maybe not tracing but certainly related code... looks like maybe threads?

apapirovski · 2018-05-24T06:50:54Z

Also if someone wants to volunteer to run valgrind that would still be helpful. My progress is coming along but it's slow and painful.

apapirovski · 2018-05-24T09:36:01Z

Ok, so at least on one system it segfaults when it's calling uv_thread_join here:

node/src/node_platform.cc

Lines 55 to 60 in 531b4be

    
           void BackgroundTaskRunner::Shutdown() { 
        
             background_tasks_.Stop(); 
        
             for (size_t i = 0; i < threads_.size(); i++) { 
        
               CHECK_EQ(0, uv_thread_join(threads_[i].get())); 
        
             } 
        
           }

Anyone have any ideas? Perhaps @bnoordhuis?

Edit: There's not a predictable thread that it segfaults on. Sometimes it's the 2nd, sometimes 3rd.

addaleax · 2018-05-24T10:57:44Z

An issue with the NodePlatform implementation would be consistent with a GC timing changes… it doesn’t look like that crash location gives us a hint about the actual issue is, though :/

test-zlib.zlib-binding.deflate is failing continuously in our CI, leaving us with 1% successful builds during the last 100 runs. This commit marks the test as flaky while the issue is not resolved. Ref: nodejs#20907

apapirovski · 2018-05-24T14:51:46Z

@addaleax FWIW I still think trace events could be the culprit. We occasionally see a segfault on our Windows machines in the test-trace-events-api test.

apapirovski · 2018-05-24T14:58:08Z

Nope. Found the cause, I think? Working on a fix.

Edit: nope, just changed the GC timing again... :|

apapirovski · 2018-05-25T05:37:18Z

Taking a break from this today. If anyone from @nodejs/build or otherwise can get access to run valgrind, I would be ecstatic. Otherwise debugging resumes at some point this weekend...

Trott · 2018-05-25T17:53:07Z

@apapirovski You can request temporary access to one of the hosts if you'd like to be able to SSH in and run valgrind yourself. To request temporary access, open an issue in the build repo explaining what host you want access to and why. I suspect your request would be approved quickly.

Alternatively, I can take a host out of the Jenkins pool and run valgrind on it for you. Do you need any CLI flags set (like --leak-check=yes)?

addaleax · 2018-05-25T17:56:33Z

Alternatively, I can take a host out of the Jenkins pool and run valgrind on it for you. Do you need any CLI flags set (like --leak-check=yes)?

@Trott I’d first run it without, and if it suggests something near the end of the output (in particular --track-origins=yes) I’d run it again with that.

Trott · 2018-05-25T22:42:03Z

I can reproduce this using test.py but not running straight from command line with ./node test/async-hooks/test-zlib.zlib-binding.deflate.sj.

Trott · 2018-05-25T22:57:23Z

The test fails with tools/test.py test/async-hooks/test-zlib.zlib-binding.deflate.js but passes with tools/test.py --valgrind test/async-hooks/test-zlib.zlib-binding.deflate.js.

[iojs@test-rackspace-fedora27-x64-1 fedora-latest-x64]$ tools/test.py test/async-hooks/test-zlib.zlib-binding.deflate.js 
=== release test-zlib.zlib-binding.deflate ===                    
Path: async-hooks/test-zlib.zlib-binding.deflate
Command: out/Release/node /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/test/async-hooks/test-zlib.zlib-binding.deflate.js
--- CRASHED (Signal: 11) ---
[00:00|% 100|+   0|-   1]: Done                                  
[iojs@test-rackspace-fedora27-x64-1 fedora-latest-x64]$ tools/test.py --valgrind test/async-hooks/test-zlib.zlib-binding.deflate.js 
[00:14|% 100|+   1|-   0]: Done                                   
[iojs@test-rackspace-fedora27-x64-1 fedora-latest-x64]$

Trott · 2018-05-25T23:04:46Z

At @addaleax's suggestion, I was able to get it to fail using --tool=helgrind. Hope this helps! (She also indicated that there's a good likelihood that errors will be false positives with --tool=helgrind so...I sure hope not. :-D )

iojs@test-rackspace-fedora27-x64-1 fedora-latest-x64]$ tools/test.py --valgrind test/async-hooks/test-zlib.zlib-binding.deflate.js 
=== release test-zlib.zlib-binding.deflate ===                    
Path: async-hooks/test-zlib.zlib-binding.deflate
==32152== Helgrind, a thread error detector
==32152== Copyright (C) 2007-2017, and GNU GPL'd, by OpenWorks LLP et al.
==32152== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==32152== Command: /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/test/async-hooks/test-zlib.zlib-binding.deflate.js
==32152== 
==32152== ---Thread-Announcement------------------------------------------
==32152== 
==32152== Thread #3 was created
==32152==    at 0x5E4615E: clone (in /usr/lib64/libc-2.26.so)
==32152==    by 0x5B39281: create_thread (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x5B3ACDF: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x4C337ED: pthread_create_WRK (hg_intercepts.c:427)
==32152==    by 0x4C348DF: pthread_create@* (hg_intercepts.c:460)
==32152==    by 0x9AC4DE: uv_thread_create (thread.c:202)
==32152==    by 0x8DBEEC: node::BackgroundTaskRunner::BackgroundTaskRunner(int) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x8DC062: node::NodePlatform::NodePlatform(int, v8::TracingController*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883ED7: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152== 
==32152== ---Thread-Announcement------------------------------------------
==32152== 
==32152== Thread #2 was created
==32152==    at 0x5E4615E: clone (in /usr/lib64/libc-2.26.so)
==32152==    by 0x5B39281: create_thread (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x5B3ACDF: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x4C337ED: pthread_create_WRK (hg_intercepts.c:427)
==32152==    by 0x4C348DF: pthread_create@* (hg_intercepts.c:460)
==32152==    by 0x9AC4DE: uv_thread_create (thread.c:202)
==32152==    by 0x8DBEEC: node::BackgroundTaskRunner::BackgroundTaskRunner(int) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x8DC062: node::NodePlatform::NodePlatform(int, v8::TracingController*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883ED7: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152== 
==32152== ----------------------------------------------------------------
==32152== 
==32152== Possible data race during read of size 8 at 0x238BF68 by thread #3
==32152== Locks held: none
==32152==    at 0x8D9970: node::BackgroundRunner(void*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x4C339E1: mythread_wrapper (hg_intercepts.c:389)
==32152==    by 0x5B3A50A: start_thread (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x5E4616E: clone (in /usr/lib64/libc-2.26.so)
==32152== 
==32152== This conflicts with a previous write of size 8 by thread #2
==32152== Locks held: none
==32152==    at 0x8D9B46: node::BackgroundRunner(void*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x4C339E1: mythread_wrapper (hg_intercepts.c:389)
==32152==    by 0x5B3A50A: start_thread (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x5E4616E: clone (in /usr/lib64/libc-2.26.so)
==32152==  Address 0x238bf68 is 0 bytes inside data symbol "_ZZN4nodeL16BackgroundRunnerEPvE27trace_event_unique_atomic21"
==32152== 
==32152== ---Thread-Announcement------------------------------------------
==32152== 
==32152== Thread #1 is the program's root thread
==32152== 
==32152== ---Thread-Announcement------------------------------------------
==32152== 
==32152== Thread #4 was created
==32152==    at 0x5E4615E: clone (in /usr/lib64/libc-2.26.so)
==32152==    by 0x5B39281: create_thread (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x5B3ACDF: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x4C337ED: pthread_create_WRK (hg_intercepts.c:427)
==32152==    by 0x4C348DF: pthread_create@* (hg_intercepts.c:460)
==32152==    by 0x9AC4DE: uv_thread_create (thread.c:202)
==32152==    by 0x8DBEEC: node::BackgroundTaskRunner::BackgroundTaskRunner(int) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x8DC062: node::NodePlatform::NodePlatform(int, v8::TracingController*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883ED7: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152== 
==32152== ----------------------------------------------------------------
==32152== 
==32152==  Lock at 0x611FB38 was first observed
==32152==    at 0x4C34CA3: pthread_mutex_init (hg_intercepts.c:787)
==32152==    by 0xE6263B: v8::internal::Heap::Heap() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF36C0E: v8::internal::Isolate::Isolate(bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA7B170: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Address 0x611fb38 is 6,680 bytes inside a block of size 24,328 alloc'd
==32152==    at 0x4C2E5DA: operator new(unsigned long) (vg_replace_malloc.c:334)
==32152==    by 0xA7B163: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Block was alloc'd by thread #1
==32152== 
==32152==  Lock at 0x6124E20 was first observed
==32152==    at 0x4C34CA3: pthread_mutex_init (hg_intercepts.c:787)
==32152==    by 0x119DA88: v8::internal::ThreadManager::ThreadManager() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF3744A: v8::internal::Isolate::Isolate(bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA7B170: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Address 0x6124e20 is 0 bytes inside a block of size 80 alloc'd
==32152==    at 0x4C2E5DA: operator new(unsigned long) (vg_replace_malloc.c:334)
==32152==    by 0xF3743F: v8::internal::Isolate::Isolate(bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA7B170: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Block was alloc'd by thread #1
==32152== 
==32152==  Lock at 0x1FFEFFE020 was first observed
==32152==    at 0x4C34CA3: pthread_mutex_init (hg_intercepts.c:787)
==32152==    by 0xE76766: v8::internal::Heap::Scavenge() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7B5BF: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7BD03: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DC28: v8::internal::Handle<v8::internal::ConsString> v8::internal::Factory::New<v8::internal::ConsString>(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationSpace) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DEBD: v8::internal::Factory::NewConsString(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, int, bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2E5B5: v8::internal::Factory::NewConsString(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x118D24A: v8::internal::IncrementalStringBuilder::AppendString(v8::internal::Handle<v8::internal::String>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF756FA: v8::internal::(anonymous namespace)::AppendFileLocation(v8::internal::Isolate*, v8::internal::StackFrameBase*, v8::internal::IncrementalStringBuilder*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF75F7D: v8::internal::JSStackFrame::ToString() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF79053: v8::internal::ErrorUtils::FormatStackTrace(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA4D970: v8::internal::Accessors::ErrorStackGetter(v8::Local<v8::Name>, v8::PropertyCallbackInfo<v8::Value> const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==  Address 0x1ffeffe020 is on thread #1's stack
==32152==  in frame #4, created by v8::internal::Heap::Scavenge() (???:)
==32152== 
==32152== Possible data race during read of size 8 at 0x1FFEFFE048 by thread #1
==32152== Locks held: 2, at addresses 0x611FB38 0x6124E20
==32152==    at 0xEC1725: v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE6140E: v8::internal::ScavengingTask::RunInParallel() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE8A89E: v8::internal::ItemParallelJob::Task::RunInternal() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE8B53C: v8::internal::ItemParallelJob::Run(std::shared_ptr<v8::internal::Counters>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE762DD: v8::internal::Heap::Scavenge() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7B5BF: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7BD03: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DC28: v8::internal::Handle<v8::internal::ConsString> v8::internal::Factory::New<v8::internal::ConsString>(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationSpace) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DEBD: v8::internal::Factory::NewConsString(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, int, bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2E5B5: v8::internal::Factory::NewConsString(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x118D24A: v8::internal::IncrementalStringBuilder::AppendString(v8::internal::Handle<v8::internal::String>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF756FA: v8::internal::(anonymous namespace)::AppendFileLocation(v8::internal::Isolate*, v8::internal::StackFrameBase*, v8::internal::IncrementalStringBuilder*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152== 
==32152== This conflicts with a previous write of size 8 by thread #4
==32152== Locks held: 1, at address 0x1FFEFFE020
==32152==    at 0xEC2A86: v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE6140E: v8::internal::ScavengingTask::RunInParallel() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE8A89E: v8::internal::ItemParallelJob::Task::RunInternal() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x8D9A34: node::BackgroundRunner(void*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x4C339E1: mythread_wrapper (hg_intercepts.c:389)
==32152==    by 0x5B3A50A: start_thread (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x5E4616E: clone (in /usr/lib64/libc-2.26.so)
==32152==  Address 0x1ffeffe048 is on thread #1's stack
==32152==  in frame #4, created by v8::internal::Heap::Scavenge() (???:)
==32152== 
==32152== ----------------------------------------------------------------
==32152== 
==32152==  Lock at 0x611FB38 was first observed
==32152==    at 0x4C34CA3: pthread_mutex_init (hg_intercepts.c:787)
==32152==    by 0xE6263B: v8::internal::Heap::Heap() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF36C0E: v8::internal::Isolate::Isolate(bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA7B170: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Address 0x611fb38 is 6,680 bytes inside a block of size 24,328 alloc'd
==32152==    at 0x4C2E5DA: operator new(unsigned long) (vg_replace_malloc.c:334)
==32152==    by 0xA7B163: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Block was alloc'd by thread #1
==32152== 
==32152==  Lock at 0x6124E20 was first observed
==32152==    at 0x4C34CA3: pthread_mutex_init (hg_intercepts.c:787)
==32152==    by 0x119DA88: v8::internal::ThreadManager::ThreadManager() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF3744A: v8::internal::Isolate::Isolate(bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA7B170: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Address 0x6124e20 is 0 bytes inside a block of size 80 alloc'd
==32152==    at 0x4C2E5DA: operator new(unsigned long) (vg_replace_malloc.c:334)
==32152==    by 0xF3743F: v8::internal::Isolate::Isolate(bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA7B170: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Block was alloc'd by thread #1
==32152== 
==32152==  Lock at 0x1FFEFFE020 was first observed
==32152==    at 0x4C34CA3: pthread_mutex_init (hg_intercepts.c:787)
==32152==    by 0xE76766: v8::internal::Heap::Scavenge() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7B5BF: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7BD03: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DC28: v8::internal::Handle<v8::internal::ConsString> v8::internal::Factory::New<v8::internal::ConsString>(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationSpace) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DEBD: v8::internal::Factory::NewConsString(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, int, bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2E5B5: v8::internal::Factory::NewConsString(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x118D24A: v8::internal::IncrementalStringBuilder::AppendString(v8::internal::Handle<v8::internal::String>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF756FA: v8::internal::(anonymous namespace)::AppendFileLocation(v8::internal::Isolate*, v8::internal::StackFrameBase*, v8::internal::IncrementalStringBuilder*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF75F7D: v8::internal::JSStackFrame::ToString() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF79053: v8::internal::ErrorUtils::FormatStackTrace(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA4D970: v8::internal::Accessors::ErrorStackGetter(v8::Local<v8::Name>, v8::PropertyCallbackInfo<v8::Value> const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==  Address 0x1ffeffe020 is on thread #1's stack
==32152==  in frame #4, created by v8::internal::Heap::Scavenge() (???:)
==32152== 
==32152== Possible data race during read of size 8 at 0x1FFEFFE048 by thread #1
==32152== Locks held: 2, at addresses 0x611FB38 0x6124E20
==32152==    at 0xEC2A30: v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE6140E: v8::internal::ScavengingTask::RunInParallel() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE8A89E: v8::internal::ItemParallelJob::Task::RunInternal() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE8B53C: v8::internal::ItemParallelJob::Run(std::shared_ptr<v8::internal::Counters>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE762DD: v8::internal::Heap::Scavenge() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7B5BF: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7BD03: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DC28: v8::internal::Handle<v8::internal::ConsString> v8::internal::Factory::New<v8::internal::ConsString>(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationSpace) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DEBD: v8::internal::Factory::NewConsString(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, int, bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2E5B5: v8::internal::Factory::NewConsString(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x118D24A: v8::internal::IncrementalStringBuilder::AppendString(v8::internal::Handle<v8::internal::String>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF756FA: v8::internal::(anonymous namespace)::AppendFileLocation(v8::internal::Isolate*, v8::internal::StackFrameBase*, v8::internal::IncrementalStringBuilder*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152== 
==32152== This conflicts with a previous write of size 8 by thread #4
==32152== Locks held: 1, at address 0x1FFEFFE020
==32152==    at 0xEC2A86: v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE6140E: v8::internal::ScavengingTask::RunInParallel() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE8A89E: v8::internal::ItemParallelJob::Task::RunInternal() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x8D9A34: node::BackgroundRunner(void*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x4C339E1: mythread_wrapper (hg_intercepts.c:389)
==32152==    by 0x5B3A50A: start_thread (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x5E4616E: clone (in /usr/lib64/libc-2.26.so)
==32152==  Address 0x1ffeffe048 is on thread #1's stack
==32152==  in frame #4, created by v8::internal::Heap::Scavenge() (???:)
==32152== 
==32152== ----------------------------------------------------------------
==32152== 
==32152==  Lock at 0x611FB38 was first observed
==32152==    at 0x4C34CA3: pthread_mutex_init (hg_intercepts.c:787)
==32152==    by 0xE6263B: v8::internal::Heap::Heap() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF36C0E: v8::internal::Isolate::Isolate(bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA7B170: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Address 0x611fb38 is 6,680 bytes inside a block of size 24,328 alloc'd
==32152==    at 0x4C2E5DA: operator new(unsigned long) (vg_replace_malloc.c:334)
==32152==    by 0xA7B163: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Block was alloc'd by thread #1
==32152== 
==32152==  Lock at 0x6124E20 was first observed
==32152==    at 0x4C34CA3: pthread_mutex_init (hg_intercepts.c:787)
==32152==    by 0x119DA88: v8::internal::ThreadManager::ThreadManager() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF3744A: v8::internal::Isolate::Isolate(bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA7B170: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Address 0x6124e20 is 0 bytes inside a block of size 80 alloc'd
==32152==    at 0x4C2E5DA: operator new(unsigned long) (vg_replace_malloc.c:334)
==32152==    by 0xF3743F: v8::internal::Isolate::Isolate(bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA7B170: v8::Isolate::New(v8::Isolate::CreateParams const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x883F99: node::Start(int, char**) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x5D71F29: (below main) (in /usr/lib64/libc-2.26.so)
==32152==  Block was alloc'd by thread #1
==32152== 
==32152==  Lock at 0x1FFEFFE2E0 was first observed
==32152==    at 0x4C34CA3: pthread_mutex_init (hg_intercepts.c:787)
==32152==    by 0xE74C8A: v8::internal::Heap::Scavenge() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7B5BF: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7BD03: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DC28: v8::internal::Handle<v8::internal::ConsString> v8::internal::Factory::New<v8::internal::ConsString>(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationSpace) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DEBD: v8::internal::Factory::NewConsString(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, int, bool) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2E5B5: v8::internal::Factory::NewConsString(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x118D24A: v8::internal::IncrementalStringBuilder::AppendString(v8::internal::Handle<v8::internal::String>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF756FA: v8::internal::(anonymous namespace)::AppendFileLocation(v8::internal::Isolate*, v8::internal::StackFrameBase*, v8::internal::IncrementalStringBuilder*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF75F7D: v8::internal::JSStackFrame::ToString() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xF79053: v8::internal::ErrorUtils::FormatStackTrace(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xA4D970: v8::internal::Accessors::ErrorStackGetter(v8::Local<v8::Name>, v8::PropertyCallbackInfo<v8::Value> const&) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==  Address 0x1ffeffe2e0 is on thread #1's stack
==32152==  in frame #6, created by v8::internal::Heap::Scavenge() (???:)
==32152== 
==32152== Possible data race during read of size 8 at 0x1FFEFFE308 by thread #4
==32152== Locks held: none
==32152==    at 0xEC2B80: v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE6140E: v8::internal::ScavengingTask::RunInParallel() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE8A89E: v8::internal::ItemParallelJob::Task::RunInternal() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x8D9A34: node::BackgroundRunner(void*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0x4C339E1: mythread_wrapper (hg_intercepts.c:389)
==32152==    by 0x5B3A50A: start_thread (in /usr/lib64/libpthread-2.26.so)
==32152==    by 0x5E4616E: clone (in /usr/lib64/libc-2.26.so)
==32152== 
==32152== This conflicts with a previous write of size 8 by thread #1
==32152== Locks held: 3, at addresses 0x611FB38 0x6124E20 0x1FFEFFE2E0
==32152==    at 0xEC2C3D: v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE6140E: v8::internal::ScavengingTask::RunInParallel() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE8A89E: v8::internal::ItemParallelJob::Task::RunInternal() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE8B53C: v8::internal::ItemParallelJob::Run(std::shared_ptr<v8::internal::Counters>) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE762DD: v8::internal::Heap::Scavenge() (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7B5BF: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE7BD03: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==    by 0xE2DC28: v8::internal::Handle<v8::internal::ConsString> v8::internal::Factory::New<v8::internal::ConsString>(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationSpace) (in /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node)
==32152==  Address 0x1ffeffe308 is on thread #1's stack
==32152==  in frame #6, created by v8::internal::Heap::Scavenge() (???:)
==32152== 
==32152== 
==32152== For counts of detected and suppressed errors, rerun with: -v
==32152== Use --history-level=approx or =none to gain increased speed, at
==32152== the cost of reduced accuracy of conflicting-access information
==32152== ERROR SUMMARY: 7 errors from 4 contexts (suppressed: 977 from 111)
Command: python -u /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/tools/run-valgrind.py out/Release/node /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/test/async-hooks/test-zlib.zlib-binding.deflate.js
[00:17|% 100|+   0|-   1]: Done                                  
[iojs@test-rackspace-fedora27-x64-1 fedora-latest-x64]$

addaleax · 2018-05-25T23:10:19Z

@Trott Thank you so much!

I personally can’t tell whether that’s a real issue or not, and we probably need somebody from @nodejs/v8 (/@ulan?) who is familiar with their GC code to take a look.

@Trott We might get bonus points for helping if we can get that output for a debug build of Node – that would give us concrete line numbers as an additional niceness… :)

ulan · 2018-05-28T11:19:26Z

[+@mlippautz]
The scavenger race might be false positive. We have few places where TSAN reports false positive data races. We have annotations to suppress those in TSAN. Maybe helgrind is hitting the same places?

Unfortunately the report doesn't contain enough data to be actionable. It would help if we knew what variable is stored at Address 0x1ffeffe308.

I looked through Scavenger::Process and don't see what variable could that be.

@Trott, any chance to get a coredump of the crash in debug mode (without helgrind)?
Do all crashes have the same stack traces as the one reported originally by @apapirovski ? (corrupted linked list in Parser)

apapirovski · 2018-05-28T16:30:42Z

@ulan The stack trace is only thrown on our LinuxONE system and it's not 100% reliable. The only 100% reliable reproduction is on the Fedora 27 system and those happen consistently when the created threads for BackgroundRunner are being joined back together.

And to clarify, those just fail with a segfault and no stack trace.

Trott · 2018-05-28T23:10:01Z

Regarding getting coredumps and/or helgrind output on debug builds: pinging @nodejs/build! Anyone available to help out with this? I'm tied up with travel the next couple days and unlikely to start up on this again in that time.

Trott · 2018-05-28T23:13:24Z

@apapirovski @addaleax Feel free to open an issue in the nodejs/build repo requesting SSH access to one or more of the relevant hosts if you'd like to be able to do the debug build and valgrind running yourself!

apapirovski · 2018-05-30T06:31:20Z

I think maybe getting access to linuxone is actually better. It's the only system that has failed with a stacktrace. Maybe debugging there will yield something different / more helpful. I'll post in nodejs/build.

Previously, the typed arrays used in this test would not automatically be kept alive by the native handle when it’s using them, so the V8 garbage collector could collect them while they are still in use by the zlib module, leading to memory corruption. Fixes: nodejs#20907

Previously, the typed arrays used in this test would not automatically be kept alive by the native handle when it’s using them, so the V8 garbage collector could collect them while they are still in use by the zlib module, leading to memory corruption. Fixes: #20907 PR-URL: #21077 Reviewed-By: Anatoli Papirovski <apapirovski@mac.com> Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Yang Guo <yangguo@chromium.org> Reviewed-By: James M Snell <jasnell@gmail.com>

apapirovski added zlib Issues and PRs related to the zlib subsystem. async_hooks Issues and PRs related to the async hooks subsystem. flaky-test Issues and PRs related to the tests with unstable failures on the CI. labels May 23, 2018

Trott mentioned this issue May 24, 2018

timers: use only a single TimerWrap instance #20555

Closed

4 tasks

Trott mentioned this issue May 24, 2018

Revert https://github.com/nodejs/node/pull/20555 #20919

Closed

1 task

shisama mentioned this issue May 24, 2018

test: add tests for fsPromises.chown to increase coverage #20574

Closed

2 tasks

mmarchini mentioned this issue May 24, 2018

test: mark test-zlib.zlib-binding.deflate as flaky #20935

Closed

2 tasks

apapirovski mentioned this issue May 30, 2018

Requesting access to test-linuxonecc-rhel72-s390x-1 nodejs/build#1291

Closed

addaleax mentioned this issue Jun 1, 2018

test: fix flaky async-hooks/test-zlib.zlib-binding.deflate #21077

Closed

3 tasks

addaleax closed this as completed in 2efe4c2 Jun 2, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flaky or bugged async-hooks/test-zlib.zlib-binding.deflate #20907

Flaky or bugged async-hooks/test-zlib.zlib-binding.deflate #20907

apapirovski commented May 23, 2018

apapirovski commented May 23, 2018

addaleax commented May 23, 2018

apapirovski commented May 23, 2018 •

edited

addaleax commented May 23, 2018

maclover7 commented May 23, 2018

addaleax commented May 23, 2018

Trott commented May 23, 2018

maclover7 commented May 23, 2018 •

edited

Trott commented May 23, 2018

mcollina commented May 23, 2018 •

edited

Trott commented May 23, 2018

mcollina commented May 23, 2018

Trott commented May 23, 2018

Trott commented May 23, 2018

Trott commented May 24, 2018

Trott commented May 24, 2018

apapirovski commented May 24, 2018 •

edited

apapirovski commented May 24, 2018

apapirovski commented May 24, 2018 •

edited

addaleax commented May 24, 2018

apapirovski commented May 24, 2018

apapirovski commented May 24, 2018 •

edited

apapirovski commented May 25, 2018

Trott commented May 25, 2018

addaleax commented May 25, 2018

Trott commented May 25, 2018

Trott commented May 25, 2018

Trott commented May 25, 2018 •

edited

addaleax commented May 25, 2018

ulan commented May 28, 2018

apapirovski commented May 28, 2018 •

edited

Trott commented May 28, 2018

Trott commented May 28, 2018

apapirovski commented May 30, 2018

Flaky or bugged async-hooks/test-zlib.zlib-binding.deflate #20907

Flaky or bugged async-hooks/test-zlib.zlib-binding.deflate #20907

Comments

apapirovski commented May 23, 2018

apapirovski commented May 23, 2018

addaleax commented May 23, 2018

apapirovski commented May 23, 2018 • edited

addaleax commented May 23, 2018

maclover7 commented May 23, 2018

addaleax commented May 23, 2018

Trott commented May 23, 2018

maclover7 commented May 23, 2018 • edited

Trott commented May 23, 2018

mcollina commented May 23, 2018 • edited

Trott commented May 23, 2018

mcollina commented May 23, 2018

Trott commented May 23, 2018

Trott commented May 23, 2018

Trott commented May 24, 2018

Trott commented May 24, 2018

apapirovski commented May 24, 2018 • edited

apapirovski commented May 24, 2018

apapirovski commented May 24, 2018 • edited

addaleax commented May 24, 2018

apapirovski commented May 24, 2018

apapirovski commented May 24, 2018 • edited

apapirovski commented May 25, 2018

Trott commented May 25, 2018

addaleax commented May 25, 2018

Trott commented May 25, 2018

Trott commented May 25, 2018

Trott commented May 25, 2018 • edited

addaleax commented May 25, 2018

ulan commented May 28, 2018

apapirovski commented May 28, 2018 • edited

Trott commented May 28, 2018

Trott commented May 28, 2018

apapirovski commented May 30, 2018

apapirovski commented May 23, 2018 •

edited

maclover7 commented May 23, 2018 •

edited

mcollina commented May 23, 2018 •

edited

apapirovski commented May 24, 2018 •

edited

apapirovski commented May 24, 2018 •

edited

apapirovski commented May 24, 2018 •

edited

Trott commented May 25, 2018 •

edited

apapirovski commented May 28, 2018 •

edited