rocketchat crashes with segmentation fault since nodejs 8.10 #19274
Comments
I'd check whether this is fixed with newer V8 and bisect for the fix. |
I could reproduce. Then I wanted to run it again with node master but there is a dependency that uses a V8 API that doesn't exist anymore |
I commented on the Rocket Chat issue. From the debug build output it looks like API abuse to me. |
I was wrong. This rather looks like the parser is running on a thread it is not supposed to be on. |
If you have a patch at some point, i'm happy to to test. |
I've seen the thread ID error when running a debug build of Node and using https://github.com/laverdet/node-fibers (which I believe RocketChat uses, since it's a Meteor app?):
Specifically, the V8 itself is designed to tolerate multiple threads as long as you're careful to lock and unlock the While the |
node-fibers mucks around with thread-local storage to trick V8 into believing it's running on multiple threads. Some recent change probably broke that hack but that's ultimately a node-fibers issue. |
umm... I'm shocked. |
Just to be clear: we have not pinned this down to fibers yet, though that seems possible, and of course those of us who rely on fibers are willing to put in work to keep them compatible with Node and V8. Let’s wait and see what the real problem is before assuming the solution will be unwelcome to anyone? |
As soon as we know this is a fibers issue for sure, I definitely support moving the discussion over to that repo! |
Definitely support finding out the root cause. I'm just expressing my surprise that node-fibers would manipulate TLS. |
In principle, fibers could be implemented by actually spawning new threads, and then there would be nothing weird about how it works. I agree there’s some trickery involved in getting V8 to think there are multiple threads involved, without incurring the full overhead of spawning threads, and if those tricks are the source of this problem, then of course that’s a fibers problem. Those ticks have worked fairly well for several years, so I’m confident we can find a solution that involves fewer tricks (including just spawning real threads, if necessary). |
Maybe someone can bisect the changes between 8.9 and 8.10 to see what introduced the issue to surface? |
@benjamn has anyone reported similar problems on 9.x? It seems odd that this is just showing up now |
@MylesBorins If this is a meteor-specific problem (fibers-related or otherwise), then it’s relevant to note that each version of meteor ships with a specific (stable) node version, and 8.9.4 is the most recent version we’ve shipped (in meteor 1.6.1). You could run an unreleased build of meteor with Node 9, but that’s uncommon. |
@benjamn with that in mind we should try and figure out how to get some of the meteor stuff into CITGM so we can find these problems early |
@hashseed It took a little bit of dancing around and shuffling commits to avoid some of the surrounding un-buildable commits, but this bisects down to bede7a3 ( Is there a documented process somewhere as to how this larger V8 update commit is formed? Is it essentially a |
@abernix thanks for bisecting! It seems that the commit updated V8 from In a V8 check out I get: git log 6.1.534..6.2.414.46 --oneline | wc -l
1043 So bisecting about 10 steps in V8 should give you the culprit. To do so, check out V8 and update Node.js like this on every bisect step v8/tools/release/update_node.py $V8_DIR $NODE_DIR Then build and try to repro. |
|
I'm not familiar with the code, but the suggestion sounds reasonable. I have a theory.
Prior to v8/v8@ea0e1e2, we would only invalidate code on the current stack. After it, we would invalidate all code from the same context. In cases where V8 is embedded with So to reproduce this bug, what needs to happen is:
So we should definitely revert that change. Thanks for the bisect! @bmeurer does that sound correct to you? |
Did not remove ActivationsFinder from `src/runtime/runtime-compiler.cc` as in the original commit as the Class is still being used prior to f0acede landing Original Commit Message: Deoptimization and multithreading. When using Lockers and Unlockers it is possible to create a scenario where multiple threads point to the same optimized code object. When that happens, if one of the threads triggers deoptimization, then the stack replacement needs to happen in the stacks of all threads. With this CL, the deoptimizer visits all threads to do so. The CL also adds three tests where V8 used to crash due to this issue. Bug: v8:6563 Change-Id: I74e9af472d4833aa8d13e579df45133791f6a503 Reviewed-on: https://chromium-review.googlesource.com/670783 Reviewed-by: Jaroslav Sevcik <jarin@chromium.org> Commit-Queue: Juliana Patricia Vicente Franco <jupvfranco@google.com> Cr-Commit-Position: refs/heads/master@{#48060} PR-URL: #19477 Fixes: #19274 Refs: v8/v8@596d55a Refs: v8/v8@f0acede Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Yang Guo <yangguo@chromium.org> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com> Reviewed-By: Benedikt Meurer <benedikt.meurer@gmail.com>
Did not remove ActivationsFinder from `src/runtime/runtime-compiler.cc` as in the original commit as the Class is still being used prior to f0acede landing Original Commit Message: Deoptimization and multithreading. When using Lockers and Unlockers it is possible to create a scenario where multiple threads point to the same optimized code object. When that happens, if one of the threads triggers deoptimization, then the stack replacement needs to happen in the stacks of all threads. With this CL, the deoptimizer visits all threads to do so. The CL also adds three tests where V8 used to crash due to this issue. Bug: v8:6563 Change-Id: I74e9af472d4833aa8d13e579df45133791f6a503 Reviewed-on: https://chromium-review.googlesource.com/670783 Reviewed-by: Jaroslav Sevcik <jarin@chromium.org> Commit-Queue: Juliana Patricia Vicente Franco <jupvfranco@google.com> Cr-Commit-Position: refs/heads/master@{#48060} PR-URL: nodejs#19477 Fixes: nodejs#19274 Refs: v8/v8@596d55a Refs: v8/v8@f0acede Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Yang Guo <yangguo@chromium.org> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com> Reviewed-By: Benedikt Meurer <benedikt.meurer@gmail.com>
I just tried the d46fafc with 8.11.1. It works so far. One question though. There is no file called test-locker.cc. Is that an additional file somewhere? thanks and cheers |
@himBeere Are you referring to the (plural) |
The patch is there, yes. But which file should I patch? It's note included in the source as far as I can see.
thanks and cheers |
In the node repo run the following command
|
Also worth noting that this patch has not yet gonna wait in an 8.x release 8.11.1 was a patch release due to infra related issues. Another version of 8.x will be coming in just over 2 weeks |
@MylesBorins good to know. I understand the meteor guys are trying to fix the issue themself for now. |
@himBeere Correct, we (as in, Meteor, myself and @benjamn included) are trying to mitigate the situation which arose after 8.11.1 was announced. Of course, since it's a security update, many of our users are updating to it in the name of security (makes sense!). Unfortunately, it includes the breaking change outlined in this issue which causes segmentation faults for many of our users. We're currently deploying a fix to our own hosting platform, Galaxy, by using our own custom build of Node 8.11.1 with the d46fafc commit applied, but that leaves non-Galaxy users needing to make the decision (and often unbeknownst to them, until they run into either problem) of whether to stay on 8.11.0 or tolerate the segmentation faults. We're anxiously awaiting the 8.11.2(?) which should solve this! |
News about the 8.11.2 release date? |
At least anecdotally, the meteor build seems to have resolved ongoing segmentation faults in our pm2 God processes; 'anecdotally' and 'seems' because proving a negative is difficult, but we've been solid since applying the patch, and we've got quite the collection of core dumps from Mars prior to it. |
Did not remove ActivationsFinder from `src/runtime/runtime-compiler.cc` as in the original commit as the Class is still being used prior to f0acede landing Original Commit Message: Deoptimization and multithreading. When using Lockers and Unlockers it is possible to create a scenario where multiple threads point to the same optimized code object. When that happens, if one of the threads triggers deoptimization, then the stack replacement needs to happen in the stacks of all threads. With this CL, the deoptimizer visits all threads to do so. The CL also adds three tests where V8 used to crash due to this issue. Bug: v8:6563 Change-Id: I74e9af472d4833aa8d13e579df45133791f6a503 Reviewed-on: https://chromium-review.googlesource.com/670783 Reviewed-by: Jaroslav Sevcik <jarin@chromium.org> Commit-Queue: Juliana Patricia Vicente Franco <jupvfranco@google.com> Cr-Commit-Position: refs/heads/master@{#48060} PR-URL: nodejs#19477 Fixes: nodejs#19274 Refs: v8/v8@596d55a Refs: v8/v8@f0acede Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Yang Guo <yangguo@chromium.org> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com> Reviewed-By: Benedikt Meurer <benedikt.meurer@gmail.com>
Fix seems to be in 8.11.2. Is that right? |
Yes, I think this can be closed! |
We didn't release an official version using 8.11.2 yet, we will do it soon |
Sounds like this can be closed. Feel free to reopen if I'm incorrect. |
Hello Devs.
Since nodejs 8.10 Rocketchat (https://rocket.chat/) crashes with segmentation fault. Compiled nodejs with debug and got this:
And:
Any idea how to fix this?
Here is the rocketchat issue: RocketChat/Rocket.Chat#10060
thanks and cheers
The text was updated successfully, but these errors were encountered: