segfaults and other stability issues

[AlexDaniel]

See this: http://irclog.perlgeek.de/perl6/2016-08-19#i_13055233

Pretty sure that it is not our fault, but we have to rakudobug it.

[AlexDaniel]

• Seems like RT #129291 is the most common problem at this moment. Once that is fixed, we will probably see other issues.

[AlexDaniel]

• RT #129291 was fixed, next problem is RT #129781.

[AlexDaniel]

• RT #129781 was fixed, next problem is that the process is not killed if there's a lot of stuff on stdout of Proc::Async. See RT #130370, but it's not a problem because a workaround has been added in commit c564d8d.

[AlexDaniel]

As of today, there are no segfaults. I'd still have to write tests for some cases mentioned here, but generally it is not an issue anymore.

[AlexDaniel]

• OK, bots are not stable anymore. I think it's due to rakudo/rakudo@9658dd98c9.

Getting stuff like this:

MoarVM panic: Internal error: invalid thread ID 284 in GC work pass

Didn't look into it deeply at all, but leaving a note here anyway.

Can be reproduced by running t/bisectable.p6 on the server (sometimes you may get lucky and the whole file will pass, but usually it crashes half way through).

[tony-o]

I’m getting those a lot too (happening debugging HTTP::Server::Async issues) On July 27, 2017 at 11:18:45 AM, Aleks-Daniel Jakimenko-Aleksejev (notifications@github.com<mailto:notifications@github.com>) wrote: OK, bots are not stable anymore. I think it's due to rakudo/rakudo@9658dd9<rakudo/rakudo@9658dd98c9>. Getting stuff like this: MoarVM panic: Internal error: invalid thread ID 284 in GC work pass Didn't look into it deeply at all, but leaving a note here anyway. Can be reproduced by running t/bisectable.p6 on the server (sometimes you may get lucky and the whole file will pass, but usually it crashes half way through). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#24 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AB75kKe5t-nLzDuQEw_fqm66YFsjJYXQks5sSNSBgaJpZM4JovXH>.

[AlexDaniel]

Seems to be alright now after fixes by @jnthn++.

[AlexDaniel]

• Currently most bots are leaking memory (which is why some things are slower than they were before).

[AlexDaniel]

The leakage was reported in RT #131879, and right now it is fixed in a way that it does not leak as much anymore. The memory usage increases if you keep throwing non-existent commits into the bots, but given 16GB of RAM on the server this is hardly a problem.

Right now, the bots are stable.

[AlexDaniel]

• Quotable does not work (and was not working for a while): RT #131961
  Greppable has a problem with it also, but it is more or less usable.

[AlexDaniel]

RT #131961 is resolved, waiting for the next bug to appear now.

[AlexDaniel]

• Well, didn't have to wait for too long. Most bots can't pass their tests, I don't know why yet. Things seem to hang.

[AlexDaniel]

• Possibly it's RT #132030.

[AlexDaniel]

Well, I guess it's not. Things are still broken though.

[AlexDaniel]

• Also, we're stuck with non-HEAD version of rakudo because of RT #132191. See also Some issue on latest rakudo lizmat/IRC-Client#51.

[AlexDaniel]

OK, RT #132191 turned out to be an issue in IRC-Client (it was relying on a rakudo bug).

Now there are at least two other problems. Bisectable fails with this output:

ok 60 - Did you mean “HEAD” (new)?
# Failed to get expected result in 11.04535627 seconds (11 nominal)
not ok 61 - Did you mean “HEAD” (old)?
# Failed test 'Did you mean “HEAD” (old)?'
# at /home/bisectable/git/whateverable/t/lib/Testable.pm6 (Testable) line 81
# expected: ["testable742093, Cannot find revision “DEAD” (did you mean “HEAD”?)"]
#  matcher: 'infix:<~~>'
#      got: []
# Test failed. Stopping test suite, because PERL6_TEST_DIE_ON_FAIL environmental variable is set to a true value.
# Failed to get expected result in 11.04317088 seconds (11 nominal)
not ok 62 - _
# Failed test '_'
# at /home/bisectable/git/whateverable/t/lib/Testable.pm6 (Testable) line 81
# expected: [-> ;; $_? is raw { #`(Block|84942264) ... }]
#  matcher: 'infix:<~~>'
#      got: []
# Test failed. Stopping test suite, because PERL6_TEST_DIE_ON_FAIL environmental variable is set to a true value.

There is no reason why test 61 would fail. Actually, it passes if you put it higher in that file. I don't know what's going on there, but most likely it's an issue in rakudo.

The second problem is that it runs some other test after the first test failed. Why? It should not be like that.

[AlexDaniel]

Ah OK, the ‘_’ test is an issue in whateverable. Nevermind that. Why does it fail in the first place is beyond me however.

[AlexDaniel]

• So, to make this clear, the tests are still failing. It simply stops dead when performing these tests:
  https://github.com/perl6/whateverable/blob/e9ccebadca9a44e4a27a2325737308828568786b/t/bisectable.t#L165-L170

The code involves a lot of calls to Text::Diff::Sift4 module, but nothing special really. This issue didn't exist a few releases ago, and I really am not sure when this happened exactly.

The same test works fine in committable.t, and actually if you move these tests higher in the bisectable.t file, they will pass. Really weird stuff going on.

[AlexDaniel]

Alright, some progress on that! First of all, it doesn't hang, it segfaults. The reason I was thinking that it hangs is because the test suite does not really detect if the bot process dies unexpectedly, so there was no easy way to notice. Now I have some code that will help notice the issue in the future, will commit that soon.

Now, the segfault happens in the react block here:
https://github.com/perl6/whateverable/blob/e9ccebadca9a44e4a27a2325737308828568786b/lib/Whateverable.pm6#L220-L232

So, that's easy now, right? Just run it under valgrind and you'll immediately see the issue…

Ha-ha.

Nope. You run it under valgrind, and the issue goes away. 💩

I'm suspecting that we may be seeing something like rakudo/rakudo#1202 here, but it's hard to tell.

[AlexDaniel]

Same issue under gdb: https://gist.github.com/MasterDuke17/0312dd2af1e3b2b498d91cfacc45343c

[AlexDaniel]

• Reportable is currently suffering from this issue (SEGV): Another SEGV in whateverable rakudo/rakudo#1278

[AlexDaniel]

• Bots are currently leaking memory like crazy. I will probably turn off some of them so that they don't max out the memory usage on the server.

[AlexDaniel]

• Just had this intermittent fail:

Cannot find method 'specialize' on object of type NQPClassHOW

On this line:
https://github.com/perl6/whateverable/blob/46337991a954885fe4c535319275bbb6f797b391/lib/Whateverable.pm6#L326

I cannot reproduce so we will just let it be…

[AlexDaniel]

• MoarVM ticket that is probably related to our current memory leaks: Large memory leak with Proc::Async MoarVM/MoarVM#680

[AlexDaniel]

• More or less isolated memory leak: Memory leaks (IO::Socket::Async?) – HTTP::UserAgent, Cro::HTTP::Client, … rakudo/rakudo#1501

[AlexDaniel]

• I think it no longer leaks as much, but now bisectable segfaults here:
  https://github.com/perl6/whateverable/blob/177b77cb2ebc045736b8e7a1cf6eb8e25fdce7b6/t/bisectable.t#L186-L191

Again, there's nothing special with this test. And if you look closely, previous tests have been commented out because they were causing another segv previously. Here's the ticket: rakudo/rakudo#1259

[AlexDaniel]

Bots no longer leak memory like crazy, so that issue is resolved. Bisectable still can't get through its tests though.

[AlexDaniel]

• Issue Shareable goes down if you just cancel the download #296 / R#1595

[AlexDaniel]

OK issue #296 can be workarounded like this:

-my $host-arch = $*KERNEL.hardware;
+my $host-arch = ‘x86_64’;
$host-arch = ‘amd64’|‘x86_64’ if $host-arch eq ‘amd64’|‘x86_64’;
-$host-arch = $*KERNEL.name ~ ‘-’ ~ $host-arch;
+$host-arch = ‘linux’ ~ ‘-’ ~ $host-arch;

Heh. Not committing this to the repo because I'm hoping it'll get resolved relatively quickly.

[lizmat]

Could you try this diff and see if that makes a difference? $ git diff diff --git a/src/core/Kernel.pm6 b/src/core/Kernel.pm6 index 1cde4c4..7ce4cf8 100644 --- a/src/core/Kernel.pm6 +++ b/src/core/Kernel.pm6 @@ -180,8 +180,8 @@ class Kernel does Systemic { } }

…

-Rakudo::Internals.REGISTER-DYNAMIC: '$*KERNEL', { +#Rakudo::Internals.REGISTER-DYNAMIC: '$*KERNEL', { PROCESS::<$KERNEL> := Kernel.new; -} +#} If it does, then it’s something in the auto-vivification of dynamic variables that’s to blame, and not something specific to $*KERNEL.

On 7 Mar 2018, at 22:34, Aleks-Daniel Jakimenko-Aleksejev ***@***.***> wrote: OK issue #296 can be workarounded like this: -my $host-arch = $*KERNEL.hardware; +my $host-arch = ‘x86_64’; $host-arch = ‘amd64’|‘x86_64’ if $host-arch eq ‘amd64’|‘x86_64’; -$host-arch = $*KERNEL.name ~ ‘-’ ~ $host-arch; +$host-arch = ‘linux’ ~ ‘-’ ~ $host-arch; Heh. Not committing this to the repo because I'm hoping it'll get resolved relatively quickly. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[AlexDaniel]

@lizmat tried that, no difference. Further discussion here: rakudo/rakudo#1595

[AlexDaniel]

• Undersightable segfaults all the time. Cro + Digest::SHA256::Native = segfault rakudo/rakudo#1660

[AlexDaniel]

• Some sort of rakudo bug I guess? 8e206d5

[AlexDaniel]

In 2023, I hit a few issues that were part of #388:

• 100% CPU usage when using Pastebin::Gist rakudo/rakudo#5463
• Invocant of method 'match' must be a type object of type 'Any' rakudo/rakudo#5464
• Plus LOTS of problems when trying to install dependencies of Whateverable. This seems like an impossible task today. The dependency tree just doesn't get resolved, and the only alternative is to install modules one by one. For dockerized bots, it was a lot easier to base everything on Rakudo Star, because that has most of the packages preinstalled, and then the job is easier.