Test failures on FreeBSD

[Kaiepi]

Test Summary Report

t/spec/MISC/bug-coverage-stress.rakudo.moar (Wstat: 512 Tests: 11 Failed: 2)
Failed tests: 2, 5
Non-zero exit status: 2
t/spec/S02-types/capture.t (Wstat: 512 Tests: 46 Failed: 2)
Failed tests: 45-46
Non-zero exit status: 2
t/spec/S17-promise/basic.t (Wstat: 512 Tests: 41 Failed: 2)
Failed tests: 34-35
Non-zero exit status: 2
t/spec/S17-supply/syntax.t (Wstat: 9 Tests: 69 Failed: 0)
Non-zero wait status: 9
Parse errors: Bad plan. You planned 83 tests but ran 69.
t/spec/S26-documentation/multiline-leading.t (Wstat: 1024 Tests: 4 Failed: 4)
Failed tests: 1-4
Non-zero exit status: 4
t/spec/S26-documentation/block-trailing.t (Wstat: 1024 Tests: 52 Failed: 4)
Failed tests: 1-2, 4, 8
Non-zero exit status: 4
t/spec/S26-documentation/block-leading.t (Wstat: 4096 Tests: 335 Failed: 16)
Failed tests: 1, 3, 5, 7-8, 10, 12, 14, 22, 24, 26, 28
43, 45, 47, 49
Non-zero exit status: 16
t/spec/S26-documentation/why-leading.t (Wstat: 1024 Tests: 355 Failed: 4)
Failed tests: 8, 10, 12, 14
Non-zero exit status: 4

Tested on:

• FreeBSD crates 11.1-RELEASE-p8 FreeBSD 11.1-RELEASE-p8 #0: Tue Mar 13 17:07:05 UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
• This is Rakudo version 2018.04-16-g29b092c94 built on MoarVM version 2018.04-19-g1b78b04c0

[Kaiepi]

These tests also fail on:

• OpenBSD bastille.kennel.qt 6.3 GENERIC.MP#1 amd64
• This is Rakudo version 2018.04.1-115-g791c3e350 built on MoarVM version 2018.04.1-121-gd634d24cf

[usev6]

That's interesting, because I got a different result on FreeBSD with my latest stresstest:

Failure summary:
note: test_summary.pl @syn does not have MIS
S16-io/words.t 11 - heuristic for testing whether handle is closed
S24-testing/11-plan-skip-all.t aborted 1 test(s)

S16-io/words.t is a flapper for me.
I always get the abortet test in S24-testing/11-plan -- it might be caused by me still using t/harness5.

Tested on master/HEAD

• FreeBSD p6smoke 11.1-RELEASE-p10 FreeBSD 11.1-RELEASE-p10 #0: Tue May 8 05:21:56 UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
• This is Rakudo version 2018.04.1-133-g89a4cf0d9 built on MoarVM version 2018.04.1-121-gd634d24cf

@Kaiepi Your rakudo version looks a bit older. Is that rakudo-star 2018.04?
(Update: I was referring to your FreeBSD build only.)

[Kaiepi]

It's an older compiled version of Rakudo. I can recompile and test again

[Kaiepi]

After updating, these are the tests that fail:

Test Summary Report
-------------------
t/spec/S07-hyperrace/stress.t                                   (Wstat: 256 Tests: 11 Failed: 1)
  Failed test:  11
  Non-zero exit status: 1
t/spec/S17-promise/basic.t                                      (Wstat: 256 Tests: 41 Failed: 1)
  Failed test:  33
  Non-zero exit status: 1

I'll test again on OpenBSD to see which tests fail

Tested with Rakudo version 2018.04-147-g847d3d60b built on MoarVM version 2018.05

[Kaiepi]

These are the tests that fail on OpenBSD, with the same version of Rakudo and MoarVM as the FreeBSD tests:

Test Summary Report
-------------------
t/spec/MISC/bug-coverage-stress.rakudo.moar                     (Wstat: 256 Tests: 11 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
t/spec/S10-packages/precompilation.rakudo.moar                  (Wstat: 256 Tests: 49 Failed: 1)
  Failed test:  48
  Non-zero exit status: 1
t/spec/S17-channel/stress.t                                     (Wstat: 256 Tests: 4 Failed: 0)
  Non-zero exit status: 1
  Parse errors: Bad plan.  You planned 5 tests but ran 4.
t/spec/S32-io/IO-Socket-Async.t                                 (Wstat: 256 Tests: 17 Failed: 0)
  Non-zero exit status: 1

The IO::Socket::Async test failures belong in a separate issue since they're caused by my system not being configured to use IPv6

[usev6]

Commenting on FreeBSD again. I haven't seen failures for those particular tests (S07-hyperrace/stress.t test 11, S17-promise/basic.t test 33) in the past.

But we have some flapping tests (that fail once in a while). Usually I try to run a failing test file manually, with one of the following commands

$ perl t/harness5 --fudge --moar t/spec/S17-promise/basic.t
$ ./perl6-m t/harness6 --fudge t/spec/S17-promise/basic.t   ## I don't use this, probably I should

Also, @zoffixznet (always?) leaves a note in #perl6-dev if he encounters a flapping test. It looks like there where problems with S07-hyperrace/stress.t from time to time (though it might have been different tests): https://irclog.perlgeek.de/perl6-dev/search/?nick=&q=zofflop&q=S07-hyperrace%2Fstress.t

[AlexDaniel]

Also, @zoffixznet (always?) leaves a note in #perl6-dev if he encounters a flapping test

Here's the list with some: https://gist.github.com/Whateverable/3e2c50853cf29f9321d5347a8265b6b5

These files were mentioned in this ticket:

• t/spec/S17-channel/stress.t – known flapper
• t/spec/S17-promise/basic.t – known flapper
• t/spec/S32-io/IO-Socket-Async.t – IPv6 issue. Maybe something we should fix on our side, but it's likely not specific to FreeBSD, or so it seems to me.
• t/spec/MISC/bug-coverage-stress.rakudo.moar – ?
• t/spec/S10-packages/precompilation.rakudo.moar – ?

@Kaiepi, can you confirm that bug-coverage-stress and precompilation are consistently failing? I'm inclined to remove blocker label for that release, but I really want to make sure that such basic things like a spectest are working properly on a variety of systems (which is why that label was there in the first place). Also, I think things will go faster if we had separate tickets for golfed issues.

BTW @usev6 thank you for the info.

[AlexDaniel]

So, things look OK on FreeBSD (although with known flappers). At first it seemed to me that for some weird reason it's worse on OpenBSD, but now that I look at the list of known flappers, bug-coverage-stress.t and precompilation.t are also there. I'm convinced this is not a blocker, but flapping tests are indeed annoying.

[Kaiepi]

t/spec/S32-io/IO-Socket-Async.t – IPv6 issue. Maybe something we should fix on our side, but it's likely not specific to FreeBSD, or so it seems to me.

On OpenBSD, this is a libuv bug, but this is fixed by building libuv from ports and installing MoarVM with --has-libuv. See #1879

[Kaiepi]

Maybe a fork of libuv with the OpenBSD patches would help

[pprocacci]

#2162

I'm having similar problems on FreeBSD. Not flapping per se, but I have some useful notes at the above link that may prove useful.