Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test failures on FreeBSD #1867

Open
Kaiepi opened this issue May 25, 2018 · 11 comments

Comments

Projects
None yet
5 participants
@Kaiepi
Copy link
Contributor

commented May 25, 2018

Test Summary Report

t/spec/MISC/bug-coverage-stress.rakudo.moar (Wstat: 512 Tests: 11 Failed: 2)
Failed tests: 2, 5
Non-zero exit status: 2
t/spec/S02-types/capture.t (Wstat: 512 Tests: 46 Failed: 2)
Failed tests: 45-46
Non-zero exit status: 2
t/spec/S17-promise/basic.t (Wstat: 512 Tests: 41 Failed: 2)
Failed tests: 34-35
Non-zero exit status: 2
t/spec/S17-supply/syntax.t (Wstat: 9 Tests: 69 Failed: 0)
Non-zero wait status: 9
Parse errors: Bad plan. You planned 83 tests but ran 69.
t/spec/S26-documentation/multiline-leading.t (Wstat: 1024 Tests: 4 Failed: 4)
Failed tests: 1-4
Non-zero exit status: 4
t/spec/S26-documentation/block-trailing.t (Wstat: 1024 Tests: 52 Failed: 4)
Failed tests: 1-2, 4, 8
Non-zero exit status: 4
t/spec/S26-documentation/block-leading.t (Wstat: 4096 Tests: 335 Failed: 16)
Failed tests: 1, 3, 5, 7-8, 10, 12, 14, 22, 24, 26, 28
43, 45, 47, 49
Non-zero exit status: 16
t/spec/S26-documentation/why-leading.t (Wstat: 1024 Tests: 355 Failed: 4)
Failed tests: 8, 10, 12, 14
Non-zero exit status: 4

Tested on:

  • FreeBSD crates 11.1-RELEASE-p8 FreeBSD 11.1-RELEASE-p8 #0: Tue Mar 13 17:07:05 UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
  • This is Rakudo version 2018.04-16-g29b092c94 built on MoarVM version 2018.04-19-g1b78b04c0
@Kaiepi

This comment has been minimized.

Copy link
Contributor Author

commented May 25, 2018

These tests also fail on:

  • OpenBSD bastille.kennel.qt 6.3 GENERIC.MP#1 amd64
  • This is Rakudo version 2018.04.1-115-g791c3e350 built on MoarVM version 2018.04.1-121-gd634d24cf
@usev6

This comment has been minimized.

Copy link
Contributor

commented May 25, 2018

That's interesting, because I got a different result on FreeBSD with my latest stresstest:

Failure summary:
note: test_summary.pl @syn does not have MIS
S16-io/words.t 11 - heuristic for testing whether handle is closed
S24-testing/11-plan-skip-all.t aborted 1 test(s)

S16-io/words.t is a flapper for me.
I always get the abortet test in S24-testing/11-plan -- it might be caused by me still using t/harness5.

Tested on master/HEAD

  • FreeBSD p6smoke 11.1-RELEASE-p10 FreeBSD 11.1-RELEASE-p10 #0: Tue May 8 05:21:56 UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
  • This is Rakudo version 2018.04.1-133-g89a4cf0d9 built on MoarVM version 2018.04.1-121-gd634d24cf

@Kaiepi Your rakudo version looks a bit older. Is that rakudo-star 2018.04?
(Update: I was referring to your FreeBSD build only.)

@Kaiepi

This comment has been minimized.

Copy link
Contributor Author

commented May 25, 2018

It's an older compiled version of Rakudo. I can recompile and test again

@Kaiepi

This comment has been minimized.

Copy link
Contributor Author

commented May 25, 2018

After updating, these are the tests that fail:

Test Summary Report
-------------------
t/spec/S07-hyperrace/stress.t                                   (Wstat: 256 Tests: 11 Failed: 1)
  Failed test:  11
  Non-zero exit status: 1
t/spec/S17-promise/basic.t                                      (Wstat: 256 Tests: 41 Failed: 1)
  Failed test:  33
  Non-zero exit status: 1

I'll test again on OpenBSD to see which tests fail

Tested with Rakudo version 2018.04-147-g847d3d60b built on MoarVM version 2018.05

@Kaiepi

This comment has been minimized.

Copy link
Contributor Author

commented May 25, 2018

These are the tests that fail on OpenBSD, with the same version of Rakudo and MoarVM as the FreeBSD tests:

Test Summary Report
-------------------
t/spec/MISC/bug-coverage-stress.rakudo.moar                     (Wstat: 256 Tests: 11 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
t/spec/S10-packages/precompilation.rakudo.moar                  (Wstat: 256 Tests: 49 Failed: 1)
  Failed test:  48
  Non-zero exit status: 1
t/spec/S17-channel/stress.t                                     (Wstat: 256 Tests: 4 Failed: 0)
  Non-zero exit status: 1
  Parse errors: Bad plan.  You planned 5 tests but ran 4.
t/spec/S32-io/IO-Socket-Async.t                                 (Wstat: 256 Tests: 17 Failed: 0)
  Non-zero exit status: 1

The IO::Socket::Async test failures belong in a separate issue since they're caused by my system not being configured to use IPv6

@usev6

This comment has been minimized.

Copy link
Contributor

commented May 26, 2018

Commenting on FreeBSD again. I haven't seen failures for those particular tests (S07-hyperrace/stress.t test 11, S17-promise/basic.t test 33) in the past.

But we have some flapping tests (that fail once in a while). Usually I try to run a failing test file manually, with one of the following commands

$ perl t/harness5 --fudge --moar t/spec/S17-promise/basic.t
$ ./perl6-m t/harness6 --fudge t/spec/S17-promise/basic.t   ## I don't use this, probably I should

Also, @zoffixznet (always?) leaves a note in #perl6-dev if he encounters a flapping test. It looks like there where problems with S07-hyperrace/stress.t from time to time (though it might have been different tests): https://irclog.perlgeek.de/perl6-dev/search/?nick=&q=zofflop&q=S07-hyperrace%2Fstress.t

@AlexDaniel

This comment has been minimized.

Copy link
Member

commented Jun 15, 2018

Also, @zoffixznet (always?) leaves a note in #perl6-dev if he encounters a flapping test

Here's the list with some: https://gist.github.com/Whateverable/3e2c50853cf29f9321d5347a8265b6b5


These files were mentioned in this ticket:

  • t/spec/S17-channel/stress.t – known flapper
  • t/spec/S17-promise/basic.t – known flapper
  • t/spec/S32-io/IO-Socket-Async.t – IPv6 issue. Maybe something we should fix on our side, but it's likely not specific to FreeBSD, or so it seems to me.
  • t/spec/MISC/bug-coverage-stress.rakudo.moar – ?
  • t/spec/S10-packages/precompilation.rakudo.moar – ?

@Kaiepi, can you confirm that bug-coverage-stress and precompilation are consistently failing? I'm inclined to remove blocker label for that release, but I really want to make sure that such basic things like a spectest are working properly on a variety of systems (which is why that label was there in the first place). Also, I think things will go faster if we had separate tickets for golfed issues.

BTW @usev6 thank you for the info.

@AlexDaniel

This comment has been minimized.

Copy link
Member

commented Jun 16, 2018

So, things look OK on FreeBSD (although with known flappers). At first it seemed to me that for some weird reason it's worse on OpenBSD, but now that I look at the list of known flappers, bug-coverage-stress.t and precompilation.t are also there. I'm convinced this is not a blocker, but flapping tests are indeed annoying.

@AlexDaniel AlexDaniel removed the BLOCKER label Jun 16, 2018

@Kaiepi

This comment has been minimized.

Copy link
Contributor Author

commented Jun 16, 2018

t/spec/S32-io/IO-Socket-Async.t – IPv6 issue. Maybe something we should fix on our side, but it's likely not specific to FreeBSD, or so it seems to me.

On OpenBSD, this is a libuv bug, but this is fixed by building libuv from ports and installing MoarVM with --has-libuv. See #1879

@Kaiepi

This comment has been minimized.

Copy link
Contributor Author

commented Jun 16, 2018

Maybe a fork of libuv with the OpenBSD patches would help

@pprocacci

This comment has been minimized.

Copy link

commented Aug 1, 2018

#2162

I'm having similar problems on FreeBSD. Not flapping per se, but I have some useful notes at the above link that may prove useful.

@lucasbuchala lucasbuchala added the BSD label Mar 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.