Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

0.55 considers SKIP in tap output as failed #7515

Closed
matthiasclasen opened this issue Jul 29, 2020 · 4 comments · Fixed by #7525
Closed

0.55 considers SKIP in tap output as failed #7515

matthiasclasen opened this issue Jul 29, 2020 · 4 comments · Fixed by #7525
Labels
bug meson tests regression
Milestone
0.55.1

Comments

@matthiasclasen
Copy link

matthiasclasen commented Jul 29, 2020 •
edited
Loading

GTK has a some tests that produce output like

1..227
...
ok 57 /FinalizeObject/GtkConstraint
ok 58 /FinalizeObject/GtkConstraintGuide
ok 59 /FinalizeObject/GtkConstraintLayoutChild # SKIP Skipping GtkConstraintLayoutChild
ok 60 /FinalizeObject/GtkConstraintLayout
ok 61 /FinalizeObject/GtkCssProvider
...
ok 227 /FinalizeObject/GtkWindowHandle

that is, 227 ok's and an exit code of 0.

And meson 0.55.0 on my local system says:

3/3 gtk:gtk+object / objects-finalize FAIL           11.85s (exit status 0)

If I shorten the test to leave out all the cases that emit a # SKIP comment, meson 0.55 is happy again.

0.54 in our ci thinks everything is just fine.

I also think that everything with our testsuite is just fine, and meson regressed.
gnomesysadmins pushed a commit to GNOME/gtk that referenced this issue Jul 29, 2020
testsuite: Skip the skipping
2e80658 
meson 0.55 considers a test failed if we skip any cases.
Until that is fixed, just skip them quietly.

See mesonbuild/meson#7515
@jpakkane jpakkane added this to the 0.55.1 milestone Jul 30, 2020
gnomesysadmins pushed a commit to GNOME/gjs that referenced this issue Aug 1, 2020
@ptomato
CI: Downgrade meson
9ad6a9b 
Meson 0.55.0's TAP support is broken. This is a workaround for
mesonbuild/meson#7515

Unreviewed, pushing to fix build.
@smcv
Copy link
Contributor

smcv commented Aug 3, 2020

Bisection says this regressed in 0c3bb15.

@smcv smcv mentioned this issue Aug 3, 2020
Merged
smcv added a commit to smcv/meson that referenced this issue Aug 3, 2020
@smcv
Add test-cases for partially skipped TAP tests
da7f29c 
issue7515.txt is the output of one of the real TAP tests in gjs, which
failed as a result of mesonbuild#7515. The version inline in meson.build is
a minimal reproducer.

Signed-off-by: Simon McVittie <smcv@debian.org>
smcv added a commit to smcv/meson that referenced this issue Aug 4, 2020
@smcv
mtest: TestResult.SKIP is not a failure
998ee63 
If some but not all tests in a run were skipped, then the overall result
is given by whether there were any failures among the non-skipped tests.

Resolves: mesonbuild#7515
Signed-off-by: Simon McVittie <smcv@debian.org>
smcv added a commit to smcv/meson that referenced this issue Aug 4, 2020
@smcv
Add test-cases for partially skipped TAP tests
c6c5e8d 
issue7515.txt is the output of one of the real TAP tests in gjs, which
failed as a result of mesonbuild#7515. The version inline in meson.build is
a minimal reproducer.

Signed-off-by: Simon McVittie <smcv@debian.org>
jpakkane pushed a commit that referenced this issue Aug 5, 2020
@smcv
mtest: TestResult.SKIP is not a failure (#7525)
7db49db 
* mtest: TestResult.SKIP is not a failure

If some but not all tests in a run were skipped, then the overall result
is given by whether there were any failures among the non-skipped tests.

Resolves: #7515
Signed-off-by: Simon McVittie <smcv@debian.org>

* Add test-cases for partially skipped TAP tests

issue7515.txt is the output of one of the real TAP tests in gjs, which
failed as a result of #7515. The version inline in meson.build is
a minimal reproducer.

Signed-off-by: Simon McVittie <smcv@debian.org>
@ptomato
Copy link
Contributor

ptomato commented Aug 31, 2020

Despite the fix being included in 0.55.1, this still doesn't seem to work. Any skipped tests still cause the test suite to fail. I believe this should be reopened.

algitbot pushed a commit to alpinelinux/aports that referenced this issue Sep 3, 2020
@Cogitri
main/meson: upgrade to 0.54.3
2ef83d7 
0.55.1 still causes build failures in libpeas and gjs, see
mesonbuild/meson#7515
nirbheek pushed a commit that referenced this issue Sep 9, 2020
@smcv @nirbheek
mtest: TestResult.SKIP is not a failure
efaf89e 
If some but not all tests in a run were skipped, then the overall result
is given by whether there were any failures among the non-skipped tests.

Resolves: #7515
Signed-off-by: Simon McVittie <smcv@debian.org>
nirbheek pushed a commit that referenced this issue Sep 9, 2020
@smcv @nirbheek
Add test-cases for partially skipped TAP tests
0c3fc73 
issue7515.txt is the output of one of the real TAP tests in gjs, which
failed as a result of #7515. The version inline in meson.build is
a minimal reproducer.

Signed-off-by: Simon McVittie <smcv@debian.org>
@andyholmes andyholmes mentioned this issue Sep 20, 2020
Closed
@ptomato
Copy link
Contributor

ptomato commented Sep 20, 2020

Despite the above, it seems to have been fixed eventually in 0.55.3, so 🤷

@nirbheek
Copy link
Member

Despite the above, it seems to have been fixed eventually in 0.55.3, so 🤷

Yes, the discussion happened on #7618.

gnomesysadmins pushed a commit to GNOME/pango that referenced this issue Sep 25, 2020
@smcv
Revert "testsuite: Don't use g_test_skip"
e84fee6 
When versions of Meson with bug
mesonbuild/meson#7515 are no longer widespread,
the Pango test suite should diagnose which tests are skipped again.
Versions 0.55.0 to 0.55.2 inclusive have the bug; version 0.55.3 is OK.

This reverts commit 440f9c2.

Signed-off-by: Simon McVittie <smcv@debian.org>
gnomesysadmins pushed a commit to GNOME/pango that referenced this issue Jan 4, 2021
@smcv
Revert "testsuite: Don't use g_test_skip"
905db39 
When versions of Meson with bug
mesonbuild/meson#7515 are no longer widespread,
the Pango test suite should diagnose which tests are skipped again.
Versions 0.55.0 to 0.55.2 inclusive have the bug; version 0.55.3 is OK.

This reverts commit 440f9c2.

Signed-off-by: Simon McVittie <smcv@debian.org>
gnomesysadmins pushed a commit to GNOME/pango that referenced this issue Jan 4, 2021
@smcv
Revert "testsuite: Don't use g_test_skip"
221a35e 
When versions of Meson with bug
mesonbuild/meson#7515 are no longer widespread,
the Pango test suite should diagnose which tests are skipped again.
Versions 0.55.0 to 0.55.2 inclusive have the bug; version 0.55.3 is OK.

This reverts commit 440f9c2.

Signed-off-by: Simon McVittie <smcv@debian.org>
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Oct 16, 2022
f! convert to meson
0272257 
get a newer version of meson than what's in chocolatey??
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

https://cirrus-ci.com/task/4818847206211584

windows-cross
ci-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Oct 16, 2022
f! convert to meson
000e7b8 
get a newer version of meson than what's in chocolatey??
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

https://cirrus-ci.com/task/4818847206211584

windows-cross
ci-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Oct 16, 2022
f! convert to meson
82c2e4c 
get a newer version of meson than what's in chocolatey??
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

https://cirrus-ci.com/task/4818847206211584

windows-cross
ci-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Oct 16, 2022
f! convert to meson
10b581c 
get a newer version of meson than what's in chocolatey??
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

https://cirrus-ci.com/task/4818847206211584

ci-os-only: windows-run-cross windows-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Oct 16, 2022
f! convert to meson
7d5d50e 
get a newer version of meson than what's in chocolatey??
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

https://cirrus-ci.com/task/4818847206211584

ci-os-only: windows-run-cross windows-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Oct 16, 2022
f! convert to meson
38be73c 
get a newer version of meson than what's in chocolatey??
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

https://cirrus-ci.com/task/4818847206211584

ci-os-only: windows-run-cross windows-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Oct 16, 2022
f! convert to meson
9f56a94 
get a newer version of meson than what's in chocolatey??
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

https://cirrus-ci.com/task/4818847206211584

ci-os-only: windows-run-cross windows-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Jun 24, 2023
TMP: cirrus/windows/cross: install python,meson,ninja
74660f8 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

ci-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Jul 5, 2023
TMP: cirrus/windows/cross: install python,meson,ninja
04051cb 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

ci-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Jul 6, 2023
TMP: cirrus/windows/cross: install python,meson,ninja
8032c0f 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

ci-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Jul 13, 2023
TMP: cirrus/windows/cross: install python,meson,ninja
c315d0b 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

ci-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Jul 19, 2023
TMP: cirrus/windows/cross: install python,meson,ninja
064b454 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

ci-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Aug 1, 2023
TMP: cirrus/windows/cross: install python,meson,ninja
a8d2b3f 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

ci-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Nov 15, 2023
TMP: cirrus/windows/cross: install python,meson,ninja
a09d52b 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

ci-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Nov 15, 2023
TMP: cirrus/windows/cross: install python,meson,ninja
6268a2c 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Nov 15, 2023
TMP: cirrus/windows/cross: install python,meson,ninja
36a293a 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Nov 15, 2023
TMP: cirrus/windows/cross: install python,meson,ninja
f88e760 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Mar 2, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
05ee8ca 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Mar 8, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
57ace7f 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Mar 8, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
71ccf3d 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Mar 8, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
044f45d 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Mar 12, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
5d87b0d 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Mar 13, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
a7bca9b 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Mar 22, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
fb64933 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Mar 26, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
b146852 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Mar 31, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
c4b2205 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Apr 3, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
05283e8 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Apr 8, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
8264ae3 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Apr 8, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
cf41f94 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Apr 11, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
320628d 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Apr 12, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
25f1225 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Apr 22, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
ff4e374 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Apr 28, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
977fe63 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue May 8, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
34c9342 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue May 12, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
74d54e2 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue May 29, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
87d568c 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
justinpryzby pushed a commit to justinpryzby/postgres that referenced this issue Nov 11, 2024
TMP: cirrus/windows/cross: install python,meson,ninja
d312637 
get a newer version of meson than what's in chocolatey
https://community.chocolatey.org/packages/meson
mesonbuild/meson#7515

//-os-only: windows-run-cross
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug meson tests regression
Projects
None yet
Milestone
0.55.1
Development

Successfully merging a pull request may close this issue.

mtest: TestResult.SKIP is not a failure smcv/meson
6 participants
@nirbheek @ptomato @smcv @dcbaker @jpakkane @matthiasclasen