Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collect test/test check results in test invocation #2608

Merged
merged 3 commits into from
Feb 20, 2024
Merged

Collect test/test check results in test invocation #2608

merged 3 commits into from
Feb 20, 2024

Conversation

happz
Copy link
Collaborator

@happz happz commented Jan 12, 2024 •
edited
Loading

This will be necessary to support after-test checks that wish to modify their behavior based on results of the test or another test check. Like the proposed "shell" check, #2606.

Pull Request Checklist

  • implement the feature

@happz happz added the step | execute Stuff related to the execute step label Jan 12, 2024
@happz happz mentioned this pull request Jan 12, 2024
Open
@happz happz force-pushed the make-results-visible-to-checks branch from 553503c to c77cbc4 Compare January 12, 2024 11:45
@happz happz added this to the 1.32 milestone Jan 22, 2024
@happz happz force-pushed the make-results-visible-to-checks branch from c77cbc4 to 04de109 Compare February 5, 2024 18:46
@happz happz marked this pull request as ready for review February 5, 2024 18:47
@happz happz force-pushed the make-results-visible-to-checks branch from 04de109 to a0cca3d Compare February 8, 2024 11:27
@happz
Copy link
Collaborator Author

happz commented Feb 8, 2024

/packit test

@happz happz force-pushed the make-results-visible-to-checks branch from a0cca3d to 7c95d47 Compare February 12, 2024 12:08
@happz happz force-pushed the make-results-visible-to-checks branch from 7c95d47 to 52777b2 Compare February 19, 2024 14:27
@happz happz added the ci | full test Pull request is ready for the full test execution label Feb 19, 2024
psss
psss approved these changes Feb 20, 2024
View reviewed changes
Copy link
Collaborator

@psss psss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@psss psss self-assigned this Feb 20, 2024
@happz @psss
Collect test/test check results in test invocation
454c1b8 
This will be necessary to support after-test checks that wish to modify
their behavior based on results of the test or another test check. Like
the proposed "shell" check, #2606.
@psss psss force-pushed the make-results-visible-to-checks branch from 52777b2 to 454c1b8 Compare February 20, 2024 08:36
@psss
Copy link
Collaborator

psss commented Feb 20, 2024

Full test seems to be failing. @happz, could you please have a look?

@happz
Copy link
Collaborator Author

happz commented Feb 20, 2024

Full test seems to be failing. @happz, could you please have a look?

I'm already checking them, the problem so far is, I'm unable to reproduce the failure, everything just works on my laptop :/

@psss
Copy link
Collaborator

psss commented Feb 20, 2024

I'm already checking them, the problem so far is, I'm unable to reproduce the failure, everything just works on my laptop :/

Hm, that's weird. Seems I'm able to reproduce the different behavior on my laptop:

cd tests/execute/result/basic
tmt run --until execute tests -n /xfail-with-reboot provision --how container execute -v

Old output:

execute
    queued execute task #1: default-0 on default-0

    execute task #1: default-0 on default-0
    how: tmt
        00:00:00 /test/xfail-with-reboot [1/1]
    warn: Guest does not support soft reboot, trying hard reboot.
        00:00:00 pass /test/xfail-with-reboot (on default-0) (original result: fail) [1/1]


    summary: 1 test executed

total: 1 test passed

New output:

execute
    queued execute task #1: default-0 on default-0

    execute task #1: default-0 on default-0
    how: tmt
        00:00:00 /test/xfail-with-reboot [1/1]
    warn: Guest does not support soft reboot, trying hard reboot.
        00:00:00 errr /test/xfail-with-reboot (on default-0) (original result: error) [1/1]
        00:00:00 pass /test/xfail-with-reboot (on default-0) (original result: fail) [1/1]


    summary: 2 tests executed

total: 1 test passed and 1 error

@happz
Copy link
Collaborator Author

happz commented Feb 20, 2024

I'm already checking them, the problem so far is, I'm unable to reproduce the failure, everything just works on my laptop :/

Hm, that's weird. Seems I'm able to reproduce the different behavior on my laptop:

Well, that's because as the smart, talented, and generally awesome developer I am, I've been trying to reproduce this issue with, well, tmt installed from RPMs... I wasted the whole morning on this because of a dumb and stupid mistake like this, and I realized it like 5 minutes ago :((( Once I tried it with tmt from dev environment, I got the same result we see in CI. Time to do something actually useful and investigate :/

@psss
Copy link
Collaborator

psss commented Feb 20, 2024

Ah! I share your pain :(

@psss
Copy link
Collaborator

psss commented Feb 20, 2024

All tests green except for the known issue #2687.

@psss psss merged commit 1e32b26 into main Feb 20, 2024
21 of 22 checks passed
@psss psss deleted the make-results-visible-to-checks branch February 20, 2024 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@psss psss psss approved these changes

@lukaszachy lukaszachy Awaiting requested review from lukaszachy lukaszachy is a code owner

@thrix thrix Awaiting requested review from thrix thrix is a code owner

@janhavlin janhavlin Awaiting requested review from janhavlin janhavlin is a code owner

Assignees

@psss psss

Labels
ci | full test Pull request is ready for the full test execution step | execute Stuff related to the execute step
Projects
None yet
Milestone
1.32
Development

Successfully merging this pull request may close these issues.
2 participants
@happz @psss