Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need kyua report --diff A B #220

Open
bsdimp opened this issue Sep 1, 2023 · 3 comments
Open

Need kyua report --diff A B #220

bsdimp opened this issue Sep 1, 2023 · 3 comments

Comments

@bsdimp
Copy link
Member

bsdimp commented Sep 1, 2023

FreeBSD has 8k tests. 1500 of them are failing in qemu. Fine, those are my problem.
However, if I run weekly tests, I just get a number (1500, 1499, 1501, etc). I have no clue what's newly failed, and what's different than the last time, or what's different from a native run (where only 650 tests fail because we haven't filtered based on jail).

It sure would be nice if there was a kyua command to do this rather than the hacked up kua report on both followed by a complex shell pipeline to get the diffs.

@lwhsu
Copy link
Member

lwhsu commented Sep 2, 2023

Before kyua supports this feature, I think a possible way is using report-junit and the tools to compare two junit reports.

@bsdimp
Copy link
Member Author

bsdimp commented Sep 2, 2023

I noticed the report-junit stuff in the ci repo, but didn't see where it was used.... Do you have a pointer to how to use those tools? I'm a complete novice.

@ngie-eign
Copy link
Contributor

ngie-eign commented Sep 8, 2023

I noticed the report-junit stuff in the ci repo, but didn't see where it was used.... Do you have a pointer to how to use those tools? I'm a complete novice.

The JUnit plugin in Jenkins CI handles results history/rendering: https://plugins.jenkins.io/junit/ . I wouldn't go down the rabbit hole of inventing a JUnit parser which duplicates what Jenkins CI does just for the purposes of tracking test case history.

If the feature was added, it should poke directly at the SQLite3 database... but some work would need to be done to make a kyua report adjacent tool for this purpose (the whole last run, etc, thing encoded in the results file isn't incredibly intuitive).

Also, if you use kyua report, you can filter by results type (--results-filter), to diff results for a much smaller failing set. It doesn't seem universally supported by all report* subcommands though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants