Collecting user stories for new UX design

[gctucker]

@khilman started a thread on the mailing list a while ago: RFC: dashboards, visualization and analytics for test results. Having a GitHub discussion can also help gather such user stories and make it easier to engage with people not on the mailing list.

The aim is to have as many user-stories as possible to understand to which extent the KernelCI data may be used. Based on this initial research, we can then design a new dashboard and a set of tools to provide a better user experience.

[gctucker]

Some initial user stories with suggestions on how to address them:

A kernel developer has sent a patch which caused a regression
-> Find the details, how to reproduce, check when it’s fixed

A maintainer is getting a branch ready for the next merge window
-> Compare results against mainline, ensure all tests were run correctly

An OEM or SoC vendor needs to upgrade their kernel or move to upstream
-> Show all results for a particular platform on various stable releases

Regular visitors who want to know how the kernel is doing
-> Highlight new regressions, show trends for pass/fail and performance
-> Does kernel X work on my hardware?

[gctucker]

A maintainer is getting a branch ready for the next merge window

• Compare results against mainline, ensure all tests were run correctly

On 22/12/2020 15:37, Chris Paterson wrote:

As part of this, it may be good to hide the failures reported that are also reported on mainline, so the maintainer knows the regressions directly caused by their tree.
Perhaps this change could also be made for the email reports.

Suggested-by: Geert Uytterhoeven geert+renesas@glider.be

[gctucker]

Stable kernel maintainer wants to know about a series of patches for a new release candidate:

• Are there any regressions compared to the previous stable release?
• Which are the patches causing regressions?

Ideally, an email would be sent automatically replying to the stable review thread with a Tested-by trailer if things worked or details about regressions found. See Greg KH's blog post about this workflow and how testing can help with LTS releases.

[gctucker]

Kernel / subsystem maintainer wants to know

• Is my code/subsystem being tested? How much coverage?
• Is the kernel size growing? Why? Which subsystems
• Is boot time / test execution time increasing/decreasing? What are the historical trends?

[acherm]

Kernel / subsystem maintainer wants to know

• Is the kernel size growing? Why? Which subsystems

Looks interesting! We are definitely interested to know more about this user story. For instance, which configurations should be considered? Does the "why" imply to know which specific options increase/decrease the size? Size of what and how do you measure it?

Btw, @HugoJPMartin has developed a tool called kpredict https://github.com/HugoJPMartin/kpredict that is capable of predicting the size of a kernel given a .config and it's working for many versions of the kernel (thanks to data collected with TuxML: https://tuxmlweb.istic.univ-rennes1.fr/ https://github.com/TuxML/tuxml/). Though the tool targets other use-cases, one can certainly envision to target this intriguing user-story related to kernel size ;)

[gctucker]

On 19/10/2020 23:10, Nick Desaulniers via groups.io wrote:

additional user story:

A developer wants to slice and dice warnings by toolchain and
toolchain version to see what errors or warnings are compiler
specific/need to be fixed.

[gctucker]

On 21/10/2020 07:39, Corentin Labbe wrote:

As a maintainer, viewing all boot including a specific hardware/driver.
Does my xxxx driver works everywhere ?

Recent example:

• I have added support for PRNG/hash on sun8i-ce, I want to check all boots with this driver.
  More complex recent example:
• since commit bbc4d71d6354 ("net: phy: realtek: fix rtl8211e rx/tx delay config") many board could loose network, I want to see all boots including this patch with this PHY.

[gctucker]

From @lag-linaro:

Usually I send 1-10 failed builds to Tux{Build,Suite}. Then, as soon as the build passes, it gets sent out to KernelCI

This shows how a maintainer can have a "short" loop with reduced coverage but fast feedback within some minutes, and then a "medium" loop with more coverage and feedback within 1h or so.

[gctucker]

A similar use-case is for Peter Zijlstra's kernelci branch with quick feedback using only 7 builds on kernelci.org, and then the changes get merged in the tip tree which gets full build and test coverage.

[gctucker]

When developers get sent a bisection report, or when they find an issue in general, they may not have the platform at their disposal to reproduce or debug the issue. Having the ability for developers to re-run the automated tests with a patch, or some arbitrary kernel branch and to configure how to build the kernel (i.e. debug options...) would make this possible. Ideally, arbitrary commands or even an interactive session on the test platform would also help but that may be much harder to enable in an automated test system.

Here's an example of how this was done by hand, with the intermediate of jobs run on kernelci.org as requested by the developer:
Re: [PATCH RESEND v2 4/5] iommu/tegra-smmu: Rework tegra_smmu_probe_device()
which led to a fix:
[PATCH] iommu/tegra-smmu: Fix mc errors on tegra124-nyan

[mgrzeschik]

For the Lab-Operators it has always been a good way to present
Kernel maintenance / development activities to others.

The current interface is missing the old per lab views.

• View of list of all hardware / SoC currently tested in all labs
• View job history on all devices with a specific hardware / SoC
• View statistics of failing / passing Tests over time for a specific hardware / SoC.
• View a list of hardware / SoC in a specific lab.
• Have a static link to said list which is possible to copy and paste.
• View the status of hardware in a specific lab:
  Are jobs scheduled for a specific hardware?
• View the history of jobs for a specific lab.

[mgrzeschik]

Did this collection of userstories result in any forward discussions or future plans on the project jet?

[gctucker]

We're now starting a working group to drive this effort, see this email thread: https://groups.io/g/kernelci/topic/kernelci_working_group_web/84484924
https://lore.kernel.org/lkml/f3874e7d-ccfe-4a2d-d054-9b7bf9e8b44d@collabora.com/

[gctucker]

From Theodore Ts'O:

For a particular xfstests run, we might have 500 subtests, and the tests are mostly independent. So generic/383 might always be failing on a particular fs config, but if ext4/044 starts failing, that's what we would want to be alerted on. And maybe generic/269 flakes 10% of a time, so it's not a regression if it fails at run N, even if it succeeded at run N-1.....

(where an fs config might be ext4/4k, or ext4/fast_commit, or xfs/v4, or xfs/v5+reflink, etc.)

[tytso]

For ext4 development, I will frequently be running aboutt a dozen file system configs (each in a separate VM). xfstests will generate a junit XML file, and I have scripts that will combine the results from each of the VM's into a single summary file which is e-mailed to me. An example of such a report can be found here.

Along with the junit XML file, gce-xfstests will generate a compressed tar file with all of the test artifacts. For example, if generic/383 fails, there might be an artifact called generic/383.out.bad and generic/383.full, as well as the section of kernel logs while generic/383 was running). If the VM crashes, the full serial console output is also available in the compressed tar file, and that might contain kernel OOPS messages, etc. If the CI dashboard doesn't want to deal with the storage costs, the URL to the test artifact might be made available in some external storage (e.g., Google Cloud Storage) and there could be a shell script that could download the tar file and then make a particular test artifact file available. Or we could upload the full test tarfile at submission time.

In the ideal world, the dashbaord would detect when a test inside xfstests, starts failing, or starts being flakey. This should be done on a per fs config (e.g., ext4/4k, or ext4/1k, or ext4/adv, xfs/v5, f2fs/default, btrfs/default, etc.) and on a per-branch basis. Some tests might always fail on the 4.9 LTS kernel, because it's impractical to backport the bugfix. Or a test for a bug might land in xfstests, before the bugfix lands in mainline. That's not supposed to happen, but in some cases, the test lands in the xfstests git tree, and then something holds up the bugfix for some reason.

I could imagine developers wanting to subscribe to reports on a per-file system basis, or on a per-branch basis (e.g., the proposed xfs backports for the stable kernels), or on a per-fs config basis. (For example, Harshad might especially care about reports for the ext4/fast_commit config.)

At the 2021 LPC File Systems Microconference, we had a discussion from the xfs, ext4, and btrfs maintainers, and we all use xfstests, and we all have our own ad-hoc shell sccripts, or python scripts to try to summarize test results --- and we are mostly just eyeballing reports to notice test regressions or new flakey tests. I believe Josef had some kind of hack to generalte an HTML report using an sqllite database --- but we all agreed that our own own systems were not sufficient, and it would be great if we could try to get some help from kernel.org or kernelci.org --- because none of us are particularly great test engineers. :-)

[tytso]

I should also note that we have an auto-build and test system, which will watch a particular git branch, and if it changes, it will launch a build VM, and then after the kernel is compiled, it will launch N different VM's for different fs configs. We're not currently saving all of the build metadata that kernel CI can accept, but this could be enhanced.

gce-xfstests also has support for running blktests (althoguh blktests as of this writing doesn't have a junit XML output yet). There is also partial support for stress_ng and the phoronix test suite --- but this is not fully automated and not ready for prime-time. Is watching for performance regression considered in scope for Kernel CI?

[tytso]

One other thing. There are new tests which are currently being added to xfstests, so it would be nice if the reports can distinguish between "an test which had previously recorded succesful runs, and now it's started failing", and "a new test was added, so there are no previous records of it running successfully on this branch, and we are seeing it fail". In many cases, the latter case will be because the test corresponds to a newly discovered bug, and so it will fail on all branches and in some cases, all fs configs, until the bugfix lands or is backported to the LTS branches.

[tytso]

And while I'm asking for a pony.... it would be nice if once a regression has been discovered, we can somehow trigger gce-xfstests's Kernel Compilation Service (KCS) to try running the failing test in isolation N times to see if this test reliably fails (e.g., "gce-xfstests ltm --repo git://git.kernel.org/... ---commit deadbeef12 -C 30 -c ext4/4k generic/383") and then start a auto-bisection to determine when a test started failing, or when it started being flakey. (The gce-xfstests ltm command can take a "--bisect-bad commit-id" and one or more "--bisect-good commit-id" arguments and if so, then KCS will use "git bisect" to determine the next kernel to be built and tested and repeat until the first failing commit can be found.) If the test doesn't fail when run 30 times by itself, this may be a test isolation issue, where the entire test suite needs to be run in order to trigger the failure --- or we could have gotten super-unlucky iand it's a very hard to reproduce failure. And at that point, we might need to try running the test 100 or 500 times, and at some point, give up and hand the analysis to a human.

If it is a newly added test which started failing, then ti would be nice if the KCS could try older kernels to determine whether the test was successful in the past, and it's a new test discovering a regression, or it's a bugfix-related test-driven-development and the test has always been failing for a long time.

Not all of this automation should probably live in Kernel CI; clearly some of the logic/support should be in the KCS --- but it would be really nice if after running all of these additional builds and tests, that an e-mail report could be sent explaining what the results of bisection and flaky test probing. "We have discovered a new regression, and it's a flakey test which fails 75% of the time, and the bisection tests appears to show that first commit which caused the flaky test failure was commit deadbeef12." And of course, we need to make sure we only do this investigation once. Basically, this is the sort of thing a human would do to investigate a test failure, and if we can automate it, then we can save a huge amount of developer toil. One of the things we would need to figure out is how much of the automation and logic should live on the Kernel CI side of the abstraction, and how much of this should be built into the KCS side ---- and hopefully this could be made more general so it could be used by another Kernel CI test runner infrastructure systems.

[igaw]

Not sure if it's the right place for wishing for features.

Anyway, for realtime testing, the currently most used approach is to run some workload in the background and run cyclictest to measure the thread wakeup latency. The normal criteria to decide the test passed is the largest observed latency. Though it's not possible to know upfront the highest value to be expected. So it's more a trail and error thing and look at historic data. So it would be really handy to have a graph which plots the trend of observed max latency values for a given board (this is really hardware dependent).

But also histograms are of interest (e.g. is the distribution changing over time?)

While this is not a hard criteria for good/bad it helps to figure if the system performance is generation over time. (over several kernel releases).

So it would be really handy to have builtin support to visualize trends and histograms.

See also:

• Add rt-test JSON feature Linaro/test-definitions#311)
• https://www.osadl.org/Examples-of-latency-regressions.latest-stable-test-latency.0.html

[gctucker]

Thank you @igaw, this is completely the right place to gather use-cases like this. Performance tracking on its own is a complex feature that KernelCI doesn't really cover yet, and visualisation is one aspect of it.

Other things we would need to enable are the ability to translate performance numbers into pass/fail results with arbitrary criteria: absolute thresholds in simple cases, relative thresholds when comparing with other results (previous revision, other platforms, reference branch such as mainline), ability to re-run the same tests to collect more stats when needed... And sometimes there just can't be a pass/fail result to report, if there isn't anything else to compare the data with. But these things are not really part of the user experience, more like data post-processing and orchestration. Still they're useful to understand the bigger picture and what users are expecting.

[freefall75]

Not sure if I'm late to this party :D , but in terms of user stories/ wishlist I would add:

• the list of skipped tcases should be monitored somehow: while some of these SKIPPED testcases could have been skipped forcibly with some static skiplist adopted by the owner of the test runs, the vast majority of SKIPs are automatically determined at runtime by the test suite: it can happen that due to an unwanted/sudden HW/SW misconf or a change in the testcases themselves some previously running test (FAIL/PASS no matter) starts suddenly to be silently SKIPped. Without any monitoring or alerts (maybe on demand/subscription) you end up easily and unvoluntarily not running some test anymore, till something broke somewhere else and you discover you were not testing that thing anymore due to some external factor that led to a skip a long time before. (consider SKIP lists could be in the order of hundreds tcases....)

• known issues: a test result can be reported to KCIDB as XFAIL already, it could be useful if the owner could report an XFAIL and also maybe its log of the reason why it is marked as an XFAIL, BUT most importantly be able to TAG such XFAIL result with an expiration date: known issues tend to be filtered out and hidden by the reporting systems since they are .. well..known noise, but the risk is to mask them forever and just push them out of the window. Ideally, I would like to be able to set an expiration date on reported known issues because usually when you mark a fail as XFAIL you have an idea already if the bug is going to be soon fixed or not..most probably you have already a fix in the kernel or in the test suite itself BUT that change will be picked up from the CI system only a lot later...so in the meantime you mark it XFAIL and filter it out BUT you DO know implicitly that by the dateX your fix should be in and the test should be run again and the result should give you a PASS, not a FAIL anymore: the dashboard could detect and loudly report if a supposed fixed KNOWN ISSUES is FAIL again after the expiration date (even though it's clearly up to the owner to start running the test again once the KNOWN is expired...but in fact, the Dashboard could also alert the owner about the expired KNOWNS which are still unexpectedly NOT being run.)

Thanks,
Cristian

[gctucker]

Thanks @freefall75 for your input, you're not late to the party. KernelCI is very late at getting the new web dashboard rolling though, but we're making progress :)

• Skipped tests

They are currently reported as "unknown" on the current frontend, for example:
https://linux.kernelci.org/test/plan/id/615bcfe043f26880e899a2ed/

This category is for tests that were either not run or failed to produce a valid pass/fail result, for the sake of simplicity. So it sounds like finer grained status would be needed in your use-case.

Then we could detect that a test that used to be run (i.e. producing pass/fail results) started to get skipped or stopped being run, as some kind of regression. The issue there is to determine whether this was done intentionally or not. I think there's a way to solve this, basically it becomes part of a wider issue tracking feature.

• Known issues

The proper way to deal with this is to have a state machine for test cases, as any issue tracker would do. It sounds like we should start designing this with states and transitions now as things are becoming clearer. Here's just a few ideas about how it might work:

* Events causing state transitions:
  * when a test is run, a result is produced: pass, fail, skipped, failed to run
  * when an automated bisection completes or extra checks are run: verified, false positive, bisected
  * manual resolution: for example, when an issue is fixed but the patch is not yet applied
* States:
  * "passing": when the test case is passing and everything is fine
  * "failing": when the test case has been reported as a failure
  * "skipped": when the test case was purposefully not run
  * "unknown": when the test case failed to provide some valid result
  * "regression": when it was detected that the test case was passing and started failing
  * "resolved": when there was a regression and it's now passing again
  * "pending": when there is a known fix but it hasn't reached this branch yet 

It would take more thinking to get this right and actually functional. Also we should keep separate records for the test case raw results and the issue tracking data (regression, resolved...). The tracking data could also include arbitrary things such as a comment when resolving a regression by hand or why a test is skipped or the results are ignored. I believe it would be better to not have to deal with ignoring results on the database / web dashboard side of things, but rather by maintaining skip lists in the test suites themselves as an extra clue for the state machine.

Ultimately we would need to verify that the state machine works according to all the user stories we've collected. We probably will have to make some compromises and start with a simpler subset during implementation, but it's good to not artificially restrict ideas at this early stage.

[fooishbar]

Forking @gctucker's discussion above:

Kernel / subsystem maintainer wants to know

I think the size of the kernel is a pretty niche case so separate from this. Taking the DRM point of view, it would be great to see:

• is the DRM test suite (IGT) actually getting run, and on which devices is it getting run?
• what do the IGT results look like on next compared to mainline? Is the next release going to be problematic?
• looking just at one driver, what are the historical trends? Are there any flaky or problematic tests which only shake out from many runs?
• looking just at one driver, what is the current status? Has someone else broken my driver through their tree?
• what are the historical trends on number of tests being run?

[alexandrasp]

Based on what we already have in the current dashboard, my guess, looking comments above and thinking about the discussion I saw from kernel developers, is that the new dashboard could provide some features like:

• Considering kernel version show tests run. I know we already have this (e.g.), but maybe we can think of an easy/direct way to provide this, fewer clicks or browsing/searching less.

• Provide some visualization to track fails, maybe some matrix/heatmap with the kernel, and tests in time.

• Collect and track well-known tests by fails, as previously mentioned by Theodore Ts'O, maintainers used to know about these. They can pollute the reports and visualizations, causing some noise.

• Provide some historical trends based on the test sets and filtered by kernel version, soc and any other relevant parameter.

[gctucker]

Yes I think that's basically about providing a "language" for making arbitrary queries, and the ability to show graphs rather than just lists and details about particular test results.

[gctucker]

Small update on this topic, we've sent an RFP for UX Analysis and received some proposals. The LF project advisory board is now going through them to decide next steps.

https://kernelci.org/blog/posts/2023/rfp-ux-analysis/