Aggregate and sub-aggregate tests dashboard

[smlambert]

Nice enhancement idea from Martijn:

Use the external tests to track Java version support amongst popular libraries and frameworks so we can also identify which ones need help.

Display a name-of-project-java-version-tested-against-<pass|fail>

Matrix such as this (with hyperlinks to actual builds):
application status jdkversion implementation platform
scala-jdk8_j9 pass 8 j9 x64_linux_docker
scala-jdk9_hs pass etc...

where we have columns for application, jdkversion, implementation, platform, status (so they can be sorted by same) that includes all apps, elasticsearch, wildfly, etc for jdk8, 9, 10, 11

[smlambert]

Additional discussions with @llxia and @adamfarley have lead me to think about this more... jotting down the ideas here:

we want to be able to provide both highly granular AND aggregate view, of more than just the external tests, this approach can and should be applied to all test types

Thanks to @adamfarley for this picture:

Putting some thought to what data gets us that:
[ # pass, # fail, # excluded, # lastPass, # lastFail, # lastExcluded, platform, impl, version, testGroup ]

That is the granular seed data that then allows us to build up an aggregate view, displaying for
all platforms, all impls, all versions, all testGroups aggregated in diffs between lastPass-pass, lastFail-fail, lastExcluded-excluded (I won't go into detail about why excluded is tricky data to gather ATM).

lastPass+lastFail+lastExcluded = lastTotal
pass+fail+excluded = total
total-lastTotal = # new tests

Or you can filter by platform, by impl, by version, by testGroup to show same type of info, but for jdk8, or for openj9, or for xLinux, or combos of those filters.

[adamfarley]

To achieve the aggregate view in the image above, here's a task breakdown of what needs to be done (WIP).

If something has already been done, or is being done right now, a link to the relevant issue should be added next to the task.

Note: Whenever we "retrieve" data, it should be processed and stored in the database, and the GUI should retrieve it from there at runtime. All the automated bits below should not be run every time the user loads the widget.

Phase 1: Pass/Fail for OpenJDK Regression Testing on xLinux (Numbers only)

• 1. Access test results from the latest OpenJDK Regression test job.
• 2. Correct interpretation of test results (pass vs fail).
• 3. Access to test results a given period of time before that (e.g. 7 days).
• 4. Display the difference between the past/present data points for passed/failed tests.
  Note: a list of unit tests will serve us better than an int, as that allows us to identify the difference better.

Phase 2: Expand functionality to include parameters and aggregate data

• 5. Allow user to specify a job name with wildcards, adapting the dashboard to allow aggregate results (e.g. all platforms for OpenJ9).
• 6. Allow user to use a "special" wildcard to be listed on the dashboard, (e.g. list all platforms covered by these results).
• 7. Allow the user to filter the jobs by parameter value and user (e.g. the last "sanity" job run by auto_test_launcher@fake_email.com).

Phase 3: Adding other test types

• 8. Allow the user to specify multiple sets of the above data (job name, parameter filter, and user).
• 9. Display the results of step 8 in a single aggregate view.
• 10. Verify that all test types used on AdoptOpenJDK show correct results.

Phase 4: Add GUI and Depth

• 11. Identify any existing functionality that could provide a pie chart similar to the one in the picture.
• 12. Connect such functionality to show the numbers output by the logic detailed above.
• 13. Create a new page, or a pop-up or such, to display a list of text.
• 14. Enable "click-through", or "mouse over" for the pie labels and/or slices. These should show the relevant data, as well as URLs to the data sources.

Phase 5: Exclusions

• 15. Enable access to GitHub data; including files, text file contents, issue titles, and issue tags.
• 16. Retrieve and interpret the contents of all the OpenJDK regression exclusion files for a given release.

For identifying which tests have been excluded for a given run, either:

• 17a1. Identify every test file (@test) in a given OpenJDK release.

• 17a2. Follow the trail of files linking the test target/s (or group, e.g. sanity) to the list of tests/directories included in such a test run.

• 17a3. Compare the list of tests to the list of excluded tests, and produce a list of tests that were excluded from that run.
  or

• 17b. Retrieve the list of excluded tests from the test results file.
  or

• 17c. Modify jtreg to output when it excludes a test, and parse the test output for that information.
  (Note: See jtreg's src/share/classes/com/sun/javatest/regtest/config/RegressionTestFinder.java's getIgnoredDirectories() method. Find out what calls that.)
  http://hg.openjdk.java.net/code-tools/jtreg/file/6f00c63c0a98/src/share/classes/com/sun/javatest/regtest/config/RegressionTestFinder.java

• 18. Identify the tags for each GitHub issue associated with an excluded test.

• 19. Determine whether any of the tags identify the issue as a blocker.

• 20. Output the number of excluded tests that are are blockers, the number that are not, and the number of "undetermined" (if any).

Phase 6: Test Plan

• 21. Find a way to connect the test plan for a full release, and the test plan for a sanity run, to this functionality.
  Both this functionality and the test execution should be linked to the same file, requiring no updates when the test plan changes.

Phase 7: Advanced exclusions

• 22. Basically Phase 5, but for all types of test involved in full release testing.

[smlambert]

Initial thoughts upon reading comment above to make sure it is clear, this test-tools repo is not intended to be a rewrite of Axxon. We need to assess what is truly needed, not what was once used (and in past, put us into a state of technical debt).

Some of the list above (disturbingly) implies adoption of heavier process. Stripping down our processes (and subsequently tools), allows us to move quickly and not own/maintain tools that go rapidly stale.

That being said, I will like to introduce features to this test-tools repo from an MVP perspective. What is the minimal amount of information we can gather to better the tasks we need to do. So a clear listing of what we need to do, for starters.

Needed: An aggregate view of test results.

The above list is well beyond what is needed to accomplish an MVP view of aggregate test results. I have strong opinions on what functionality we should support, and will share more upon my return later this week.

[adamfarley]

The Action Plan above was based on two principles:

• These functionalities are, I believe, necessary to achieve the dashboard image above.
  and
• These functionalities are to be investigated, confirmed to work, utilized where present, and implemented where needed. The Action Plan is most definitely not a list of functionalities that need to be written from scratch.

The combination of these functionalities into providing a single widget was intended to meet the requirements of the profile in my use case, which is:

User: Managers, team leaders, and others seeking a high-level quality overview leading up to a release.
Motivation: A need to gauge the quality of the product in terms of test failure and bug severity.
Requirement 1: An aggregate, at-a-glance view of the latest pass/fail test results for all release test types and platforms.
Requirement 2: The severity of the known bugs, and the number of tests known to be affected by them.
Requirement 3: The difference between the results from this week and last week.

I broke these requirements down into phases, and broke those down into the functionalities I thought we'd need to complete those phases.

Requirement 1: Phases 1-4.
Requirement 2: Phases 5 and 7.
Requirement 3: Phase 6.

Also note that the phases are phases. MVP is Phase 1. Then we go for the MVP for the next Phase.

Since this is all proving very wordy, it sounds like it'd be good to get Shelley and Lan Xia into a room (metaphorically) to chew through this. I will readily admit that they know far more about TRSS that I, and it'd be good to either improve the action plan, or replace it with a better plan.

[piyush286]

Just leaving this post #37 (comment) here in case that might be of some help.

[smlambert]

This epic will get split into 2 high-level aspects (separating into server & client work), each can be further broken down into consumable/actionable tasks.

Server

• Define API for aggregate dashboard data
• MVP implementation of aggregate API

Client

• Design / select UI views for displaying aggregate data (client-side)

The API definition and implementation are the most important, as the client/UI can and will change depending on what 3rd party chartware we select.

Initial MVP implementation can be a headless listing of totals that will focus on data that we already have (Pass, Fail, Skipped). Excludes info will need to be tackled in a second phase, as we do not currently collect and share this info via Jenkins jobs, the different types of tests handle excludes in very different ways, and there will need to be many changes to other tools and repos to gather the info required to include into Jenkins job reports.

Any further details can be discussed in the child issues of this epic.

Reminder:
At each stage in the creation of this dashboard, we will need to verify that we building something that is useful. We will need to check if users can actually make useful decisions and take action from it. If not, we will need to rethink the value of this enhancement and decide to redesign or park it.