multiple requests to Treeherder group_results API for same revision scheduled by same mozCI decision task, should be only one

[Archaeopteryx]

During an investigation, it has been noticed mozCI scheduled 532 requests for the Treeherder group_results API endpoint in a 2 minute window.

Except for 2 revisions (1 and 7 requests), each revision had 15-45 requests. mozCI's decision task schedules 15 tasks which do the requests - it seems each will do a request for each revision at least once.

There is also the question if mozCI should request data for each revision multiple times over time and get everything returned but this is likely out of scope here.

cc @jmaher

[jmaher]

looking into this locally I ran mozci on a push to get tasks and it never cached it, so it queried each time. This got me questioning- do we have a cache setup for our CI system, and is it always there or sometimes there? @marco-c ?

in addition, I found that we never got data from the treeherder api for certain tasks like:

• marionette
• test-verify
• test-apk* (newer tasks from the fenix merge)

because these never come in on the first initial query, we query treeherder API for each of these.

A few ways to potentially solve this:

1. adjust TH api to denote when missing data and return some code in the API results; then have mozci read that, therefore adding something to the cache
2. add a list of tasks that are not supported to get group results to mozci; I hacked this locally and this seems to work; this has a problem of getting out of date, but could solve our problems in the short term.

@marco-c do you have a preference or other ideas?

[Archaeopteryx]

1. adjust TH api to denote when missing data and return some code in the API results; then have mozci read that, therefore adding something to the cache

Treeherder itself only learns of tasks when all its dependencies have completed successfully. Before, such tasks are in the scheduled state. For analysis, e.g. about upcoming worker load with the ability to investigate which repositories, pushes and job types cause these, it would be beneficial to also ingest this state. We can discuss later if this shall be implemented.

The current implementation causes pushes to sometimes report all their tasks are complete: when a task completes but dependent tasks still need to report back as 'pending' from 'scheduled' before. Just something to be aware.

[jmaher]

the TH side of things will be difficult as we specifically start the query from the group table: https://github.com/mozilla/treeherder/blob/35d37127df89f3ed59503550f1907e6937d0635e/treeherder/log_parser/failureline.py#L189

if a task has no groups, we will not be able to know about it easily.

It seems easier to have mozci email warnings when there are tasks that are not saved and then we build a list of tasks that have no groups. There will be edge cases (like a task that failed prematurely).

Another option would be to have mozci add a blank shim for the given task in the cache- this means we would query multiple times while iterating through the first time (initial query + 1x for each task with no groups), but when we query again, the task will be discovered and there is no need for the additional queries.

[jmaher]

the group_status query returns json like this;
{"DcIkhAv8TJOX_3aXoeN-xQ":{"browser/components/backup/tests/xpcshell/xpcshell.toml":true,"browser/extensions/formautofill/test/unit/xpcshell.toml":true,"dom/base/test/unit/xpcshell.toml":true,"toolkit/components/url-classifier/tests/unit/xpcshell.toml":true,"toolkit/crashreporter/test/unit_ipc/xpcshell-phc.toml":true,"tools/profiler/tests/xpcshell/xpcshell.toml":true},"QnbLmx-lS_2bOwpFB7m-0w":{"devtools/client/responsive/test/browser/browser.toml":true}