Update action missing latest runs?

[gsnedders]

The "Update gh-pages" action last ran on 27 Oct 2021, 13:27 BST, and produced 4722ba4

However, the only seems to have picked up runs to 2021-10-25, and not any of the more recent runs on wpt.fyi, especially for stable where we had our first Safari 15 run in that intervening period: https://wpt.fyi/runs?label=master&label=stable&max-count=100&from=2021-10-21T00%3A00&product=chrome&product=firefox&product=safari&aligned

[stephenmcgruer]

A few things at play here:

1. The TO_DATE in the build.sh script uses date -d "yesterday 13:00" '+%Y-%m-%d', so when we run on 2021-10-27 we are only passing a --to of 2021-10-26 to compat2021/main.js

2. main.js then fetches exclusively (https://github.com/Ecosystem-Infra/wpt-results-analysis/blob/main/compat-2021/main.js#L95), so the last fetch URL for a run on 2021-10-27 will be:

   https://wpt.fyi/api/runs?&label=master&label=stable&product=chrome&product=firefox&product=safari&aligned=true&max-count=1&from=2021-10-25&to=2021-10-26

3. Even for one day more than that, the wpt.fyi API is giving me back Safari 14.1 still:

   https://wpt.fyi/api/runs?&label=master&label=stable&product=chrome&product=firefox&product=safari&aligned=true&max-count=1&from=2021-10-26&to=2021-10-27

   It's not until I change it to 27 to 28 that I get the Safari 15.0 run:
   https://wpt.fyi/api/runs?&label=master&label=stable&product=chrome&product=firefox&product=safari&aligned=true&max-count=1&from=2021-10-27&to=2021-10-28

   Looking at the JSON, looks like the time_start for that 15.0 run was 2021-10-27T01:21:50.549Z

So we will pick up that Safari 15.0 run tomorrow I think (the gh-pages action runs Mon,Wed,Fri), but it's a reasonable criticism to note that we have a 36-48 hour lag on picking up a given run even if we were running the action every day of the week.

Minimizing this delay would likely involve making build.sh pick up runs until 'today' (not 'yesterday') and then carefully aligning the action times so that the runs for that day have (probably) already happened and have already been ingested to https://github.com/ecosystem-Infra/wpt-results . Unfortunately that isn't work I have time for, but PRs are welcome :).

[gsnedders]

While I know @stephenmcgruer has now manually triggered an update (especially for WPT), and Compat 2021 now shows STP 135, various people have questioned both that and the Browser-Specific-Failure (BSF) graph not being updated more readily.

It's a shame that the action only runs a few times a week (and changing that to daily should be trivial!).

AIUI, the big challenge here is the wpt.fyi API for queries such as the above (e.g., https://wpt.fyi/api/runs?&label=master&label=stable&product=chrome&product=firefox&product=safari&aligned=true&max-count=1&from=2021-10-27&to=2021-10-28) returns the latest test runs where the test_start time is on 2021-10-27, and inevitably those test runs may only complete at some point into 2021-10-28 (or potentially even further into the future; I don't think we strictly have an upper bound on test execution time!), hence we don't actually know what the latest test run for a given date is until all test runs have completed.

Minimizing this delay would likely involve making build.sh pick up runs until 'today' (not 'yesterday') and then carefully aligning the action times so that the runs for that day have (probably) already happened and have already been ingested to https://github.com/ecosystem-Infra/wpt-results .

build.sh does this itself, as part of the BSF update, no? One could imagine updating multiple times per day (after all, we're using little GitHub Actions capacity here!), and just overwriting the data for the last two or three days in case there's been any change about what runs are being used there.

Unfortunately that isn't work I have time for, but PRs are welcome :).

Unfortunately, that's difficult as long as copyright is held by the authors and/or their employers and this isn't licensed under an open-source license (c.f. #58). 🙃

[foolip]

I'm noticing this issue today again because STP 136 was just released, but triggering all the updates manually still doesn't update the data due to the TO_DATE limit.

@stephenmcgruer do you recall why this was included in 2fe8143 in the first place? browser-specific-failures.js already uses aligned runs... Was it something like runs appearing out of order and wanting to leave a margin for that noise to settle?

[stephenmcgruer]

@stephenmcgruer do you recall why this was included in 2fe8143 in the first place? browser-specific-failures.js already uses aligned runs... Was it something like runs appearing out of order and wanting to leave a margin for that noise to settle?

From what I recall, it was a conservative guard against failure due to no runs detected, especially due to the caching logic (which is likely irrelevant to the GitHub action but is relevant locally).

I don't think there's anything fundamentally impossible about bringing the cut-off date closer, but we previously weren't interested in cutting-edge run results for compat2021 and so decided better safe than sorry ;)

[foolip]

Thanks @stephenmcgruer! Sounds like there are issues we might run into, but we can give it a try to shorten the pipeline and see what happens.

[foolip]

Closing in favor of web-platform-tests/results-analysis#65 ahead of archiving this repo.