add method to update test traces/artifacts with modern Chrome

[brendankenny]

two TODOs for this:

• lighthouse/core/lib/tracehouse/trace-processor.js

  Lines 88 to 89 in 1b843de

  	// COMPAT: support pre-m67 test traces before `args.data` added to all navStart events.
  	// TODO: remove next line when old test traces (e.g. progressive-app-m60.json) are updated.

• https://github.com/GoogleChrome/lighthouse/pull/14287/files#diff-5a3b0489bd666c8cb3258c1713d8251a9e1f5bb7c91daa37c71e3bba0111fc2aR519-R520

Some of the test traces/devtoolsLogs are old enough now that they're interfering with code we write by forcing it to handle cases that haven't been relevant for years.

Main issues with easy trace updates:

• they are used in many tests, and any replacements would need to be sure to cover approximately the same test surface area (e.g. if a trace is of a page with redirects that are tested for somewhere, any replacement would also need to have redirects)
• many of them were generated for precisely the purpose of their original PR from bespoke pages with no hint of how to replicate today

To avoid getting back into this situation in a few years, we discussed creating test pages and a method to update test traces. Similar to yarn update:sample-artifacts this will likely still be noisy and fiddly, but it should be much easier to update than trying to parse what was special about a trace from just the name "jumpy-cls-m90.json" and stepping through blame.

There are a lot of test traces, so we can start gradually, e.g. the first TODO above mentions including fallback code for pre-m67 test traces, which is just a handful of them.

The test pages could live in the Lighthouse repo (they could be the smoke tests, even), but we should consider hosting them in https://github.com/ChromeDevTools/performance-stories and sharing them with DevTools perf panel testing.

[jackfranklin]

To avoid getting back into this situation in a few years, we discussed creating test pages and a method to update test traces. Similar to yarn update:sample-artifacts this will likely still be noisy and fiddly, but it should be much easier to update than trying to parse what was special about a trace from just the name "jumpy-cls-m90.json" and stepping through blame.

I have a long term vision for having tooling like this in the performance-stories repo and being able to run npm generate-trace --story=foo-bar and have it spit out foo-bar-$DATE.json.gz or similar.

[connorjclark]

somewhat related #13886

[paulirish]

Also lets use git LFS for these traces. And not use minify-trace against these fixtures.

[connorjclark]

We discussed the benefits of re-generating these traces at runtime, instead of offline-then-checked in; which sounds useful, though I'd like to control that behavior with a flag (enabled in CI) and continue checking in actual files to use locally (so that things are reproducible). We can consider this more later but... for now just writing a script to update these traces is good.

[connorjclark]

1. list trace assets, sort by date
2. document what they represent
3. try to dynamically create a couple of the oldest ones, using a user flow to extract a trace/devtoolsLog

[connorjclark]

Trace fixtures sorted by author date:

2016-03-07 21:53:11 +0000 core/test/fixtures/traces/animation.json
2016-03-07 21:53:11 +0000 core/test/fixtures/traces/load.json
2016-04-23 15:29:30 +0000 core/test/fixtures/traces/progressive-app.json
2016-04-29 20:05:53 +0000 core/test/fixtures/traces/devtools-homepage-w-screenshots-trace.json
2016-05-31 18:16:48 +0000 core/test/fixtures/traces/trace-user-timings.json
2016-08-10 19:46:31 +0000 core/test/fixtures/traces/network-records.json
2016-10-02 00:23:03 +0000 core/test/fixtures/traces/threeframes-blank_content_more.json
2016-10-07 21:12:43 +0000 core/test/fixtures/traces/bad-nav-start-ts.json
2016-12-14 01:43:59 +0000 core/test/fixtures/traces/tracingstarted-after-navstart.json
2017-01-05 02:44:54 +0000 core/test/fixtures/traces/preactjs.com_ts_of_undefined.json
2017-02-04 01:37:42 +0000 core/test/fixtures/traces/airhorner_no_fcp.json
2017-02-04 01:37:42 +0000 core/test/fixtures/traces/no_fmp_event.json
2017-02-04 01:37:42 +0000 core/test/fixtures/traces/no_navstart_event.json
2017-03-24 02:41:15 +0000 core/test/fixtures/traces/backgrounded-tab-missing-paints.json
2017-04-30 23:37:09 +0000 core/test/fixtures/traces/progressive-app-m60.json
2017-05-10 17:24:32 +0000 core/test/fixtures/traces/progressive-app-m60.devtools.log.json
2017-05-15 21:03:24 +0000 core/test/fixtures/traces/screenshots/progressive-app-frame-0.jpg
2017-05-15 21:03:24 +0000 core/test/fixtures/traces/screenshots/progressive-app-frame-1.jpg
2017-05-15 21:03:24 +0000 core/test/fixtures/traces/screenshots/progressive-app-frame-2.jpg
2017-05-15 21:03:24 +0000 core/test/fixtures/traces/screenshots/progressive-app-frame-3.jpg
2017-05-15 21:03:24 +0000 core/test/fixtures/traces/screenshots/progressive-app-frame-4.jpg
2017-05-15 21:03:24 +0000 core/test/fixtures/traces/screenshots/progressive-app-frame-5.jpg
2017-05-15 21:03:24 +0000 core/test/fixtures/traces/screenshots/progressive-app-frame-6.jpg
2017-05-15 21:03:24 +0000 core/test/fixtures/traces/screenshots/progressive-app-frame-7.jpg
2017-05-15 21:03:24 +0000 core/test/fixtures/traces/screenshots/progressive-app-frame-8.jpg
2017-05-15 21:03:24 +0000 core/test/fixtures/traces/screenshots/progressive-app-frame-9.jpg
2017-06-06 22:07:42 +0000 core/test/fixtures/traces/site-with-redirect.devtools.log.json
2017-06-06 22:07:42 +0000 core/test/fixtures/traces/site-with-redirect.json
2019-04-29 19:36:02 +0000 core/test/fixtures/traces/no-tracingstarted-m74.json
2019-10-30 15:56:16 +0000 core/test/fixtures/traces/iframe-m79.devtoolslog.json
2019-10-30 15:56:16 +0000 core/test/fixtures/traces/iframe-m79.trace.json
2019-11-10 04:32:59 +0000 core/test/fixtures/traces/lcp-m78.devtools.log.json
2019-11-10 04:32:59 +0000 core/test/fixtures/traces/lcp-m78.json
2020-04-01 15:14:10 +0000 core/test/fixtures/traces/cutoff-load-m83.devtoolslog.json
2020-04-01 15:14:10 +0000 core/test/fixtures/traces/cutoff-load-m83.trace.json
2020-07-13 20:48:15 +0000 core/test/fixtures/traces/amp-m86.devtoolslog.json
2020-07-13 20:48:15 +0000 core/test/fixtures/traces/amp-m86.trace.json
2020-07-30 14:06:22 +0000 core/test/fixtures/traces/speedindex-1ms-layout-m84.devtoolslog.json
2020-07-30 14:06:22 +0000 core/test/fixtures/traces/speedindex-1ms-layout-m84.trace.json
2020-08-28 16:00:00 +0000 core/test/fixtures/traces/cpu-profiler-m86.trace.json
2020-11-26 02:05:33 +0000 core/test/fixtures/traces/frame-metrics-m89.devtools.log.json
2020-11-26 02:05:33 +0000 core/test/fixtures/traces/frame-metrics-m89.json
2020-12-02 19:18:27 +0000 core/test/fixtures/traces/video-embeds-m84.devtools.log.json
2020-12-02 19:18:27 +0000 core/test/fixtures/traces/video-embeds-m84.json
2021-02-03 20:33:16 +0000 core/test/fixtures/traces/frame-metrics-m90.devtools.log.json
2021-02-03 20:33:16 +0000 core/test/fixtures/traces/frame-metrics-m90.json
2021-02-09 21:18:25 +0000 core/test/fixtures/traces/jumpy-cls-m90.devtoolslog.json
2021-02-09 21:18:25 +0000 core/test/fixtures/traces/jumpy-cls-m90.json
2021-06-24 19:08:33 +0000 core/test/fixtures/traces/timespan-trace-m91.json
2022-04-27 23:42:14 +0000 core/test/fixtures/traces/timespan-responsiveness-m103.trace.json
2023-01-26 00:48:59 +0000 core/test/fixtures/traces/pid-change.json

Command:

git ls-tree -r --name-only HEAD -z core/test/fixtures/traces/ | TZ=UTC xargs -0n1 -I_ sh -c 'git --no-pager log --follow --date=iso-local --format="%ad _" -- _ | tail -1' | sort

No longer used:

traces/devtools-homepage-w-screenshots-trace.json
traces/network-records.json

[jackfranklin]

Humble nudge towards if any of these make sense in https://github.com/ChromeDevTools/performance-stories

[brendankenny]

@jackfranklin we've generally settled on being fine with using external URLs for generating some of these (for now, at least), so performance-stories seems like a nice middle ground...external but much more controllable than real sites. Also designed by perf experts to replicate specific scenarios :)

• How stable are you imagining these will be?
• If we wanted to do something, like, add unicode to the event names in the user-timings story, how much work would that also entail on the DevTools side (or should we instead think of existing stories as primarily read-only, short of things like LCP spec changes or whatever)