feat: improved e2e test report #31323

makemesteaks · 2025-03-26T15:48:07Z

Description

Adds much better custom E2E test reports and summaries to GitHub Actions.

Related issues

Fixes: #31442

Manual testing steps

Ask QA if they are satisfied with E2E test reporting

Screenshots/Recordings

Pre-merge author checklist

I've followed MetaMask Contributor Docs and MetaMask Extension Coding Standards.
I've completed the PR template to the best of my ability
I’ve included tests if applicable
I’ve documented my code using JSDoc format if applicable
I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

github-actions · 2025-03-26T15:48:21Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

metamaskbot · 2025-03-28T12:53:37Z

Builds ready [16bd292]

builds: chrome, firefox
builds (flask): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
bundle size: Bundle Size Stats
user-actions-benchmark: User Actions Stats
storybook: Storybook
typescript migration: Dashboard
all CircleCI artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6, 7, 8
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
- ui: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
- content-script: 0
- offscreen: 0

UI Startup Metrics (1187 ± 55 ms)

Platform	BuildType	Page	Metric	Mean (ms)	Min (ms)	Max (ms)	Std Dev (ms)	P 75 (ms)	P 95 (ms)
Chrome	Browserify	Home	uiStartup	1187	1096	1344	55	1218	1299
			load	1032	901	1168	53	944	992
			domContentLoaded	1025	895	1163	53	953	987
			domInteractive	16	13	32	4	16	27
			firstPaint	756	131	1172	391	257	988
			backgroundConnect	10	6	53	8	9	10
			firstReactRender	20	15	55	6	20	36
			getState	10	4	42	7	6	8
			initialActions	0	0	1	0	0	1
			loadScripts	812	688	950	52	840	914
			setupStore	8	4	23	3	7	9
	Webpack	Home	uiStartup	1003	781	1280	84	954	996
			load	851	644	951	63	886	937
			domContentLoaded	845	633	942	63	882	925
			domInteractive	17	13	50	8	15	39
			firstPaint	510	57	929	355	873	922
			backgroundConnect	17	11	53	8	16	39
			firstReactRender	15	12	37	4	14	26
			getState	7	4	15	2	7	8
			initialActions	0	0	1	0	0	1
			loadScripts	843	623	931	62	881	920
			setupStore	7	5	14	2	8	9
Firefox	Browserify	Home	uiStartup	1375	1198	2125	160	1415	1749
			load	1239	1071	1974	153	1263	1585
			domContentLoaded	1238	1070	1974	153	1263	1585
			domInteractive	103	38	209	31	87	98
			firstPaint	NaN	NaN	NaN	NaN	NaN	NaN
			backgroundConnect	24	17	54	7	25	35
			firstReactRender	22	19	34	3	25	28
			getState	7	3	25	3	7	9
			initialActions	0	0	1	0	0	1
			loadScripts	1216	1052	1950	152	1240	1563
			setupStore	6	4	13	2	6	7
	Webpack	Home	uiStartup	984	840	1703	178	896	993
			load	859	732	1462	154	839	988
			domContentLoaded	859	732	1462	153	837	987
			domInteractive	116	32	174	24	150	95
			firstPaint	NaN	NaN	NaN	NaN	NaN	NaN
			backgroundConnect	22	13	115	17	20	34
			firstReactRender	18	15	28	2	19	23
			getState	8	4	73	9	7	9
			initialActions	0	0	1	0	0	1
			loadScripts	842	718	1442	150	829	973
			setupStore	9	4	67	9	7	8

Bundle size diffs [🚀 Bundle size reduced!]

background: -572.44 KiB (-8.82%)
ui: -974.6 KiB (-12.59%)
common: -874.33 KiB (-8.51%)

metamaskbot · 2025-03-31T08:28:59Z

Builds ready [1ea8454]

builds: chrome, firefox
builds (flask): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
bundle size: Bundle Size Stats
user-actions-benchmark: User Actions Stats
storybook: Storybook
typescript migration: Dashboard
all CircleCI artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6, 7, 8
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
- ui: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
- content-script: 0
- offscreen: 0

UI Startup Metrics (1196 ± 55 ms)

Platform	BuildType	Page	Metric	Mean (ms)	Min (ms)	Max (ms)	Std Dev (ms)	P 75 (ms)	P 95 (ms)
Chrome	Browserify	Home	uiStartup	1196	1091	1382	55	1239	1276
			load	1046	950	1236	56	1236	986
			domContentLoaded	1039	945	1232	57	947	981
			domInteractive	16	12	36	5	16	29
			firstPaint	821	85	1163	375	270	984
			backgroundConnect	10	6	55	8	9	10
			firstReactRender	19	14	43	4	19	26
			getState	11	4	32	7	6	8
			initialActions	0	0	1	0	0	1
			loadScripts	824	714	1029	56	859	900
			setupStore	7	5	15	2	7	9
	Webpack	Home	uiStartup	949	796	1330	85	961	983
			load	804	589	967	67	841	917
			domContentLoaded	798	567	950	67	834	905
			domInteractive	15	12	54	8	13	37
			firstPaint	478	57	930	342	828	851
			backgroundConnect	16	11	53	9	15	41
			firstReactRender	14	12	37	4	13	26
			getState	7	3	12	2	7	8
			initialActions	0	0	1	0	0	0
			loadScripts	795	557	938	68	832	895
			setupStore	7	5	16	2	7	9
Firefox	Browserify	Home	uiStartup	1371	1179	2022	159	1402	1776
			load	1235	1062	1839	151	1275	1587
			domContentLoaded	1235	1062	1839	151	1275	1587
			domInteractive	104	40	179	31	87	96
			firstPaint	NaN	NaN	NaN	NaN	NaN	NaN
			backgroundConnect	24	16	114	12	24	35
			firstReactRender	22	19	45	4	23	26
			getState	6	3	17	2	7	8
			initialActions	0	0	1	0	0	1
			loadScripts	1212	1046	1816	150	1250	1553
			setupStore	6	4	16	2	6	7
	Webpack	Home	uiStartup	979	824	1546	180	884	990
			load	860	720	1349	159	791	928
			domContentLoaded	859	720	1348	159	791	928
			domInteractive	118	56	241	28	158	98
			firstPaint	NaN	NaN	NaN	NaN	NaN	NaN
			backgroundConnect	20	13	99	14	20	36
			firstReactRender	18	16	25	2	18	23
			getState	9	4	73	11	7	9
			initialActions	0	0	2	0	0	1
			loadScripts	843	708	1326	155	777	915
			setupStore	7	4	63	6	7	8

Bundle size diffs [🚀 Bundle size reduced!]

background: -635.53 KiB (-9.79%)
ui: -971.99 KiB (-12.56%)
common: -805.6 KiB (-7.84%)

metamaskbot · 2025-04-10T23:22:43Z

Builds ready [e2709a2]

builds: chrome, firefox
builds (flask): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
bundle size: Bundle Size Stats
user-actions-benchmark: User Actions Stats
storybook: Storybook
typescript migration: Dashboard
all CircleCI artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6, 7, 8
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
- ui: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
- content-script: 0
- offscreen: 0

UI Startup Metrics (1237 ± 61 ms)

Platform	BuildType	Page	Metric	Mean (ms)	Min (ms)	Max (ms)	Std Dev (ms)	P 75 (ms)	P 95 (ms)
Chrome	Browserify	Home	uiStartup	1237	1127	1435	61	1274	1355
			load	1082	967	1295	57	1131	992
			domContentLoaded	1077	963	1290	57	1126	988
			domInteractive	18	14	63	7	17	31
			firstPaint	733	113	1295	435	211	305
			backgroundConnect	6	4	30	3	7	9
			firstReactRender	19	16	39	3	20	23
			getState	13	4	40	8	7	9
			initialActions	0	0	1	0	0	1
			loadScripts	836	723	1042	55	860	947
			setupStore	8	4	14	2	7	9
	Webpack	Home	uiStartup	2160	1740	2552	186	2258	2453
			load	1681	1339	2031	137	1736	1959
			domContentLoaded	1674	1335	2019	135	1729	1947
			domInteractive	17	12	80	13	14	56
			firstPaint	165	64	423	63	233	95
			backgroundConnect	32	9	281	36	36	9
			firstReactRender	180	53	372	109	57	93
			getState	18	4	278	37	7	9
			initialActions	3	1	8	1	3	5
			loadScripts	1669	1334	1990	130	1726	1917
			setupStore	22	6	256	30	46	9
Firefox	Browserify	Home	uiStartup	1309	1121	1756	126	1356	1571
			load	1166	1002	1624	119	1228	1431
			domContentLoaded	1166	1001	1624	119	1227	1431
			domInteractive	100	38	252	34	88	96
			firstPaint	NaN	NaN	NaN	NaN	NaN	NaN
			backgroundConnect	19	13	52	7	20	35
			firstReactRender	22	19	54	5	22	32
			getState	10	4	191	19	7	8
			initialActions	0	0	1	0	0	1
			loadScripts	1148	988	1607	119	1213	1430
			setupStore	8	4	170	17	6	7
	Webpack	Home	uiStartup	1522	1332	1904	105	1596	1717
			load	1309	1159	1644	96	1373	1491
			domContentLoaded	1308	1158	1643	95	1373	1491
			domInteractive	93	46	182	21	89	97
			firstPaint	NaN	NaN	NaN	NaN	NaN	NaN
			backgroundConnect	23	16	67	8	25	40
			firstReactRender	35	29	51	5	38	44
			getState	10	4	32	7	8	9
			initialActions	0	0	2	1	1	1
			loadScripts	1288	1142	1577	94	1357	1471
			setupStore	8	4	33	5	7	9

metamaskbot · 2025-04-14T22:04:26Z

Builds ready [46e3fc4]

builds: chrome, firefox
builds (flask): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
bundle size: Bundle Size Stats
user-actions-benchmark: User Actions Stats
storybook: Storybook
typescript migration: Dashboard
all CircleCI artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6, 7, 8
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
- ui: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
- content-script: 0
- offscreen: 0

UI Startup Metrics (1204 ± 71 ms)

Platform	BuildType	Page	Metric	Mean (ms)	Min (ms)	Max (ms)	Std Dev (ms)	P 75 (ms)	P 95 (ms)
Chrome	Browserify	Home	uiStartup	1204	1108	1544	71	1233	1338
			load	1041	960	1332	64	1063	1179
			domContentLoaded	1035	952	1325	64	1057	1174
			domInteractive	17	13	63	8	16	31
			firstPaint	706	83	1207	405	1048	1129
			backgroundConnect	7	4	20	2	7	10
			firstReactRender	19	15	41	4	20	27
			getState	14	5	45	8	21	26
			initialActions	0	0	1	0	0	1
			loadScripts	797	719	1064	59	819	924
			setupStore	8	4	28	3	8	12
	Webpack	Home	uiStartup	2181	1739	2667	192	2333	2466
			load	1658	1294	1975	156	1793	1887
			domContentLoaded	1651	1288	1948	154	1787	1877
			domInteractive	16	11	57	10	13	44
			firstPaint	171	61	411	53	190	281
			backgroundConnect	30	13	428	43	31	58
			firstReactRender	233	56	396	114	330	360
			getState	17	4	331	33	17	42
			initialActions	3	2	11	1	4	5
			loadScripts	1646	1286	1925	152	1785	1863
			setupStore	23	7	301	33	24	38
Firefox	Browserify	Home	uiStartup	1338	1149	1808	115	1391	1596
			load	1192	1017	1674	120	1244	1426
			domContentLoaded	1192	1017	1673	120	1244	1425
			domInteractive	102	38	265	29	113	156
			firstPaint	NaN	NaN	NaN	NaN	NaN	NaN
			backgroundConnect	20	12	108	11	21	37
			firstReactRender	22	18	40	4	22	30
			getState	7	4	37	5	8	13
			initialActions	0	0	2	0	0	1
			loadScripts	1174	987	1654	120	1224	1410
			setupStore	6	3	34	4	6	8
	Webpack	Home	uiStartup	1491	1349	1871	104	1562	1691
			load	1272	1147	1554	89	1339	1466
			domContentLoaded	1271	1147	1554	89	1339	1465
			domInteractive	75	30	160	22	76	144
			firstPaint	NaN	NaN	NaN	NaN	NaN	NaN
			backgroundConnect	23	14	79	10	24	41
			firstReactRender	35	29	49	5	38	47
			getState	10	5	32	7	9	31
			initialActions	0	0	1	0	1	1
			loadScripts	1252	1128	1532	89	1323	1448
			setupStore	9	5	41	7	8	25

Bundle size diffs

background: 0 Bytes (0%)
ui: 0 Bytes (0%)
common: 0 Bytes (0%)

metamaskbot · 2025-04-15T05:17:04Z

Builds ready [41ac80f]

builds: chrome, firefox
builds (flask): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
bundle size: Bundle Size Stats
user-actions-benchmark: User Actions Stats
storybook: Storybook
typescript migration: Dashboard
all CircleCI artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6, 7, 8
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
- ui: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
- content-script: 0
- offscreen: 0

UI Startup Metrics (1218 ± 58 ms)

Platform	BuildType	Page	Metric	Mean (ms)	Min (ms)	Max (ms)	Std Dev (ms)	P 75 (ms)	P 95 (ms)
Chrome	Browserify	Home	uiStartup	1218	1091	1418	58	1257	1340
			load	1057	951	1198	52	1088	1152
			domContentLoaded	1052	948	1188	51	1081	1147
			domInteractive	17	13	36	5	16	29
			firstPaint	715	87	1205	415	1068	1144
			backgroundConnect	6	4	10	1	7	9
			firstReactRender	22	15	53	7	22	38
			getState	13	5	43	8	18	30
			initialActions	0	0	1	0	0	1
			loadScripts	813	723	947	49	838	911
			setupStore	8	5	23	3	8	15
	Webpack	Home	uiStartup	2107	1739	2440	184	2263	2385
			load	1612	1316	1876	144	1732	1822
			domContentLoaded	1605	1313	1871	143	1723	1811
			domInteractive	17	11	76	13	13	58
			firstPaint	170	67	389	57	202	287
			backgroundConnect	27	10	344	34	27	54
			firstReactRender	192	55	509	122	324	359
			getState	18	4	332	40	16	29
			initialActions	3	1	9	1	3	5
			loadScripts	1600	1311	1870	143	1720	1788
			setupStore	21	6	295	39	19	32
Firefox	Browserify	Home	uiStartup	1325	1166	1774	118	1387	1579
			load	1183	1008	1637	123	1252	1451
			domContentLoaded	1183	1008	1637	123	1252	1451
			domInteractive	109	40	437	48	116	184
			firstPaint	NaN	NaN	NaN	NaN	NaN	NaN
			backgroundConnect	20	12	122	13	19	34
			firstReactRender	22	18	36	3	22	27
			getState	7	4	41	4	7	9
			initialActions	0	0	1	0	0	1
			loadScripts	1165	997	1624	123	1237	1431
			setupStore	6	3	32	5	6	23
	Webpack	Home	uiStartup	1535	1362	2076	129	1603	1828
			load	1306	1176	1816	111	1365	1524
			domContentLoaded	1305	1176	1815	111	1365	1524
			domInteractive	81	53	300	31	87	131
			firstPaint	NaN	NaN	NaN	NaN	NaN	NaN
			backgroundConnect	23	15	68	8	23	44
			firstReactRender	37	30	56	5	39	49
			getState	9	5	33	4	9	20
			initialActions	1	0	2	1	1	1
			loadScripts	1285	1155	1794	110	1345	1509
			setupStore	9	5	42	6	9	28

Bundle size diffs

background: 0 Bytes (0%)
ui: 0 Bytes (0%)
common: 0 Bytes (0%)

metamaskbot added the team-dev-ops DevOps team label Mar 26, 2025

ci: test failed report

ace3f41

makemesteaks force-pushed the ci-test-report branch from 42acd19 to ace3f41 Compare March 26, 2025 16:31

makemesteaks added 2 commits March 26, 2025 17:05

ci: test

be8721b

ci: draft

e38f8bb

makemesteaks changed the title ~~ci: test failed report~~ ci: test failed report [DRAFT] Mar 27, 2025

makemesteaks force-pushed the ci-test-report branch from 45661d4 to f2e5314 Compare March 28, 2025 11:49

ci: test

b41b1ad

makemesteaks force-pushed the ci-test-report branch from f2e5314 to b41b1ad Compare March 28, 2025 11:50

ci: update

16bd292

ci: test

a318244

makemesteaks force-pushed the ci-test-report branch from b674b9a to a318244 Compare March 30, 2025 14:23

ci: run after e2e

583878b

makemesteaks force-pushed the ci-test-report branch from 7b5cf17 to 583878b Compare March 30, 2025 14:37

makemesteaks added 2 commits March 30, 2025 16:04

ci: path

2bf7091

ci: remove retries to test

1ea8454

itsyoboieltr linked an issue Mar 31, 2025 that may be closed by this pull request

Improve E2E test reporting #31442

Closed

9 tasks

itsyoboieltr added team-extension-platform Extension Platform team and removed team-dev-ops DevOps team labels Apr 7, 2025

itsyoboieltr self-assigned this Apr 7, 2025

itsyoboieltr added this to PR review queue Apr 7, 2025

github-project-automation bot moved this to Needs dev review in PR review queue Apr 7, 2025

itsyoboieltr changed the title ~~ci: test failed report [DRAFT]~~ feat: improved e2e test report Apr 7, 2025

itsyoboieltr moved this from Needs dev review to Needs more work from the author in PR review queue Apr 7, 2025

itsyoboieltr added 3 commits April 8, 2025 00:04

Update run-e2e.yml

21e7301

fix: remove

8c5497b

Merge branch 'main' into ci-test-report

eced2ec

itsyoboieltr added 5 commits April 10, 2025 22:00

fix: run attempt

ea46e4a

feat: firefox report

cb164cb

test

5a562a6

debug

9e7531c

fix: d

78e31c1

itsyoboieltr moved this from Needs more work from the author to Needs dev review in PR review queue Apr 10, 2025

itsyoboieltr force-pushed the ci-test-report branch 3 times, most recently from 99051c0 to 78e31c1 Compare April 11, 2025 12:11

itsyoboieltr added 4 commits April 15, 2025 00:35

feat: separation of runs

a4bc354

Merge branch 'main' into ci-test-report

cc7bd88

fix: hyphen

9c20aa1

feat: failed tests first

46e3fc4

itsyoboieltr previously approved these changes Apr 14, 2025

View reviewed changes

itsyoboieltr enabled auto-merge April 14, 2025 21:52

few small changes

41ac80f

HowardBraham dismissed itsyoboieltr’s stale review via 41ac80f April 15, 2025 04:25

HowardBraham approved these changes Apr 15, 2025

View reviewed changes

itsyoboieltr approved these changes Apr 15, 2025

View reviewed changes

itsyoboieltr added this pull request to the merge queue Apr 15, 2025

github-project-automation bot moved this from Needs dev review to Review finalised - Ready to be merged in PR review queue Apr 15, 2025

Merged via the queue into main with commit 703ed49 Apr 15, 2025
178 checks passed

itsyoboieltr deleted the ci-test-report branch April 15, 2025 06:22

github-project-automation bot moved this from Review finalised - Ready to be merged to Merged, Closed or Archived in PR review queue Apr 15, 2025

github-actions bot locked and limited conversation to collaborators Apr 15, 2025

metamaskbot added the release-12.18.0 Issue or pull request that will be included in release 12.18.0 label Apr 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: improved e2e test report #31323

feat: improved e2e test report #31323

Uh oh!

makemesteaks commented Mar 26, 2025 •

edited by HowardBraham

Loading

Uh oh!

github-actions bot commented Mar 26, 2025

Uh oh!

metamaskbot commented Mar 28, 2025

Uh oh!

metamaskbot commented Mar 31, 2025

Uh oh!

metamaskbot commented Apr 10, 2025

Uh oh!

metamaskbot commented Apr 14, 2025

Uh oh!

metamaskbot commented Apr 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

feat: improved e2e test report #31323

feat: improved e2e test report #31323

Uh oh!

Conversation

makemesteaks commented Mar 26, 2025 • edited by HowardBraham Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues

Manual testing steps

Screenshots/Recordings

Pre-merge author checklist

Pre-merge reviewer checklist

Uh oh!

github-actions bot commented Mar 26, 2025

Uh oh!

metamaskbot commented Mar 28, 2025

Uh oh!

metamaskbot commented Mar 31, 2025

Uh oh!

metamaskbot commented Apr 10, 2025

Uh oh!

metamaskbot commented Apr 14, 2025

Uh oh!

metamaskbot commented Apr 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

makemesteaks commented Mar 26, 2025 •

edited by HowardBraham

Loading