Non-terminating runtime events

[Sudha247]

While looking at runtime events trace (via olly trace) of ocurrent solver service's Eio backend, @kayceesrk, @talex5 and I found some non-terminating runtime events -- tarides/runtime_events_tools#20. My first instinct was that this is probably a bug in olly. Digging further, I found EV_MAJOR_EPHE_MARK was missing a closing event in the GC. I've attempted to fix that in #12583. #12583 has fixed most of the non-terminating runtime events, although there are still a few left. Opening an issue here to make a note of it.

Here's the list of events that don't terminate on the example I quoted above. Instructions to run this can be found here.

Perfetto has a nice SQL interface with which I managed to extract all the non-terminating events.

Non-terminating events on 5.1.0

List of non-terminated events

id	type	ts	dur	track_id	category	name	depth	stack_id	parent_stack_id	parent_id	arg_set_id	thread_ts	thread_dur	thread_instruction_count	thread_instruction_delta	cat	slice_id
164	internal_slice	685124716188396	-1	0	PERF	major	0	93870467779153	0	NULL	0	NULL	NULL	NULL	NULL	PERF	164
189	internal_slice	685124722170883	-1	0	PERF	domain_condition_wait	1	582844627363265	93870467779153	164	0	NULL	NULL	NULL	NULL	PERF	189
190	internal_slice	685124722241057	-1	0	PERF	domain_condition_wait	2	5473168104511281	582844627363265	189	0	NULL	NULL	NULL	NULL	PERF	190
191	internal_slice	685124722312474	-1	0	PERF	domain_condition_wait	3	5382744721703585	5473168104511281	190	0	NULL	NULL	NULL	NULL	PERF	191
192	internal_slice	685124722378343	-1	0	PERF	domain_condition_wait	4	6424529566506769	5382744721703585	191	0	NULL	NULL	NULL	NULL	PERF	192
193	internal_slice	685124722456996	-1	0	PERF	domain_condition_wait	5	5349743112261505	6424529566506769	192	0	NULL	NULL	NULL	NULL	PERF	193
198	internal_slice	685124726154686	-1	0	PERF	major	6	35760485015501	5349743112261505	193	0	NULL	NULL	NULL	NULL	PERF	198
258	internal_slice	685124730152105	-1	0	PERF	domain_condition_wait	7	1837664836887597	35760485015501	198	0	NULL	NULL	NULL	NULL	PERF	258
276	internal_slice	685124739750282	-1	0	PERF	major	8	6519236769159753	1837664836887597	258	0	NULL	NULL	NULL	NULL	PERF	276
324	internal_slice	685124744168460	-1	0	PERF	major	9	879976138922597	6519236769159753	276	0	NULL	NULL	NULL	NULL	PERF	324
368	internal_slice	685124747002148	-1	0	PERF	major	10	6923549132753425	879976138922597	324	0	NULL	NULL	NULL	NULL	PERF	368
420	internal_slice	685124753251392	-1	0	PERF	major	11	3054084919065405	6923549132753425	368	0	NULL	NULL	NULL	NULL	PERF	420
479	internal_slice	685124760280258	-1	0	PERF	major	12	1733943168323129	3054084919065405	420	0	NULL	NULL	NULL	NULL	PERF	479
562	internal_slice	685124768996123	-1	0	PERF	major	13	3381056199758197	1733943168323129	479	0	NULL	NULL	NULL	NULL	PERF	562
620	internal_slice	685124772921283	-1	0	PERF	major	14	4779937067418369	3381056199758197	562	0	NULL	NULL	NULL	NULL	PERF	620
678	internal_slice	685124778930384	-1	0	PERF	major	15	8606443157172813	4779937067418369	620	0	NULL	NULL	NULL	NULL	PERF	678
751	internal_slice	685124788529074	-1	0	PERF	major	16	7526460533748265	8606443157172813	678	0	NULL	NULL	NULL	NULL	PERF	751
822	internal_slice	685124795535831	-1	0	PERF	major	17	7886344790780037	7526460533748265	751	0	NULL	NULL	NULL	NULL	PERF	822
880	internal_slice	685124800644199	-1	0	PERF	major	18	6443541637957361	7886344790780037	822	0	NULL	NULL	NULL	NULL	PERF	880
954	internal_slice	685124810312195	-1	0	PERF	major	19	3254595022694749	6443541637957361	880	0	NULL	NULL	NULL	NULL	PERF	954
1061	internal_slice	685124825552344	-1	0	PERF	major	20	3075983799911961	3254595022694749	954	0	NULL	NULL	NULL	NULL	PERF	1061
1177	internal_slice	685124845283666	-1	0	PERF	major	21	8553668073922965	3075983799911961	1061	0	NULL	NULL	NULL	NULL	PERF	1177
1287	internal_slice	685124865933130	-1	0	PERF	major	22	7012132410662113	8553668073922965	1177	0	NULL	NULL	NULL	NULL	PERF	1287
1491	internal_slice	685124896361546	-1	0	PERF	major	23	4912854511002733	7012132410662113	1287	0	NULL	NULL	NULL	NULL	PERF	1491
3935	internal_slice	685125171616478	-1	0	PERF	major	24	4909849211339529	4912854511002733	1491	0	NULL	NULL	NULL	NULL	PERF	3935
3989	internal_slice	685125184466995	-1	0	PERF	major	25	1177315222527141	4909849211339529	3935	0	NULL	NULL	NULL	NULL	PERF	3989
3990	internal_slice	685125184467262	-1	0	PERF	major_slice	26	3162542395711763	1177315222527141	3989	0	NULL	NULL	NULL	NULL	PERF	3990
4093	internal_slice	685125189707475	-1	0	PERF	major	27	4661243508956395	3162542395711763	3990	0	NULL	NULL	NULL	NULL	PERF	4093
4094	internal_slice	685125189707739	-1	0	PERF	major_slice	28	6268157343209597	4661243508956395	4093	0	NULL	NULL	NULL	NULL	PERF	4094
5911	internal_slice	685125239783864	-1	0	PERF	major	29	687999584551929	6268157343209597	4094	0	NULL	NULL	NULL	NULL	PERF	5911
5964	internal_slice	685125241780122	-1	0	PERF	major	30	7499407352770741	687999584551929	5911	0	NULL	NULL	NULL	NULL	PERF	5964
6017	internal_slice	685125242894312	-1	0	PERF	major	31	6362623519437761	7499407352770741	5964	0	NULL	NULL	NULL	NULL	PERF	6017
8588	internal_slice	685125306172940	-1	0	PERF	major	32	8741275413303693	6362623519437761	6017	0	NULL	NULL	NULL	NULL	PERF	8588
8642	internal_slice	685125307311839	-1	0	PERF	major	33	3333548459063273	8741275413303693	8588	0	NULL	NULL	NULL	NULL	PERF	8642
8695	internal_slice	685125308467343	-1	0	PERF	major	34	1112103498704837	3333548459063273	8642	0	NULL	NULL	NULL	NULL	PERF	8695
11545	internal_slice	685125385574129	-1	0	PERF	major	35	3206328464338097	1112103498704837	8695	0	NULL	NULL	NULL	NULL	PERF	11545
11598	internal_slice	685125386719587	-1	0	PERF	major	36	3651569626613917	3206328464338097	11545	0	NULL	NULL	NULL	NULL	PERF	11598
11651	internal_slice	685125387866577	-1	0	PERF	major	37	8710991563698393	3651569626613917	11598	0	NULL	NULL	NULL	NULL	PERF	11651
15020	internal_slice	685125473034504	-1	0	PERF	major	38	3168954309615317	8710991563698393	11651	0	NULL	NULL	NULL	NULL	PERF	15020
15074	internal_slice	685125474400187	-1	0	PERF	major	39	1395462495031457	3168954309615317	15020	0	NULL	NULL	NULL	NULL	PERF	15074
15179	internal_slice	685125476525541	-1	0	PERF	major	40	2394338999508909	1395462495031457	15074	0	NULL	NULL	NULL	NULL	PERF	15179
18713	internal_slice	685125571503677	-1	0	PERF	major	41	8332178069058761	2394338999508909	15179	0	NULL	NULL	NULL	NULL	PERF	18713
18767	internal_slice	685125574014528	-1	0	PERF	major	42	2423712011425253	8332178069058761	18713	0	NULL	NULL	NULL	NULL	PERF	18767
18768	internal_slice	685125574014739	-1	0	PERF	major_slice	43	3387167772755283	2423712011425253	18767	0	NULL	NULL	NULL	NULL	PERF	18768
22033	internal_slice	685125720711279	-1	0	PERF	major	44	7258936217299371	3387167772755283	18768	0	NULL	NULL	NULL	NULL	PERF	22033
22087	internal_slice	685125722139368	-1	0	PERF	major	45	1501756882795347	7258936217299371	22033	0	NULL	NULL	NULL	NULL	PERF	22087
22140	internal_slice	685125724546793	-1	0	PERF	major	46	8382857914261419	1501756882795347	22087	0	NULL	NULL	NULL	NULL	PERF	22140
25979	internal_slice	685125944804703	-1	0	PERF	major	47	812799974256979	8382857914261419	22140	0	NULL	NULL	NULL	NULL	PERF	25979
26084	internal_slice	685125949188318	-1	0	PERF	major	48	8596175127893419	812799974256979	25979	0	NULL	NULL	NULL	NULL	PERF	26084
31755	internal_slice	685126269504961	-1	0	PERF	major	49	3220813220030291	8596175127893419	26084	0	NULL	NULL	NULL	NULL	PERF	31755
31808	internal_slice	685126272587229	-1	0	PERF	major	50	2486340332063659	3220813220030291	31755	0	NULL	NULL	NULL	NULL	PERF	31808
31862	internal_slice	685126274436200	-1	0	PERF	major	51	199828039310675	2486340332063659	31808	0	NULL	NULL	NULL	NULL	PERF	31862
39479	internal_slice	685126581317307	-1	0	PERF	major	52	4806905144918443	199828039310675	31862	0	NULL	NULL	NULL	NULL	PERF	39479
39532	internal_slice	685126585550277	-1	0	PERF	major	53	1234017377096531	4806905144918443	39479	0	NULL	NULL	NULL	NULL	PERF	39532
39533	internal_slice	685126585550530	-1	0	PERF	major_slice	54	2357913749588453	1234017377096531	39532	0	NULL	NULL	NULL	NULL	PERF	39533
48482	internal_slice	685127095275865	-1	0	PERF	major	55	5313778627483793	2357913749588453	39533	0	NULL	NULL	NULL	NULL	PERF	48482
48536	internal_slice	685127096393339	-1	0	PERF	major	56	5420675684854461	5313778627483793	48482	0	NULL	NULL	NULL	NULL	PERF	48536
48587	internal_slice	685127101486104	-1	0	PERF	major	57	682265974971577	5420675684854461	48536	0	NULL	NULL	NULL	NULL	PERF	48587
59226	internal_slice	685127551118812	-1	0	PERF	major	58	7206474258366709	682265974971577	48587	0	NULL	NULL	NULL	NULL	PERF	59226
59279	internal_slice	685127552844147	-1	0	PERF	major	59	6304763418792321	7206474258366709	59226	0	NULL	NULL	NULL	NULL	PERF	59279
59332	internal_slice	685127554326657	-1	0	PERF	major	60	1339802988159437	6304763418792321	59279	0	NULL	NULL	NULL	NULL	PERF	59332
75135	internal_slice	685128098217918	-1	0	PERF	major	61	3843739827568809	1339802988159437	59332	0	NULL	NULL	NULL	NULL	PERF	75135
75188	internal_slice	685128101315539	-1	0	PERF	major	62	4308310254442501	3843739827568809	75135	0	NULL	NULL	NULL	NULL	PERF	75188
75241	internal_slice	685128108865497	-1	0	PERF	major	63	1846283620390257	4308310254442501	75188	0	NULL	NULL	NULL	NULL	PERF	75241
75243	internal_slice	685128108865737	-1	0	PERF	major_slice	64	3025257997659655	1846283620390257	75241	0	NULL	NULL	NULL	NULL	PERF	75243
92127	internal_slice	685128915156914	-1	0	PERF	major	65	3416105459314719	3025257997659655	75243	0	NULL	NULL	NULL	NULL	PERF	92127
92181	internal_slice	685128916164971	-1	0	PERF	major	66	4638669419962887	3416105459314719	92127	0	NULL	NULL	NULL	NULL	PERF	92181
92285	internal_slice	685128924296489	-1	0	PERF	major	67	2212416274654239	4638669419962887	92181	0	NULL	NULL	NULL	NULL	PERF	92285
116212	internal_slice	685129864037302	-1	0	PERF	major	68	1237927981143559	2212416274654239	92285	0	NULL	NULL	NULL	NULL	PERF	116212
116268	internal_slice	685129864959957	-1	0	PERF	major	69	4814697016383519	1237927981143559	116212	0	NULL	NULL	NULL	NULL	PERF	116268
116318	internal_slice	685129868061206	-1	0	PERF	major	70	8536711344736775	4814697016383519	116268	0	NULL	NULL	NULL	NULL	PERF	116318
146005	internal_slice	685130990114440	-1	0	PERF	major	71	2458878328523807	8536711344736775	116318	0	NULL	NULL	NULL	NULL	PERF	146005
146060	internal_slice	685130991028099	-1	0	PERF	major	72	1198541415104007	2458878328523807	146005	0	NULL	NULL	NULL	NULL	PERF	146060
175535	internal_slice	685131538309008	-1	0	PERF	major	73	8321043286638623	1198541415104007	146060	0	NULL	NULL	NULL	NULL	PERF	175535
175644	internal_slice	685131540333042	-1	0	PERF	major	74	8188580181815815	8321043286638623	175535	0	NULL	NULL	NULL	NULL	PERF	175644
175645	internal_slice	685131540333330	-1	0	PERF	major_slice	75	4034337033092721	8188580181815815	175644	0	NULL	NULL	NULL	NULL	PERF	175645
175646	internal_slice	685131540333536	-1	0	PERF	major_ephe_mark	76	4784645784674880	4034337033092721	175645	0	NULL	NULL	NULL	NULL	PERF	175646
214337	internal_slice	685132441289282	-1	0	PERF	major	77	2405910071119604	4784645784674880	175646	0	NULL	NULL	NULL	NULL	PERF	214337
214391	internal_slice	685132442275577	-1	0	PERF	major	78	7023109763704440	2405910071119604	214337	0	NULL	NULL	NULL	NULL	PERF	214391
214498	internal_slice	685132444266243	-1	0	PERF	major	79	3526519689228732	7023109763704440	214391	0	NULL	NULL	NULL	NULL	PERF	214498
250928	internal_slice	685133288683939	-1	0	PERF	major	80	4227684434476112	3526519689228732	214498	0	NULL	NULL	NULL	NULL	PERF	250928
250983	internal_slice	685133289510965	-1	0	PERF	major	81	8045446406916836	4227684434476112	250928	0	NULL	NULL	NULL	NULL	PERF	250983
293511	internal_slice	685134204003479	-1	0	PERF	major	82	5298290114878600	8045446406916836	250983	0	NULL	NULL	NULL	NULL	PERF	293511
293567	internal_slice	685134205202845	-1	0	PERF	major	83	981125477016748	5298290114878600	293511	0	NULL	NULL	NULL	NULL	PERF	293567
293677	internal_slice	685134207604672	-1	0	PERF	major	84	8434316861997664	981125477016748	293567	0	NULL	NULL	NULL	NULL	PERF	293677
293751	internal_slice	685134229587249	-1	0	PERF	major	85	5463094463281364	8434316861997664	293677	0	NULL	NULL	NULL	NULL	PERF	293751
336732	internal_slice	685135059907126	-1	0	PERF	major	86	1152204372830360	5463094463281364	293751	0	NULL	NULL	NULL	NULL	PERF	336732
336842	internal_slice	685135061873344	-1	0	PERF	major	87	4852541909269916	1152204372830360	336732	0	NULL	NULL	NULL	NULL	PERF	336842
336950	internal_slice	685135063714523	-1	0	PERF	major	88	6866415945242992	4852541909269916	336842	0	NULL	NULL	NULL	NULL	PERF	336950
382077	internal_slice	685135896660906	-1	0	PERF	major	89	2736987892781508	6866415945242992	336950	0	NULL	NULL	NULL	NULL	PERF	382077
382191	internal_slice	685135898795702	-1	0	PERF	major	90	1618379931664296	2736987892781508	382077	0	NULL	NULL	NULL	NULL	PERF	382191
382244	internal_slice	685135899890144	-1	0	PERF	major	91	3797741507925644	1618379931664296	382191	0	NULL	NULL	NULL	NULL	PERF	382244
382352	internal_slice	685135902167324	-1	0	PERF	major	92	5777618382531200	3797741507925644	382244	0	NULL	NULL	NULL	NULL	PERF	382352
383460	internal_slice	685135922320357	-1	0	PERF	major	93	3782089060777908	5777618382531200	382352	0	NULL	NULL	NULL	NULL	PERF	383460
423444	internal_slice	685136703782253	-1	0	PERF	major	94	261061491705272	3782089060777908	383460	0	NULL	NULL	NULL	NULL	PERF	423444
423554	internal_slice	685136705893534	-1	0	PERF	major	95	4505163030938748	261061491705272	423444	0	NULL	NULL	NULL	NULL	PERF	423554
423719	internal_slice	685136709220504	-1	0	PERF	major	96	2012719120692880	4505163030938748	423554	0	NULL	NULL	NULL	NULL	PERF	423719
423830	internal_slice	685136711310142	-1	0	PERF	major	97	7808659888396452	2012719120692880	423719	0	NULL	NULL	NULL	NULL	PERF	423830
425394	internal_slice	685136742566697	-1	0	PERF	major	98	5576953709567176	7808659888396452	423830	0	NULL	NULL	NULL	NULL	PERF	425394
473460	internal_slice	685137759306184	-1	0	PERF	major	99	7119467154447724	5576953709567176	425394	0	NULL	NULL	NULL	NULL	PERF	473460
473520	internal_slice	685137760366756	-1	0	PERF	major	100	1169771539074464	7119467154447724	473460	0	NULL	NULL	NULL	NULL	PERF	473520
473575	internal_slice	685137761493900	-1	0	PERF	major	101	8032487392784788	1169771539074464	473520	0	NULL	NULL	NULL	NULL	PERF	473575
523313	internal_slice	685138838918308	-1	0	PERF	major	102	6554061118860760	8032487392784788	473575	0	NULL	NULL	NULL	NULL	PERF	523313
523419	internal_slice	685138841107160	-1	0	PERF	major	103	3723186428021596	6554061118860760	523313	0	NULL	NULL	NULL	NULL	PERF	523419
523477	internal_slice	685138842190375	-1	0	PERF	major	104	5717895218140080	3723186428021596	523419	0	NULL	NULL	NULL	NULL	PERF	523477
574817	internal_slice	685140216638217	-1	0	PERF	major	105	3980216119325828	5717895218140080	523477	0	NULL	NULL	NULL	NULL	PERF	574817
574928	internal_slice	685140218820888	-1	0	PERF	major	106	6687843547246312	3980216119325828	574817	0	NULL	NULL	NULL	NULL	PERF	574928
574929	internal_slice	685140218821212	-1	0	PERF	major_slice	107	3308091384376302	6687843547246312	574928	0	NULL	NULL	NULL	NULL	PERF	574929
574930	internal_slice	685140218821399	-1	0	PERF	major_ephe_mark	108	8585751427181463	3308091384376302	574929	0	NULL	NULL	NULL	NULL	PERF	574930
627737	internal_slice	685141526288700	-1	0	PERF	major	109	4027220917169039	8585751427181463	574930	0	NULL	NULL	NULL	NULL	PERF	627737
627791	internal_slice	685141527378799	-1	0	PERF	major	110	6782335935619479	4027220917169039	627737	0	NULL	NULL	NULL	NULL	PERF	627791
627848	internal_slice	685141528485166	-1	0	PERF	major	111	6053464242397583	6782335935619479	627791	0	NULL	NULL	NULL	NULL	PERF	627848
627901	internal_slice	685141529586686	-1	0	PERF	major	112	4936216970286999	6053464242397583	627848	0	NULL	NULL	NULL	NULL	PERF	627901
679168	internal_slice	685142734201775	-1	0	PERF	major	113	6030491383928719	4936216970286999	627901	0	NULL	NULL	NULL	NULL	PERF	679168
679277	internal_slice	685142736618806	-1	0	PERF	major	114	4069385948204439	6030491383928719	679168	0	NULL	NULL	NULL	NULL	PERF	679277
679387	internal_slice	685142738839562	-1	0	PERF	major	115	1143578736627087	4069385948204439	679277	0	NULL	NULL	NULL	NULL	PERF	679387
728776	internal_slice	685144080814053	-1	0	PERF	major	116	54459885175703	1143578736627087	679387	0	NULL	NULL	NULL	NULL	PERF	728776
728829	internal_slice	685144082174763	-1	0	PERF	major	117	5640872004158351	54459885175703	728776	0	NULL	NULL	NULL	NULL	PERF	728829
728882	internal_slice	685144084548181	-1	0	PERF	major	118	1065262368455063	5640872004158351	728829	0	NULL	NULL	NULL	NULL	PERF	728882
778282	internal_slice	685145356101537	-1	0	PERF	major	119	592258929719695	1065262368455063	728882	0	NULL	NULL	NULL	NULL	PERF	778282
778335	internal_slice	685145357068232	-1	0	PERF	major	120	7553299475738519	592258929719695	778282	0	NULL	NULL	NULL	NULL	PERF	778335
778387	internal_slice	685145358055843	-1	0	PERF	major	121	52988303601551	7553299475738519	778335	0	NULL	NULL	NULL	NULL	PERF	778387
825532	internal_slice	685146713477962	-1	0	PERF	major	122	7491504401638807	52988303601551	778387	0	NULL	NULL	NULL	NULL	PERF	825532
825639	internal_slice	685146715320931	-1	0	PERF	major	123	1520732577555855	7491504401638807	825532	0	NULL	NULL	NULL	NULL	PERF	825639
825694	internal_slice	685146716343336	-1	0	PERF	major	124	1346027720154007	1520732577555855	825639	0	NULL	NULL	NULL	NULL	PERF	825694
825751	internal_slice	685146717312226	-1	0	PERF	major	125	654157320346511	1346027720154007	825694	0	NULL	NULL	NULL	NULL	PERF	825751
875825	internal_slice	685148282432099	-1	0	PERF	major	126	5934655614767511	654157320346511	825751	0	NULL	NULL	NULL	NULL	PERF	875825
875988	internal_slice	685148286631720	-1	0	PERF	major	127	4774881057165711	5934655614767511	875825	0	NULL	NULL	NULL	NULL	PERF	875988
933076	internal_slice	685150079572527	-1	0	PERF	major	128	2093037103613847	4774881057165711	875988	0	NULL	NULL	NULL	NULL	PERF	933076
933184	internal_slice	685150081727568	-1	0	PERF	major	129	4444506986137487	2093037103613847	933076	0	NULL	NULL	NULL	NULL	PERF	933184
933292	internal_slice	685150084159312	-1	0	PERF	major	130	4990883587243415	4444506986137487	933184	0	NULL	NULL	NULL	NULL	PERF	933292
974298	internal_slice	685150976801002	-1	0	PERF	major	131	4271796327300495	4990883587243415	933292	0	NULL	NULL	NULL	NULL	PERF	974298
974461	internal_slice	685150981212533	-1	0	PERF	major	132	1342940005266327	4271796327300495	974298	0	NULL	NULL	NULL	NULL	PERF	974461
974625	internal_slice	685150985316770	-1	0	PERF	major	133	4687670386733967	1342940005266327	974461	0	NULL	NULL	NULL	NULL	PERF	974625
974699	internal_slice	685150986571338	-1	0	PERF	major	134	8918084594706839	4687670386733967	974625	0	NULL	NULL	NULL	NULL	PERF	974699
1023043	internal_slice	685152502385635	-1	0	PERF	major	135	4317913699625359	8918084594706839	974699	0	NULL	NULL	NULL	NULL	PERF	1023043
1023151	internal_slice	685152505698377	-1	0	PERF	major	136	7094956163719063	4317913699625359	1023043	0	NULL	NULL	NULL	NULL	PERF	1023151
1023208	internal_slice	685152507905667	-1	0	PERF	major	137	259658254983055	7094956163719063	1023151	0	NULL	NULL	NULL	NULL	PERF	1023208
1065050	internal_slice	685153753596609	-1	0	PERF	major	138	7378264755606935	259658254983055	1023208	0	NULL	NULL	NULL	NULL	PERF	1065050
1065153	internal_slice	685153756143535	-1	0	PERF	major	139	6351925264717199	7378264755606935	1065050	0	NULL	NULL	NULL	NULL	PERF	1065153
1116127	internal_slice	685156012447782	-1	0	PERF	major	140	8120584989428631	6351925264717199	1065153	0	NULL	NULL	NULL	NULL	PERF	1116127
1116179	internal_slice	685156013870631	-1	0	PERF	major	141	8199509323975567	8120584989428631	1116127	0	NULL	NULL	NULL	NULL	PERF	1116179
1116237	internal_slice	685156015183616	-1	0	PERF	major	142	7788195777347991	8199509323975567	1116179	0	NULL	NULL	NULL	NULL	PERF	1116237
1170850	internal_slice	685157618234659	-1	0	PERF	major	143	5644870619201935	7788195777347991	1116237	0	NULL	NULL	NULL	NULL	PERF	1170850
1171013	internal_slice	685157622279885	-1	0	PERF	major	144	5663350427203479	5644870619201935	1170850	0	NULL	NULL	NULL	NULL	PERF	1171013
1171122	internal_slice	685157624974074	-1	0	PERF	major	145	2900524874987407	5663350427203479	1171013	0	NULL	NULL	NULL	NULL	PERF	1171122
1171123	internal_slice	685157624974400	-1	0	PERF	major_slice	146	7521543097990905	2900524874987407	1171122	0	NULL	NULL	NULL	NULL	PERF	1171123
1230105	internal_slice	685159300281422	-1	0	PERF	major	147	4411348966706101	7521543097990905	1171123	0	NULL	NULL	NULL	NULL	PERF	1230105
1230165	internal_slice	685159301742245	-1	0	PERF	major	148	655615117160129	4411348966706101	1230105	0	NULL	NULL	NULL	NULL	PERF	1230165
1230221	internal_slice	685159303188102	-1	0	PERF	major	149	8849596752981133	655615117160129	1230165	0	NULL	NULL	NULL	NULL	PERF	1230221
1287660	internal_slice	685161090526843	-1	0	PERF	major	150	4937425023988457	8849596752981133	1230221	0	NULL	NULL	NULL	NULL	PERF	1287660
1287767	internal_slice	685161092773835	-1	0	PERF	major	151	4797382315623109	4937425023988457	1287660	0	NULL	NULL	NULL	NULL	PERF	1287767
1287877	internal_slice	685161095044386	-1	0	PERF	major	152	7698597600012209	4797382315623109	1287767	0	NULL	NULL	NULL	NULL	PERF	1287877
1339408	internal_slice	685162580501113	-1	0	PERF	major	153	6570756829750173	7698597600012209	1287877	0	NULL	NULL	NULL	NULL	PERF	1339408
1339465	internal_slice	685162581874038	-1	0	PERF	major	154	7201262826400729	6570756829750173	1339408	0	NULL	NULL	NULL	NULL	PERF	1339465
1339627	internal_slice	685162586139300	-1	0	PERF	major	155	8810538329209301	7201262826400729	1339465	0	NULL	NULL	NULL	NULL	PERF	1339627
1395773	internal_slice	685164416559159	-1	0	PERF	major	156	5370265538071457	8810538329209301	1339627	0	NULL	NULL	NULL	NULL	PERF	1395773
1395832	internal_slice	685164418018493	-1	0	PERF	major	157	8243856694688429	5370265538071457	1395773	0	NULL	NULL	NULL	NULL	PERF	1395832
1395885	internal_slice	685164419526672	-1	0	PERF	major	158	6267112824270793	8243856694688429	1395832	0	NULL	NULL	NULL	NULL	PERF	1395885
1445950	internal_slice	685166051408744	-1	0	PERF	major	159	746991791206629	6267112824270793	1395885	0	NULL	NULL	NULL	NULL	PERF	1445950
1446113	internal_slice	685166054772032	-1	0	PERF	major	160	8397466794549137	746991791206629	1445950	0	NULL	NULL	NULL	NULL	PERF	1446113
1446221	internal_slice	685166056860868	-1	0	PERF	major	161	8186179420557757	8397466794549137	1446113	0	NULL	NULL	NULL	NULL	PERF	1446221
1497424	internal_slice	685167500282290	-1	0	PERF	major	162	6921749555454905	8186179420557757	1446221	0	NULL	NULL	NULL	NULL	PERF	1497424
1497533	internal_slice	685167503155328	-1	0	PERF	major	163	313733190330357	6921749555454905	1497424	0	NULL	NULL	NULL	NULL	PERF	1497533
1497591	internal_slice	685167505037771	-1	0	PERF	major	164	1818035151673473	313733190330357	1497533	0	NULL	NULL	NULL	NULL	PERF	1497591
1545332	internal_slice	685168949790226	-1	0	PERF	major	165	1018286247233741	1818035151673473	1497591	0	NULL	NULL	NULL	NULL	PERF	1545332
1545387	internal_slice	685168953691779	-1	0	PERF	major	166	2563750602278825	1018286247233741	1545332	0	NULL	NULL	NULL	NULL	PERF	1545387
1545441	internal_slice	685168957861312	-1	0	PERF	major	167	3108929011465989	2563750602278825	1545387	0	NULL	NULL	NULL	NULL	PERF	1545441
1595651	internal_slice	685170544415756	-1	0	PERF	major	168	641381796928625	3108929011465989	1545441	0	NULL	NULL	NULL	NULL	PERF	1595651
1595710	internal_slice	685170546145481	-1	0	PERF	major	169	5192001310179293	641381796928625	1595651	0	NULL	NULL	NULL	NULL	PERF	1595710
1595765	internal_slice	685170547473672	-1	0	PERF	major	170	7454989973642137	5192001310179293	1595710	0	NULL	NULL	NULL	NULL	PERF	1595765
1595818	internal_slice	685170548993824	-1	0	PERF	major	171	7247196995720725	7454989973642137	1595765	0	NULL	NULL	NULL	NULL	PERF	1595818
1650130	internal_slice	685172550179767	-1	0	PERF	major	172	3419379786911841	7247196995720725	1595818	0	NULL	NULL	NULL	NULL	PERF	1650130
1650243	internal_slice	685172555006994	-1	0	PERF	major	173	2701752923289325	3419379786911841	1650130	0	NULL	NULL	NULL	NULL	PERF	1650243
1650352	internal_slice	685172557812612	-1	0	PERF	major	174	2955829656982665	2701752923289325	1650243	0	NULL	NULL	NULL	NULL	PERF	1650352
1706453	internal_slice	685174247869621	-1	0	PERF	major	175	463509356367141	2955829656982665	1650352	0	NULL	NULL	NULL	NULL	PERF	1706453
1706510	internal_slice	685174248917017	-1	0	PERF	major	176	6017171525432401	463509356367141	1706453	0	NULL	NULL	NULL	NULL	PERF	1706510
1762795	internal_slice	685175987623711	-1	0	PERF	major	177	7314676719335677	6017171525432401	1706510	0	NULL	NULL	NULL	NULL	PERF	1762795
1762849	internal_slice	685175988678922	-1	0	PERF	major	178	7679248767542137	7314676719335677	1762795	0	NULL	NULL	NULL	NULL	PERF	1762849
1762902	internal_slice	685175989763191	-1	0	PERF	major	179	8680990634585909	7679248767542137	1762849	0	NULL	NULL	NULL	NULL	PERF	1762902
1784306	internal_slice	685177098940396	-1	0	PERF	stw_leader	180	2670172626217810	8680990634585909	1762902	0	NULL	NULL	NULL	NULL	PERF	1784306

Non-terminating events on #12583

List of non-terminated events

id	type	ts	dur	track_id	category	name	depth	stack_id	parent_stack_id	parent_id	arg_set_id	thread_ts	thread_dur	thread_instruction_count	thread_instruction_delta	cat	slice_id
189	internal_slice	685423768780116	-1	0	PERF	domain_condition_wait	0	5963369316465045	0	NULL	0	NULL	NULL	NULL	NULL	PERF	189
190	internal_slice	685423768836981	-1	0	PERF	domain_condition_wait	1	497954583781637	5963369316465045	189	0	NULL	NULL	NULL	NULL	PERF	190
1809400	internal_slice	685488580122940	-1	0	PERF	stw_leader	2	6153086810092546	497954583781637	190	0	NULL	NULL	NULL	NULL	PERF	1809400

There are still a couple of non-terminated events left, namely domain_condition_wait and stw_leader. My hunch is that these domain_condition_waits are parent events of the non-terminating stw_leader, and hence they are running forever until the program terminates, too. In particular, I suspect there's some code path in stw_leader which doesn't emit the closing event. This almost always happens (only) for the very last stw_leader event in Domain 0, in the solver service I mentioned above, and other programs I've tested it in (binary_trees and LU_decomposition). Fixing this would make olly useful. Would appreciate any pointers on how to fix it.

Note: The tests were run on 5.1.0 because 'olly' depends on ppx, and support for ppx on trunk is not available yet.

cc @sadiqj

[gasche]

Another reason why an event may be non-terminated would be that the runtime-events machinery shut itself down before we get to the corresponding CAML_EV_END call. This looks like a plausible explanation for your stray stw_leader event: that events is emitted when a STW section starts, but notice that the runtime-events machinery is itself stopped by a STW section:

ocaml/runtime/runtime_events.c

Lines 230 to 247 in 74f8c99

	/* Write a lifecycle event and then trigger a stop the world to tear down the
	ring buffers */
	void caml_runtime_events_destroy(void) {
	if (atomic_load_acquire(&runtime_events_enabled)) {
	write_to_ring(
	EV_RUNTIME, (ev_message_type){.runtime=EV_LIFECYCLE}, EV_RING_STOP, 0,
	NULL, 0);
	/* clean up runtime_events when we exit if we haven't been instructed to
	preserve the file. */
	int remove_file = preserve_ring ? 0 : 1;
	do {
	caml_try_run_on_all_domains(&stw_teardown_runtime_events,
	&remove_file, NULL);
	}
	while( atomic_load_acquire(&runtime_events_enabled) );
	}
	}

So I would guess that the following happens:

• someone calls caml_runtime_events_destroy
• the STW section starts, emitting a STW_LEADER end begin event
• the stw leader stops runtime events
• when it exits the STW section, runtime events are disabled and no STW_LEADER end event is emitted

The domain_condition_wait might have a similar cause, I don't know. It might be the call to Condition.wait that is inside Domain.join, and it could be that this call returns after runtime events have been destroyed?

[eutro]

The STW_LEADER assessment seems correct. In fact, it seems to me that the STW_LEADER end event getting dropped should be guaranteed, since stw_teardown_runtime_events protects it with a barrier:

ocaml/runtime/runtime_events.c

Lines 187 to 199 in 94c5327

	/* Stop-the-world which calls the teardown code */
	static void stw_teardown_runtime_events(caml_domain_state *domain_state,
	void *remove_file_data, int num_participating,
	caml_domain_state **participating_domains) {
	caml_global_barrier();
	if (participating_domains[0] == domain_state) {
	int remove_file = *(int*)remove_file_data;
	runtime_events_teardown_raw(remove_file);
	}
	caml_global_barrier();
	}

The DOMAIN_CONDITION_WAIT section could also easily end after runtime events have been destroyed, given that the teardown STW request could be handled by the backup thread.