tests/flake: TestMultipleMountsFiltered fails with missing process info

[willfindlay]

The TestMultipleMountsFiltered tests has been failing occasionally in CI.

Example of a failed run: https://github.com/cilium/tetragon/runs/6492564815?check_suite_focus=true

The root cause seems to be due to missing process fields in the kprobe events, like so:

{"process_kprobe":{"process":{"pid":11225,"start_time":"2022-05-18T15:49:26.557Z"},"parent":{},"function_name":"__x64_sys_write","args":[{"file_arg":{"path":"/tmp2/tmp3/tmp4/tmp5/testfile"}},{"bytes_arg":"eyJwcm9jZXNzX2twcm9iZSI6eyJwcm9jZXNzIjp7InBpZCI6MTEyMjUsInN0YXJ0X3RpbWUiOiIyMDIyLTA1LTE4VDE1OjQ5OjI2LjU1N1oifSwicGFyZW50Ijp7fSwiZnVuY3Rpb25fbmFtZSI6Il9feDY0X3N5c193cml0ZSIsImFyZ3MiOlt7ImZpbGVfYXJnIjp7InBhdGgiOiIvdG1wMi90bXAzL3RtcDQvdG1wNS90ZXN0ZmlsZSJ9fSx7ImJ5dGVzX2FyZyI6ImV5SndjbTlqWlhOelgydHdjbTlpWlNJNmV5SndjbTlqWlhOeklqcDdJbkJwWkNJNk1URXlNalVzSW5OMFlYSjBYM1JwYldVaU9pSXlNREl5TFRBMUxURTRWREUxT2pRNU9qSTJMalUxTjFvaWZTd2ljR0Z5Wlc1MElqcDdmU3dpWm5WdVkzUnBiMjVmYm1GdFpTSTZJbDlmZURZMFgzTjVjMTkzY21sMFpTSXNJbUZ5WjNNaU9sdDdJbVpwYkdWZllYSm5JanA3SW5CaGRHZ2lPaUl2ZEcxd01pOTBiWEF6TDNSdGNEUXZkRzF3TlM5MFpYTjBabWxzWlNKOWZTeDdJblJ5ZFc1allYUmxaRjlpZVhSbGMxOWhjbWNpT25zaVlubDBaWE5mWVhKbklqb2laWGxLZDJOdE9XcGFXRTU2V0RKMGQyTnRPV2xhVTBrMlpYbEtkMk50T1dwYVdFNTZTV3B3TjBsdVFuQmFRMGsyVFZSRmVVMXFWWE5KYms0d1dWaEtNRmd6VW5CaVYxVnBUMmM5UFNJc0ltOXlhV2RmYzJsNlpTSTZJalF4TlRFaWZYMHNleUp6YVhwbFgyRnlaeUk2SWpReE5URWlmVjBzSW1GamRHbHZiaUk2SWt0UVVrOUNSVjlCUTFSSlQwNWZVRTlUVkNKOUxDSjBhVzFsSWpvaU1qQXlNaTB3TlMweE9GUXhOVG8xTURveU55NHhOVGhhSW4wSyJ9LHsic2l6ZV9hcmciOiI0MTQifV0sImFjdGlvbiI6IktQUk9CRV9BQ1RJT05fUE9TVCJ9LCJ0aW1lIjoiMjAyMi0wNS0xOFQxNTo1MDoyNy4xNThaIn0K"},{"size_arg":"846"}],"action":"KPROBE_ACTION_POST"},"time":"2022-05-18T15:50:27.158Z"}
{"process_kprobe":{"process":{"pid":11225,"start_time":"2022-05-18T15:49:26.557Z"},"parent":{},"function_name":"__x64_sys_write","args":[{"file_arg":{"path":"/tmp2/tmp3/tmp4/tmp5/testfile"}},{"bytes_arg":"eyJwcm9jZXNzX2twcm9iZSI6eyJwcm9jZXNzIjp7InBpZCI6MTEyMjUsInN0YXJ0X3RpbWUiOiIyMDIyLTA1LTE4VDE1OjQ5OjI2LjU1N1oifSwicGFyZW50Ijp7fSwiZnVuY3Rpb25fbmFtZSI6Il9feDY0X3N5c193cml0ZSIsImFyZ3MiOlt7ImZpbGVfYXJnIjp7InBhdGgiOiIvdG1wMi90bXAzL3RtcDQvdG1wNS90ZXN0ZmlsZSJ9fSx7InRydW5jYXRlZF9ieXRlc19hcmciOnsiYnl0ZXNfYXJnIjoiZXlKd2NtOWpaWE56WDJ0d2NtOWlaU0k2ZXlKd2NtOWpaWE56SWpwN0luQnBaQ0k2TVRFeU1qVXNJbk4wWVhKMFgzUnBiV1VpT2lJeU1ESXlMVEExTFRFNFZERTFPalE1T2pJMkxqVTFOMW9pZlN3aWNHRnlaVzUwSWpwN2ZTd2lablZ1WTNScGIyNWZibUZ0WlNJNklsOWZlRFkwWDNONWMxOTNjbWwwWlNJc0ltRnlaM01pT2x0N0ltWnBiR1ZmWVhKbklqcDdJbkJoZEdnaU9pSXZkRzF3TWk5MGJYQXpMM1J0Y0RRdmRHMXdOUzkwWlhOMFptbHNaU0o5ZlN4N0ltSjVkR1Z6WDJGeVp5STZJbVY1U25kamJUbHFXbGhPZWxneWRIZGpiVGxwV2xOSk5tVjVTbmRqYlRscVdsaE9la2xxY0RkSmJrSndXa05KTmsxVVJYbE5hbFZ6U1c1T01GbFlTakJZTTFKd1lsZFZhVTlwU1hsTlJFbDVURlJCTVV4VVJUUldSRVV4VDJwUk5VOXFTVEpNYWxVeFRqRnZhV1pUZDJsalIwWjVXbGMxTUVscWNEZG1VM2RwV201V2RWa3pVbkJpTWpWbVltMUdkRnBUU1RaSmJEbG1aVVJaTUZnelRqVmpNVGt6WTIxc01GcFRTWE5KYlVaNVdqTk5hVTlzZERkSmJWcHdZa2RXWmxsWVNtNUphbkEzU1c1Q2FHUkhaMmxQYVVsMlpFY3hkMDFwT1RCaVdFRjZURE5TZEdORVVYWmtSekYzVGxNNU1GcFlUakJhYld4eldsTktPV1pUZURkSmJVbzFaRWRXZWxneVJubGFlVWsyU1cxV05WTnVaR3BpVkd4eFYyeG9UMlZzWjNsa1NHUnFZbFJzY0Zkc1RrcE9iVlkxVTI1a2FtSlViSEZYYkdoUFpXdHNjV05FWkVwaWEwcDNWMnRPU2s1ck1WVlNXR3hPWVd4V2VsTlhOVTlOUm14WlUycENXVTB4U25kWmJHUldZVlU1Y0ZOWWJFNVNSV3cxVkVaU1FrMVZlRlZTVkZKWFVrVlZlRlF5Y0ZKT1ZUbHhVMVJLVFdGc1ZYaFVha1oyWVZkYVZHUXliR3BTTUZvMVYyeGpNVTFGYkhGalJHUnRWVE5rY0ZkdE5WZGtWbXQ2Vlc1Q2FVMXFWbTFaYlRGSFpFWndWRk5VV2twaVJHeHRXbFZTV2sxR1ozcFVhbFpxVFZScmVsa3lNWE5OUm5CVVUxaE9TbUpWV2pWWGFrNU9ZVlU1YzJSRVpFcGlWbkIzV1d0a1YxcHNiRmxUYlRWS1lXNUJNMU5YTlVOaFIxSklXakpzVUdGVmJESmFSV040WkRBeGNFOVVRbWxYUlVZMlZFUk9VMlJIVGtWVldGcHJVbnBHTTFSc1RUVk5SbkJaVkdwQ1lXSlhlSHBYYkU1TFQxZGFWR1ZFWkVwaWJFbzFXa1pqTVdGc2JGbFZiWGhoVW1wc2NGcFciLCJvcmlnX3NpemUiOiI0OTM5In19LHsic2l6ZV9hcmciOiI0OTM5In1dLCJhY3Rpb24iOiJLUFJPQkVfQUNUSU9OX1BPU1QifSwidGltZSI6IjIwMjItMDUtMThUMTU6NTA6MjcuMTU4WiJ9Cg=="},{"size_arg":"1462"}],"action":"KPROBE_ACTION_POST"},"time":"2022-05-18T15:50:27.158Z"}
{"process_kprobe":{"process":{"pid":11225,"start_time":"2022-05-18T15:49:26.557Z"},"parent":{},"function_name":"__x64_sys_write","args":[{"file_arg":{"path":"/tmp2/tmp3/tmp4/tmp5/testfile"}},{"bytes_arg":"eyJwcm9jZXNzX2twcm9iZSI6eyJwcm9jZXNzIjp7InBpZCI6MTEyMjUsInN0YXJ0X3RpbWUiOiIyMDIyLTA1LTE4VDE1OjQ5OjI2LjU1N1oifSwicGFyZW50Ijp7fSwiZnVuY3Rpb25fbmFtZSI6Il9feDY0X3N5c193cml0ZSIsImFyZ3MiOlt7ImZpbGVfYXJnIjp7InBhdGgiOiIvdG1wMi90bXAzL3RtcDQvdG1wNS90ZXN0ZmlsZSJ9fSx7ImJ5dGVzX2FyZyI6ImV5SndjbTlqWlhOelgydHdjbTlpWlNJNmV5SndjbTlqWlhOeklqcDdJbkJwWkNJNk1URXlNalVzSW5OMFlYSjBYM1JwYldVaU9pSXlNREl5TFRBMUxURTRWREUxT2pRNU9qSTJMalUxTjFvaWZTd2ljR0Z5Wlc1MElqcDdmU3dpWm5WdVkzUnBiMjVmYm1GdFpTSTZJbDlmZURZMFgzTjVjMTkzY21sMFpTSXNJbUZ5WjNNaU9sdDdJbVpwYkdWZllYSm5JanA3SW5CaGRHZ2lPaUl2ZEcxd01pOTBiWEF6TDNSdGNEUXZkRzF3TlM5MFpYTjBabWxzWlNKOWZTeDdJbUo1ZEdWelgyRnlaeUk2SW1WNVNuZGpiVGxxV2xoT2VsZ3lkSGRqYlRscFdsTkpObVY1U25kamJUbHFXbGhPZWtscWNEZEpia0p3V2tOSk5rMVVSWGxOYWxWelNXNU9NRmxZU2pCWU0xSndZbGRWYVU5cFNYbE5SRWw1VEZSQk1VeFVSVFJXUkVVeFQycFJOVTlxU1RKTWFsVXhUakZ2YVdaVGQybGpSMFo1V2xjMU1FbHFjRGRtVTNkcFdtNVdkVmt6VW5CaU1qVm1ZbTFHZEZwVFNUWkpiRGxtWlVSWk1GZ3pUalZqTVRrelkyMXNNRnBUU1hOSmJVWjVXak5OYVU5c2REZEpiVnB3WWtkV1psbFlTbTVKYW5BM1NXNUNhR1JIWjJsUGFVbDJaRWN4ZDAxcE9UQmlXRUY2VEROU2RHTkVVWFprUnpGM1RsTTVNRnBZVGpCYWJXeHpXbE5LT1daVGVEZEpibEo1WkZjMWFsbFlVbXhhUmpscFpWaFNiR014T1doamJXTnBUMjV6YVZsdWJEQmFXRTVtV1ZoS2JrbHFiMmxhV0d4TFpESk9kRTlYY0dGWFJUVTJWMFJLTUdReVRuUlBWMnhoVlRCck1scFliRXRrTWs1MFQxZHdZVmRGTlRaVFYzQjNUakJzZFZGdVFtRlJNR3N5VkZaU1JtVlZNWEZXV0U1S1ltczBkMWRXYUV0TlJtZDZWVzVDYVZZeFZuQlVNbU01VUZOSmMwbHRPWGxoVjJSbVl6SnNObHBUU1RaSmFsRjRUbFJGYVdaWU1ITmxlVXA2WVZod2JGZ3lSbmxhZVVrMlNXcFJlRTVVUldsbVZqQnpTVzFHYW1SSGJIWmlhVWsyU1d0MFVWVnJPVU5TVmpsQ1VURlNTbFF3TldaVlJUbFVWa05LT1V4RFNqQmhWekZzU1dwdmFVMXFRWGxOYVRCM1RsTXdlRTlHVVhoT1ZHOHhUVVJ2ZVU1NU5IaE9WR2hoU1c0d1N5SjlMSHNpYzJsNlpWOWhjbWNpT2lJME1UUWlmVjBzSW1GamRHbHZiaUk2SWt0UVVrOUNSVjlCUTFSSlQwNWZVRTlUVkNKOUxDSjBhVzFsSWpvaU1qQXlNaTB3TlMweE9GUXhOVG8xTURveU55NHhOVGhhSW4wSyJ9LHsic2l6ZV9hcmciOiI4NDYifV0sImFjdGlvbiI6IktQUk9CRV9BQ1RJT05fUE9TVCJ9LCJ0aW1lIjoiMjAyMi0wNS0xOFQxNTo1MDoyNy4xNThaIn0K"},{"size_arg":"1422"}],"action":"KPROBE_ACTION_POST"},"time":"2022-05-18T15:50:27.158Z"}

[tpapagian]

These process_kprobe events doesn't seem to be related with this test. In this test we issue a single write with about 10 bytes while these events write about 1000 bytes. Still looking.

[tpapagian]

It seems that it has been a while from the last time we have seen this error. Maybe we can close that for now. I wasn't able to reproduce that locally.

@willfindlay feel free to reopen that if this is not the case.