Deflake `openssl_trace_bpf_test`

[vihangm]

//src/stirling/source_connectors/socket_tracer:openssl_trace_bpf_test has a <90% pass rate in CI.
De-flake the test, run locally with runs_per_test set to a large number to ensure that the pass rate is higher than 90% and then re-enable the test in CI by removing the disabled_flaky_test tag.

[ddelnano]

After reviewing our build buddy logs, it appears all of the failures originate from failing to attach kprobes. These result in the error seen below:

Value of: IsOK(::px::StatusAdapter(source_->Init()))
  Actual: false (Internal : Unable to attach kprobe for __x64_sys_connect using syscall__probe_entry_connect)
Expected: true
I20230113 17:51:57.281163 11814 container_runner.cc:59] docker rm -f node_server_791890848756 &>/dev/null
[  FAILED  ] OpenSSLTraceRawFptrsTest/2.ssl_capture_node_client, where TypeParam = px::stirling::Node12_3_1ContainerWrapper (11274 ms)
[----------] 2 tests from OpenSSLTraceRawFptrsTest/2 (33423 ms total)
[----------] 1 test from OpenSSLTraceRawFptrsTest/3, where TypeParam = px::stirling::Node14_18_1AlpineContainerWrapper
[ RUN      ] OpenSSLTraceRawFptrsTest/3.ssl_capture_ruby_client
I20230113 17:51:57.798910 11814 container_runner.cc:43] Loaded image: bazel/src/stirling/source_connectors/socket_tracer/testing/containers:node_14_18_1_alpine_image
I20230113 17:51:57.798959 11814 container_runner.cc:121] docker run --rm --pid=host --name=node_server_802148102183 bazel/src/stirling/source_connectors/socket_tracer/testing/containers:node_14_18_1_alpine_image
Error: No such object: node_server_802148102183
I20230113 17:51:57.960911 11814 container_runner.cc:160] Container node_server_802148102183 status: 
I20230113 17:51:57.961170 11814 container_runner.cc:169] Container node_server_802148102183 not yet running, will try again (60 attempts remaining).
I20230113 17:51:58.989207 11814 container_runner.cc:160] Container node_server_802148102183 status: running
I20230113 17:51:59.016650 11814 container_runner.cc:191] Container node_server_802148102183 process PID: 28925
I20230113 17:51:59.016683 11814 container_runner.cc:193] Container node_server_802148102183 waiting for log message: Nodejs https server started!
I20230113 17:51:59.042769 11814 container_runner.cc:205] Container node_server_802148102183 status: running
I20230113 17:51:59.042809 11814 container_runner.cc:241] Container node_server_802148102183 is ready.
I20230113 17:52:00.043026 11814 source_connector.cc:35] Initializing source connector: socket_trace_connector
I20230113 17:52:00.043084 11814 linux_headers.cc:94] Obtained Linux version string from `uname`: 5.19.0-0.deb11.2-cloud-amd64
I20230113 17:52:00.043090 11814 linux_headers.cc:635] Detected kernel release (uname -r): 5.19.0-0.deb11.2-cloud-amd64
I20230113 17:52:00.043119 11814 bcc_wrapper.cc:121] Using linux headers found at /lib/modules/5.19.0-0.deb11.2-cloud-amd64/source for BCC runtime.
I20230113 17:52:00.043164 11814 bcc_wrapper.cc:170] Initializing BPF program ...
Error: No such container: nginx_746460065823
I20230113 17:52:08.184121 11814 scoped_timer.h:48] Timer(init_bpf_program) : 8.14 s
cannot attach kprobe, File exists
./src/stirling/source_connectors/socket_tracer/testing/socket_trace_bpf_test_fixture.h:53: Failure
Value of: IsOK(::px::StatusAdapter(source_->Init()))
  Actual: false (Internal : Unable to attach kprobe for __x64_sys_connect using syscall__probe_entry_connect)
Expected: true

[ddelnano]

I believe we are hitting this issue since a kprobe's identifier is based on its function and probed function name, which won't be unique if multiple openssl_trace_bpf_tests are running in parallel (leading to the cannot attach kprobe, File exists error).

I wasn't able to determine which kernel versions are experiencing the issue above, but @etep in the past we've noticed that kprobes based on tracepoints cannot be duplicated in kernel v4.14 but can be in v4.15. So it's possible that this is a 4.14 specific issue.

There is work to have a qemu based test runner (#709), which would allow each test to have its own isolated kernel. This should fix the problem and may stabilize some of our other flaky tests. We could also solve this by providing a mechanism to make the probe function name unique or prevent parallel execution of this test. I don't think the former would be an easy change and would have limited value outside of fixing the test instability. The latter is something we want to avoid to ensure our test suite runs quickly.

[ddelnano]

Regarding which kernel version this is happening on, the output above shows that the test failed under a 5.19 kernel. So this issue is not fixed by more recent kernel versions.

I20230113 17:52:00.043084 11814 linux_headers.cc:94] Obtained Linux version string from uname: 5.19.0-0.deb11.2-cloud-amd64