Implement new tls tracing method behind stirling cli flag (feature toggle)

[ddelnano]

Summary: Implement new tls tracing method behind stirling cli flag (feature toggle)

The tls tracing method added in this PR determines a connection's identity (socket fd) through a different mechanism from our existing tracing. Instead of relying on struct offsets of user space data structures, it accesses the socket fd via the underlying socket syscalls while SSL_write / SSL_read calls occur. This is a prerequisite to support BoringSSL because its rolling release style makes the previous method of user space offsets untenable. This has the added benefit of reducing our maintenance cost for our existing OpenSSL tracing. Assuming future versions of OpenSSL maintain the same contract, we will not require any code changes to support them -- Pixie will gain OpenSSL v3 support once this new tracing is the default (as mentioned in the later testing).

Our assumption is that the applications that explicitly set the socket fd on the SSL struct (the applications supported by our previous tracing technique) -- nginx, python all use Openssl with its native BIO interface. In order to verify that assumption, this change will be feature flagged and monitored carefully as its enabled on internal clusters. The existing conn_stats_bytes metric will be monitored for volume of tls traffic traced to verify there is no loss of instrumentation coverage between the new and old method.

Relevant Issues: #692

Type of change: /kind feature

Test Plan: Verified the following

• New test passes with enable_openssl_v3_testing bool flag enabled (P336). This verifies that the new tracing technique and the feature toggle are functional since our previous tracing does not support OpenSSL v3
• Existing openssl_trace_bpf_test and netty_tls_trace_bpf_test passes (P335). This is explicitly mentioned since openssl_trace_bpf_test is still disabled Deflake openssl_trace_bpf_test #699
• Verified the bool_flag added from PR feedback conditionally enables the OpenSSL v3 testing (P337)
• Validate metrics from Instrument OpenSSL tracing to detect validity of assumptions for new tls tracing method #1161 to verify our assumptions are correct

[JamesMBartlett]

My understanding is that there's two flags to control the new behaviour: a build time flag and a runtime flag. It seems to me like both flags need to be set for the new behaviour to be enabled. As far as I can tell, if the build-time flag is enabled but the runtime flag is not, it will attempt to trace openssl v3 traffic, but fail since the new fd tracing is not enabled (maybe?). And if only the runtime flag is enabled it won't attempt to trace openssl v3 traffic. If this is the case I think it might make sense to just have a runtime flag and not have a build time flag. Additionally, you won't be able to change the build-time flag with launchdarkly, so that side of things will be easier with just a runtime flag.

[ddelnano]

My understanding is that there's two flags to control the new behaviour: a build time flag and a runtime flag

@JamesMBartlett regarding your comment above, is the build time flag you are referencing the enable_openssl_v3_testing config setting? If not, can you please identify what you are referencing?

The enable_openssl_v3_testing setting is temporary and only intended for running tests. If kLibSSLMatchers was configurable at runtime, I would have opted for the test to override its entries rather than using a build setting.

This new method of tls tracing should provide the same coverage as our existing method, meaning that a PEM should trace the same amount of bytes before and after the change. However, I want to verify that assumption. In order to test that, I intend to roll out the new implementation while keeping our supported tls tracing targets constant (OpenSSL v3 must remain disabled as the feature toggle is enabled). Once the new implementation is fully rolled out and the coverage is validated against our existing method, we can remove the config setting and release OpenSSL v3 support.

[JamesMBartlett]

My understanding is that there's two flags to control the new behaviour: a build time flag and a runtime flag

@JamesMBartlett regarding your comment above, is the build time flag you are referencing the enable_openssl_v3_testing config setting? If not, can you please identify what you are referencing?

Yes that's the flag I was referring to.

The enable_openssl_v3_testing setting is temporary and only intended for running tests. If kLibSSLMatchers was configurable at runtime, I would have opted for the test to override its entries rather than using a build setting.

This new method of tls tracing should provide the same coverage as our existing method, meaning that a PEM should trace the same amount of bytes before and after the change. However, I want to verify that assumption. In order to test that, I intend to roll out the new implementation while keeping our supported tls tracing targets constant (OpenSSL v3 must remain disabled as the feature toggle is enabled). Once the new implementation is fully rolled out and the coverage is validated against our existing method, we can remove the config setting and release OpenSSL v3 support.

I see, I misunderstood the purpose since kLibSSLMatchers is not in a testing file, but this makes sense now.

[ddelnano]

Moving this back to a draft since there is some additional testing the Stirling team wants to do prior to pushing forward with this.

[ddelnano]

@etep this is ready for review again.