s2n_aead_aes_test times out occasionally

[maddeleine]

Security issue notifications

If you discover a potential security issue in s2n we ask that you notify
AWS Security via our vulnerability reporting page. Please do not create a public github issue.

Problem:

We have noticed the s2n_aead_aes_test will occasionally take way longer than usual to run and then timeout. This appears to be flaky as a retry fixes the problem. Here's an example of the failing test.

Solution:

Figure out what is causing the flakiness. It's possible it has something to do with not having aes-optimized hardware available, but that's just a theory. Investigation is needed.

[goatgoose]

After investigating this issue a bit, it seems that this isn't a problem with the aead_aes test itself, but rather an issue with the kTLS CI job running all tests very slowly sometimes. The slow CI job causes the aead_aes test to timeout since it's the slowest unit test.

I compared a successful kTLS test run to a kTLS test run with the aead_aes test timing out, and many of the other tests were also slow:

Test	Success Duration (s)	Timeout Duration (s)	% Increase
s2n_quic_transport_params_extension_test	0.05	521.34	1042580%
s2n_ssl_prf_test	0.04	83.91	209675%
s2n_tls13_secrets_rfc8448_test	0.07	36.38	51871%
s2n_ktls_io_sendfile_test	0.08	41.32	51550%
s2n_pq_kem_test	0.05	23.2	46300%
s2n_record_size_test	0.09	23.39	25889%

We should determine why the kTLS job is sometimes running some of the tests very slowly. It might be related to how qemu is configured.