Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(integv2): reduce parameter selection #5161

Merged
merged 6 commits into from
Mar 12, 2025

Conversation

johubertj
Copy link
Contributor

@johubertj johubertj commented Mar 4, 2025

Resolved issues:

Integ test bloat #4375

Description of changes:

  1. Keep ALL_TEST_CERTS as is
  2. only test_happy_path.py will use ALL_TEST_CERTS (preserve coverage)
  3. all other tests will use MINIMAL_TEST_CERTS
  4. add ECDSA_521 to MINIMAL_TEST_CERTS to ensure coverage over all key types.

Problem:

This PR reduces the number of certificates used in integration tests, limiting them to only the minimal required set, except for test_happy_path.py, which retains all certs for comprehensive coverage. Certain tests has slightly increased because we added ECDSA_521 to MINIMAL_TEST_CERTS

Before Reduction After Reduction Change
8701 test_buffered_send.py 9781 test_buffered_send.py 🟢🔼 +1080
8840 test_happy_path.py 8840 test_happy_path.py ➖ No change
12308 test_signature_algorithms.py 2588 test_signature_algorithms.py 🔴🔽 -10720
4159 test_external_psk.py 1999 test_external_psk.py 🔴🔽 -2160
7664 test_session_resumption.py 1724 test_session_resumption.py 🔴🔽 -5940
1682 test_client_authentication.py 1682 test_client_authentication.py ➖ No change
6925 test_version_negotiation.py 1525 test_version_negotiation.py 🔴🔽 -5400
1220 test_renegotiate.py 1275 test_renegotiate.py 🟢🔼 +55
1220 test_npn.py 1275 test_npn.py 🟢🔼 +55
3964 test_early_data.py 1144 test_early_data.py 🔴🔽 -2820
3504 test_cross_compatibility.py 864 test_cross_compatibility.py 🔴🔽 -2640
576 test_ocsp.py 576 test_ocsp.py ➖ No change
1320 test_dynamic_record_sizes.py 300 test_dynamic_record_sizes.py 🔴🔽 -1020
253 test_record_padding.py 271 test_record_padding.py 🟢🔼 +18
457 test_hello_retry_requests.py 127 test_hello_retry_requests.py 🔴🔽 -330
88 test_sni_match.py 88 test_sni_match.py ➖ No change
175 test_sslyze.py 75 test_sslyze.py 🔴🔽 -100
72 test_fragmentation.py 72 test_fragmentation.py ➖ No change
28 test_renegotiate_apache.py 28 test_renegotiate_apache.py ➖ No change
15 test_pq_handshake.py 15 test_pq_handshake.py ➖ No change
8 test_serialization.py 8 test_serialization.py ➖ No change
7 test_key_update.py 7 test_key_update.py ➖ No change
1 test_well_known_endpoints.py 1 test_well_known_endpoints.py ➖ No change
1 test_sslv2_client_hello.py 1 test_sslv2_client_hello.py ➖ No change
1 63188/443728 tests collected 1 34266/290490 tests collected 🔴🔽 -28922 tests
(380540 deselected) in 21.82s (256224 deselected) in 11.89s 🔴🔽 -124316 deselected

Testing

Verified that reducing the number of certificates did not impact coverage of other test parameters (buffer sizes, protocols, ciphers). The test compared the original and modified test outputs to ensure that all non-certificate parameters remained unchanged. Attached below is the test case.

📌 Testcase

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@github-actions github-actions bot added the s2n-core team label Mar 4, 2025
@johubertj johubertj requested review from jmayclin and dougch March 4, 2025 23:21
@johubertj johubertj marked this pull request as ready for review March 4, 2025 23:33
Copy link
Contributor

@dougch dougch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat, spot checked one of the aarch64 jobs, run time reduced from 35min to 19min.

@jmayclin
Copy link
Contributor

jmayclin commented Mar 5, 2025

Could you include the diff in the collected tests in the PR description? before/after. test_happy_path.py seems oddly low ...

Copy link
Contributor

@jmayclin jmayclin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: it would be good to clarify the scope in the PR title. test(integv2): reduce parameter selection

@johubertj johubertj added this pull request to the merge queue Mar 12, 2025
@johubertj johubertj changed the title test: reduce parameter selection test(integv2): reduce parameter selection Mar 12, 2025
Merged via the queue into aws:main with commit b6eb5fc Mar 12, 2025
46 checks passed
@johubertj johubertj deleted the test/reduce-parameter-selection branch March 12, 2025 20:23
@johubertj johubertj mentioned this pull request Mar 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants