Backport parallel AuTest runner to 10.1.x#13150
Closed
bneradt wants to merge 9 commits into
Closed
Conversation
bneradt
force-pushed
the
backport/10.1-autest-j
branch
from
5ed98b4 to
b86fe0f
Compare
For context, this continues the work started in apache#12371. The goal is to take full advantage of the autest extension capabilities to make writing ATS autests easier. This adds the Test.ATSReplayTest autest extensions so that tests can be written through a replay.yaml file that specifies both the configuration of the test processes as well as the HTTP Proxy Verifier traffic for the test. This patch updates the header_rewrite_bundle to use this new framework. (cherry picked from commit 4b3354f)
This commit adds a new documentation page (autests.en.rst) that provides guidance for writing end-to-end tests using the AuTest framework, with a focus on the new ATSReplayTest approach that uses YAML-based replay files for traffic specification and verification. This, hopefully living document, will be helpful for devs and AI tools that could use guidance on how to write autests. (cherry picked from commit 32d37f2)
Convert various cache-*.test.py tests to use the new ATSReplayTest extension. These were generally pretty easy to convert because many of them were Proxy Verifier replay file based tests already anyway. To support this, this also adds diags.log and traffic.out log verification support to ATSReplayTest. (cherry picked from commit d74c795)
uv has garnered a lot of Python community support recently, consolidating and replacing virtualenv, Pipenv, pip, etc. In addition to consolidating a variety of tools under one interface, it is faster and, at this point, I feel is more recognizable than Pipenv. This commit transitions us from Pipenv to uv. (cherry picked from commit c6d0978)
Add autest-parallel.py, a wrapper that runs autest tests in parallel using isolated sandboxes and port ranges. The built-in autest -j flag does not work with ATS tests, hence the need for this approach. Features: - Discovers tests, partitions across workers, aggregates results - LPT (Longest Processing Time) load balancing using recorded timings - Serial test list for tests with hardcoded ports or resource contention - Live progress line with ETA, skip/fail tracking - SIGTERM/Ctrl+C handling with clean partial result collection - CMake .in template for correct build-tree paths and integration with autest.sh (-j N delegates to the parallel runner) Also fixes: - ports.py: Add AUTEST_PORT_OFFSET support with validation - Default autest sandbox to /tmp/sb_<hash> to avoid Unix socket path length issues in deep build directories - Fix test hygiene: generated gold files and temp replay files now use managed temp directories cleaned up at exit - Convert tests with hardcoded ports to use dynamic ports Co-authored-by: Brian Neradt <brian.neradt@gmail.com> (cherry picked from commit 1305a1c) (cherry picked from commit 52e9317)
The parallel runner captured full autest output per worker but only displayed failed test names in the summary. This made CI failures hard to debug compared to the sequential runner. Extract and print the per-test failure detail block (sub-step pass/fail, reasons, file paths) so the same diagnostic information is available in parallel mode. (cherry picked from commit 9a450eb) (cherry picked from commit 0be74ad)
…pache#12894) (cherry picked from commit 0ad27f1)
Parallel autest runs can fail with worker exceptions that are not tied to a specific failed test. In CI that currently leaves only the summary counts, which makes the failure hard to diagnose. Print worker diagnostics whenever a worker reports exceptions or exits non-zero without attributing the problem to failed tests. Also accept plural summary keys while parsing autest output. Co-authored-by: bneradt <bneradt@yahooinc.com> (cherry picked from commit d81de6c)
bneradt
force-pushed
the
backport/10.1-autest-j
branch
from
b86fe0f to
761692d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports the parallel AuTest runner and autest.sh -j support to 10.1.x.
This is the third PR in the backport stack and should be reviewed/merged after #13148 and #13149.
Cherry-picked/backported commits:
Backport notes:
Verification: