Split long-running integ tests to stay under 10 min#8887
Merged
reedham-aws merged 14 commits intodevelopfrom Apr 7, 2026
Merged
Split long-running integ tests to stay under 10 min#8887reedham-aws merged 14 commits intodevelopfrom
reedham-aws merged 14 commits intodevelopfrom
Conversation
- Split integ-buildcmd-main: extract python tests (19 tests, 42% of load) into integ-buildcmd-python - Split integ-all-other: extract local tests (413 tests, 82% of load) into integ-local
- Split integ-buildcmd-arm64 into python-node and other groups - Split integ-buildcmd-dotnet out of integ-buildcmd-main - Separate integ-telemetry to run without parallelism (fixes port 18298 conflict) - Split Windows smoke tests from 3 groups (thirds) to 4 groups (quarters)
With -n 2, two workers run sam init simultaneously, both cloning templates to the same shared directory (~/.sam/), causing conflicts on ubuntu. The group only has 89 tests and finishes in ~4 min, so parallelism isn't needed.
…r integ-all-other - Move Ruby into arm64-python-node for better balance (11 vs 10) - Split node out of integ-buildcmd-main (was 10 min on Windows) - Isolate integ-init (sequential, avoids clone conflict) - Restore -n 2 for integ-all-other (pipeline+root+scripts+docs)
…test changes Per reviewer feedback, reverted most integration test splits to reduce total job count. Kept only: - Split integ-buildcmd-python out of integ-buildcmd-main (17 min on Windows) - Smoke tests: ubuntu 1 group → 2 halves, windows 3 thirds → 4 quarters
Add @pytest.mark.pr_skip to skip tests in build.yml that are still run in integration-tests.yml (nightly/merge): - EsbuildFunctions_With_External_Manifest (6 tests): redundant with EsbuildFunctions (same esbuild logic) and NodeFunctions_With_External_Manifest (same external manifest resolution) - EsbuildFunctionProperties (12 tests): already skipped on Windows CI; sourcemap/node options is a variation of the same esbuild build path covered by EsbuildFunctions - PythonFunctions_With_Specified_Architecture (12 tests): architecture flag is a thin wrapper; same runtimes and build logic already covered by WithoutDocker_al2023 - TestDocsCommand (25 tests): static URL mapping validation that rarely changes and is low-risk for PR validation
- Add pr_skip to EsbuildFunctions_With_External_Manifest_arm64 (3 tests): redundant with EsbuildFunctions_arm64 and NodeFunctions_arm64 - Revert parallelism from n=3 back to n=2 for arm64, buildcmd-main, and all-other (n=3 didn't help, 2 cores on runners)
- Rust_arm64 (16 tests): cargo-lambda builder already covered by non-arm64 test_build_cmd_rust.py; arm64 cross-compilation is a cargo flag, not SAM CLI logic - TestInitForParametersCompatibility (7 tests): parameter validation edge cases well covered by unit tests - TestCopyTerraformBuiltArtifacts (11 tests): Terraform-specific script tests, low risk for PR validation
Java tests only need Python + Java. Skip Go, Ruby, Node, Dotnet, and Rust setup steps to save ~3 min of setup overhead on each Java test runner.
With -n 2 and skipped setup steps, two parallel Gradle builds start faster and collide on the shared gradle-cache lock file on Windows. Running sequentially avoids the lock conflict. Combined with the setup skip saving ~3 min, total time should stay around 10 min.
reedham-aws
approved these changes
Apr 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Optimize PR CI workflow (
build.yml) so all jobs complete within 10 minutes. Changes include splitting one slow test group, splitting smoke tests into smaller groups, skipping redundant tests using a newpr_skipmarker, and increasing parallelism for Java tests. All skipped tests continue to run in the nightlyintegration-tests.ymlworkflow.Files changed
.github/workflows/build.ymltests/smoke/test_all_commands.pytests/integration/buildcmd/test_build_cmd_node.pytests/integration/buildcmd/test_build_cmd_python.pytests/integration/buildcmd/test_build_cmd_arm64.pytests/integration/buildcmd/test_build_cmd_java.pytests/integration/init/test_init_command.pytests/integration/scripts/test_copy_terraform_built_artifacts.pytests/integration/docs/test_docs_command.pypytest.iniIntegration test split
Split
integ-buildcmd-pythonout ofinteg-buildcmd-mainto address the slowest job (17m 42s on Windows):integ-buildcmd-main(python+dotnet+provided+rust+node, 17m 42s on Windows)integ-buildcmd-python(python only)integ-buildcmd-main(dotnet+provided+rust+node)Smoke test split
Split smoke tests into smaller groups to reduce per-job runtime:
Smoke & Functional Tests - All(ubuntu, all templates)Smoke Tests - First Half(ubuntu, 50%)Smoke Tests - Second Half & Functional Tests(ubuntu, 50% + functional)Smoke Tests - First Third(windows)Smoke Tests - First Quarter(windows)Smoke Tests - Second Third(windows)Smoke Tests - Second Quarter(windows)Smoke Tests - Third Third & Functional Tests(windows)Smoke Tests - Third Quarter(windows)Smoke Tests - Fourth Quarter & Functional Tests(windows)Java parallelism
Increased
integ-buildcmd-javafrom-n 2to-n 3to reduce Windows runtime from ~10m to ~7m. Java build tests are I/O-bound (invoking gradle/maven), so the third worker helps while the others wait on build tool I/O.Tests skipped in PR workflow (
pr_skipmarker)Introduced a new
@pytest.mark.pr_skipmarker for tests that are redundant or low-risk for PR validation. These tests are skipped only inbuild.ymland continue to run in the nightlyintegration-tests.ymlworkflow.test_build_cmd_node.pyEsbuildFunctions_With_External_ManifestEsbuildFunctions; external manifest resolution covered byNodeFunctions_With_External_Manifesttest_build_cmd_node.pyEsbuildFunctionPropertiesEsbuildFunctionstest_build_cmd_python.pyPythonFunctions_With_Specified_ArchitectureWithoutDocker_al2023test_build_cmd_arm64.pyEsbuildFunctions_With_External_Manifest_arm64test_build_cmd_arm64.pyRust_arm64cargo-lambdabuilder already covered by non-arm64test_build_cmd_rust.py; arm64 cross-compilation is a cargo flag, not SAM CLI logictest_init_command.pyTestInitForParametersCompatibilitytest_copy_terraform_built_artifacts.pyTestCopyTerraformBuiltArtifactstest_docs_command.pyTestDocsCommandsam docs <command>→ URL); these mappings rarely changeTotal: 92 tests skipped in PR builds.
Results
Before (develop branch, run #24024561824):
integ-buildcmd-mainwin 3.10integ-buildcmd-javawin 3.11integ-all-otherwin 3.10integ-buildcmd-arm64ubuntu 3.11Smoke & Functional Tests - AllubuntuSmoke Tests - Second Thirdwin 3.10After (run #24056335724):
integ-buildcmd-mainubuntu 3.10integ-buildcmd-javawin 3.11integ-all-otherwin 3.10integ-buildcmd-arm64ubuntu 3.10Smoke Tests - Fourth Quarter & Functional Testswin 3.11All jobs now complete within 10 minutes.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.