infra: Improve pr pipeline speed#522
Conversation
| ./scripts/test-pipeline tester -q $(BRANCH_FLAG) | ||
| ./scripts/test-pipeline scale-official -q $(BRANCH_FLAG) | ||
| ./scripts/test-pipeline full-validation -q $(BRANCH_FLAG) | ||
| ./scripts/test-pipeline --parallel -q $(BRANCH_FLAG) \ |
There was a problem hiding this comment.
unfortunately the az-cli version in pipelines has a race condition that prevents the use fo --parallel :(
There was a problem hiding this comment.
Pull request overview
This PR improves pipeline speed by enabling parallel execution of pipeline validation checks and reorganizing Azure DevOps pipeline jobs to run concurrently.
Changes:
- Modified
scripts/test-pipelineto accept multiple pipeline names and support a--parallelflag for concurrent execution - Updated
Makefileto call the script once with all pipeline names instead of sequential invocations - Restructured Azure DevOps pipeline jobs into separate parallel jobs (Make_Pipelines, Check_Licenses, Make)
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| scripts/test-pipeline | Added multi-pipeline support with nargs="+" and parallel execution via ThreadPoolExecutor |
| Makefile | Consolidated 11 sequential pipeline checks into single parallelized invocation |
| .pipelines/templates/stages/validate_makefile/dev-build.yml | Split single Make job into three parallel jobs for pipeline validation, license checking, and build tasks |
|
|
||
| with concurrent.futures.ThreadPoolExecutor() as executor: | ||
| futures = { | ||
| executor.submit(check_pipeline, name, selected_branch): name |
There was a problem hiding this comment.
The check_pipeline function is called without the quiet parameter in parallel mode. This means output will be printed even when --parallel is used with --quiet, which contradicts the requirement enforced at line 82-83. Pass args.quiet as the third argument to check_pipeline.
| executor.submit(check_pipeline, name, selected_branch): name | |
| executor.submit(check_pipeline, name, selected_branch, args.quiet): name |
| args = parser.parse_args() | ||
|
|
||
| if args.parallel and not args.quiet: | ||
| parser.error("--parallel requires --quiet (-q)") |
There was a problem hiding this comment.
The error message could be more descriptive about why --parallel requires --quiet. Consider expanding it to explain that parallel execution may interleave output, making it unreadable without the quiet flag.
| parser.error("--parallel requires --quiet (-q)") | |
| parser.error( | |
| "--parallel requires --quiet (-q) because parallel execution may interleave " | |
| "output from multiple pipelines, making it unreadable without quiet mode." | |
| ) |
🔍 Description
Parallelize a couple things where possible