Feat: devcontainer pytest integration by junkataoka · Pull Request #110 · microsoft/dstoolkit-devcontainers

junkataoka · 2025-12-16T07:58:38Z

This pull request introduces a major refactor and modernization of the CI pipeline infrastructure for both Azure DevOps and GitHub Actions. The main improvements are the switch to modular, template-driven pipeline definitions, enhanced support for devcontainer-based testing, and improved code coverage reporting. The changes streamline CI setup, make it easier to add new projects, and improve compatibility across platforms (Linux and macOS). Documentation and configuration files have also been updated for clarity and consistency.

Azure DevOps Pipeline Modernization

Both .azuredevops/ado-ci-pipeline-ms-hosted.yml and .azuredevops/ado-ci-pipeline-self-hosted.yml are refactored to use a two-stage pipeline (Lint and Test), with jobs and steps defined via new templates for devcontainer testing and coverage publishing. This replaces legacy scripts and direct task definitions, making the pipeline more maintainable and extensible. [1] [2] [3]

Template-Driven CI Architecture

New reusable templates are introduced under .azuredevops/templates/, including test-devcontainer-job.yml for per-project devcontainer testing, run-devcontainer.yml for platform-aware container execution, setup-devcontainer.yml for environment setup, publish-test-results.yml for publishing test results and coverage, and merge-coverage.yml for merging and publishing code coverage reports. [1] [2] [3] [4] [5]

GitHub Actions Pipeline Improvements

.github/workflows/ci.yaml is reworked to use matrix jobs for devcontainer-based testing, artifact upload for results and coverage, and a dedicated lint job. A new job aggregates and publishes combined test results and coverage reports, improving visibility and maintainability. [1] [2]

Documentation and Configuration Updates

References to legacy scripts (e.g., ci-tests.sh) are removed from README.md and .env.example is updated for clarity, reflecting the new devcontainer and pipeline workflow. [1] [2] [3]

Platform Compatibility and Usability Enhancements

The new pipeline templates include logic for handling both Linux and macOS (Apple Silicon) agents, and ensure clean workspace setup and artifact handling for reliable CI runs. [1] [2]

Does this introduce a breaking change?

Yes
No

Author pre-publish checklist

No PII in logs or output
Made corresponding changes to the documentation
All new packages used are included in requirements.txt
Functions use type hints, and there are no type hint errors

Pull Request Type

What kind of change does this Pull Request introduce?

junkataoka · 2025-12-16T08:21:58Z

@junkataoka please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"
Contributor License Agreement

@microsoft-github-policy-service agree [company="microsoft"]

junkataoka · 2025-12-16T16:39:44Z

@microsoft-github-policy-service agree
@microsoft-github-policy-service agree company="Microsoft"

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

fujikosu · 2026-02-06T15:58:55Z

@@ -13,44 +13,109 @@ permissions:
  contents: read


I wonder if it's really worth splitting into several jobs instead of having having them as multiple steps based on job complextity we have. One thing I noticed is this does repo clone 4 times as it used to be just one for example and that's one of downsides of going with this structure

Agreed — consolidated everything back into a single build job to avoid multiple repo clones. Lint runs first as steps, followed by sequential devcontainer tests (cpu, gpu, notebooks), then publish results. Keeps the structure simple while using the new devcontainer-based testing approach.

Thanks. This simple structure looks better. Should we follow the same approach on AzDO side and remove templates and revert back to the original approach while updating test step only to keep the update simple?

Good idea — done in a7187fc. Removed all 5 templates under .azuredevops/templates/ and inlined the devcontainer test steps directly into both ado-ci-pipeline-ms-hosted.yml and ado-ci-pipeline-self-hosted.yml, mirroring the flat structure of ci.yaml. The pipelines now only differ from main by replacing the ci-tests.sh step with the equivalent devcontainer up / devcontainer exec steps per project, plus the .env setup and @devcontainers/cli install. Net change: +213 / −303 vs the previous template-based version.

A couple of notes:

I switched the coverage publish task to PublishCodeCoverageResults@2 so it can pick up coverage-*.xml via glob and merge the per-project files automatically — this avoids needing the old merge-coverage.yml (no dotnet install / reportgenerator step).

I dropped the macOS / Apple Silicon support that was added on the self-hosted side, since restoring it would have required keeping the platform-conditional Docker handling that lived in run-devcontainer.yml. The self-hosted pipeline is now Linux-only again, matching main. Happy to add macOS support back in a follow-up if you'd like.

fujikosu

Thanks for your PR! I left several comments so please check them

junkataoka · 2026-03-31T00:36:21Z

@fujikosu Thank you for the thorough review! I've addressed all the comments — really appreciate you taking the time to go through everything in detail. The feedback helped improve the PR significantly.

fujikosu · 2026-04-02T08:32:14Z

          name: pytest
-          path: |
-            **/test-results-*.xml
+          path: 'test-results-*.xml'


what's this path udpate for?

In the original ci-tests.sh flow, each project's test results landed in nested folders (e.g. src/sample_cpu_project/test-results-*.xml), so the reporter needed **/test-results-*.xml to find them. With the new devcontainers/ci approach we explicitly copy the JUnit XMLs back to the workspace root (test-results-cpu.xml, test-results-gpu.xml), so a flat glob is sufficient — and it also fixes the pre-existing typo (test-reuslts → test-results) from the original path.

fujikosu · 2026-04-02T08:32:31Z

          output: both
          format: markdown
-          filename: coverage.xml
+          filename: 'coverage-*.xml'


what's this filename update for?

Previously ci-tests.sh produced a single merged coverage.xml. With the per-devcontainer test steps we now emit one coverage file per project (coverage-cpu.xml, coverage-gpu.xml), so the glob coverage-*.xml lets CodeCoverageSummary aggregate them into one report. (irongut/CodeCoverageSummary supports glob patterns in filename.)

fujikosu · 2026-04-02T08:32:48Z

+            test-results-*.xml
+            coverage-*.xml


what are these path update for?

Same reason as the test-reporter path change: result/coverage files are now written to the workspace root (test-results-*.xml, coverage-*.xml) instead of nested project folders, so the **/ glob is no longer needed. This also fixes the pre-existing typo test-reuslts-*.xml → test-results-*.xml so the artifact actually picks them up.

fujikosu

thanks for the update! I added comments to your update

Per fujikosu's review feedback on PR #110, simplify the AzDO pipelines to mirror the flat structure of the GitHub Actions workflow. The templates added in this PR (setup-devcontainer, run-devcontainer, test-devcontainer-job, publish-test-results, merge-coverage) are removed and the devcontainer-based test steps are inlined directly into ado-ci-pipeline-ms-hosted.yml and ado-ci-pipeline-self-hosted.yml, keeping the diff vs main minimal — only the 'Run pytest in docker containers' step is replaced with the new devcontainer steps. PublishCodeCoverageResults@2 is used so that the multiple per-project coverage XML files (coverage-*.xml) are picked up via glob and merged without requiring a separate ReportGenerator stage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace ci-tests.sh with devcontainer-based CI that builds and runs pytest inside each project's actual devcontainer, ensuring CI uses the exact same environment as developers. GitHub Actions: - Use devcontainers/ci@v0.3 to build and exec inside each devcontainer - Per-project pytest runs (cpu, gpu) with junit + coverage XML - Smoke build for notebooks devcontainer Azure DevOps (ms-hosted and self-hosted): - Inline @devcontainers/cli usage (no template files) - Same per-project pytest pattern; PublishTestResults + PublishCodeCoverageResults - Self-hosted variant adds Docker prune + _work cleanup CI workarounds for non-modifiable devcontainer.json: - Pre-create .env at relative paths (./, ../, ../../) expected by devcontainer.json runArgs --env-file paths - Strip --gpus all from GPU devcontainer.json at workflow runtime since GitHub-hosted runners lack GPU device drivers (local users keep auto GPU passthrough) Other: - Add hostRequirements gpu=optional hint to GPU devcontainer.json - pyproject.toml: coverage source/relative_files and exclude_lines - Remove obsolete ci-tests.sh Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

junkataoka force-pushed the feature/devcontainer-pytest-integration branch from 44faaca to ec09cba Compare December 16, 2025 08:43

junkataoka force-pushed the feature/devcontainer-pytest-integration branch 5 times, most recently from 2523a23 to 0ebed45 Compare December 18, 2025 15:06

junkataoka changed the title ~~Feature/devcontainer pytest integration~~ feat: devcontainer pytest integration Dec 18, 2025

junkataoka changed the title ~~feat: devcontainer pytest integration~~ Feat: devcontainer pytest integration Dec 18, 2025

junkataoka force-pushed the feature/devcontainer-pytest-integration branch from d73bb8e to 82fb7b9 Compare December 19, 2025 00:10

junkataoka requested a review from Copilot December 19, 2025 00:20

Copilot started reviewing on behalf of junkataoka December 19, 2025 00:20 View session

This comment was marked as outdated.

Sign in to view

junkataoka force-pushed the feature/devcontainer-pytest-integration branch from fe5a82e to 78f787f Compare December 19, 2025 01:29

junkataoka requested a review from Copilot December 19, 2025 01:48

Copilot started reviewing on behalf of junkataoka December 19, 2025 01:48 View session

This comment was marked as outdated.

Sign in to view

junkataoka force-pushed the feature/devcontainer-pytest-integration branch 5 times, most recently from a279838 to 5f6316e Compare December 19, 2025 05:03

junkataoka requested a review from Copilot December 19, 2025 05:04

Copilot started reviewing on behalf of junkataoka December 19, 2025 05:05 View session

Copilot AI reviewed Dec 19, 2025

View reviewed changes

Comment thread pyproject.toml

Comment thread .github/workflows/ci.yaml Outdated

junkataoka marked this pull request as ready for review December 19, 2025 05:21

junkataoka requested review from bhavikm and fujikosu December 19, 2025 05:21

junkataoka force-pushed the feature/devcontainer-pytest-integration branch from 0e665e8 to 5f6316e Compare December 19, 2025 09:25