ci: Refactor pipeline, add oidc auth and integrate smoke testing automation by Vamshi-Microsoft · Pull Request #541 · microsoft/content-processing-solution-accelerator

Vamshi-Microsoft · 2026-04-14T14:14:27Z

Purpose

This pull request makes significant improvements to the GitHub Actions workflows, focusing on standardizing Azure authentication, enhancing security, and adding flexibility to deployment environments. The main changes include switching to federated identity for Azure logins, updating permissions, and allowing the selection of deployment environments (Linux or Windows) via workflow inputs.

Key changes:

Azure Authentication & Security Enhancements

Replaced direct use of Azure service principal credentials with the azure/login@v2 GitHub Action across all workflows, enabling federated credentials and improving security. [1] [2] [3] [4] [5] [6]
Added id-token: write permission to workflows, which is required for federated authentication with Azure. [1] [2] [3]

Deployment Flexibility & Inputs

Introduced a runner_os input in the deploy-linux.yml workflow, allowing users to select the deployment environment (codespace for Linux/Ubuntu or Local for Windows), and dynamically resolving the runner accordingly. [1] [2] [3] [4]

Workflow Permissions & Environment Standardization

Standardized the use of the environment: production property for jobs in deployment, build, and test workflows to improve environment management and auditing. [1] [2] [3] [4] [5] [6] [7]
Removed redundant or unnecessary explicit permissions blocks from multiple workflow files, relying on defaults or centralized permission management. [1] [2] [3] [4] [5] [6] [7]

Other Improvements

Updated Docker build workflow to authenticate with Azure using azure/login@v2 before logging in to Azure Container Registry, aligning with new authentication standards.
Removed deprecated --headed flag from pytest test commands in the test automation workflow for cleaner test runs. [1] [2] [3]

These changes collectively modernize the CI/CD pipeline, improve security by adopting federated identity, and provide greater flexibility and maintainability for deployment workflows.

Does this introduce a breaking change?

Yes
No

Golden Path Validation

I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

...

Other Information

…t paths

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Refactors GitHub Actions CI/CD to use Azure OIDC authentication and standardize environments, while adding a Playwright/pytest-based E2E smoke test suite with HTML reporting and artifacts.

Changes:

Switched multiple workflows from service-principal login to azure/login@v2 (OIDC) and standardized environment: production.
Added Playwright-based E2E tests + pytest HTML reporting (including screenshots/log capture) under tests/e2e-test.
Updated deployment pipeline inputs to dynamically choose runner OS for v2 deployments.

Reviewed changes

Copilot reviewed 22 out of 29 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
tests/e2e-test/tests/test_contentProcessing_st_tc.py	Adds Content Processing V2 E2E test cases (golden path + validations).
tests/e2e-test/tests/conftest.py	Adds Playwright session fixture + custom pytest-html logging/screenshot hooks.
tests/e2e-test/requirements.txt	Introduces Python dependencies for running Playwright/pytest E2E suite.
tests/e2e-test/readme.MD	Documents how to set up and run the E2E tests locally.
tests/e2e-test/pytest.ini	Configures pytest logging and markers for the E2E suite.
tests/e2e-test/pages/loginPage.py	Adds page object for authentication.
tests/e2e-test/pages/HomePageV2.py	Adds page object implementing Content Processing V2 UI interactions/assertions.
tests/e2e-test/config/constants.py	Loads target web URL from `.env`/environment.
tests/e2e-test/base/base.py	Adds base page object helpers.
tests/e2e-test/.gitignore	Ignores E2E outputs (reports, screenshots, venvs, etc.).
infra/scripts/checkquota.sh	Removes SP login; relies on pre-authenticated Azure CLI (OIDC).
.github/workflows/test-automation.yml	Migrates to `azure/login@v2` and sets environment.
.github/workflows/test-automation-v2.yml	Migrates to `azure/login@v2`, removes `--headed`, uploads artifacts.
.github/workflows/job-docker-build.yml	Uses OIDC login before ACR login and build/push.
.github/workflows/job-deploy.yml	Migrates Azure auth to OIDC and standardizes environment.
.github/workflows/job-deploy-windows.yml	Adds OIDC auth + azd federated login and standardizes environment.
.github/workflows/job-deploy-linux.yml	Adds OIDC auth + azd federated login and standardizes environment.
.github/workflows/job-cleanup-deployment.yml	Migrates Azure auth to OIDC and standardizes environment.
.github/workflows/deploy.yml	Adds `id-token: write`, OIDC login, and standardizes environment.
.github/workflows/deploy-windows.yml	Adds `id-token: write` for OIDC readiness.
.github/workflows/deploy-orchestrator.yml	Removes explicit permissions block (now relies on caller).
.github/workflows/deploy-linux.yml	Adds `runner_os` input + OIDC permissions and passes resolved runner to orchestrator.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-15T05:29:51Z

+@pytest.hookimpl(tryfirst=True)
+def pytest_runtest_setup(item):
+    """Prepare StringIO for capturing logs"""
+    stream = io.StringIO()
+    handler = logging.StreamHandler(stream)
+    handler.setLevel(logging.INFO)
+
+    logger = logging.getLogger()
+    logger.addHandler(handler)
+
+    # Save handler and stream
+    log_streams[item.nodeid] = (handler, stream)


The log handler is added in pytest_runtest_setup, but removed (and log_streams is popped) in pytest_runtest_makereport for the first phase that runs (typically when == 'setup'). This means the handler is no longer attached during the actual test body (when == 'call'), so call-phase logs won’t be captured in the HTML report. A concrete fix is to only remove/pop the handler on the last report phase (e.g., when == 'teardown'), or to attach/remove handlers in pytest_runtest_call/pytest_runtest_teardown so the handler spans the call phase.

Copilot · 2026-04-15T05:29:51Z

+    handler, stream = log_streams.get(item.nodeid, (None, None))
+
+    if handler and stream:
+        # Make sure logs are flushed
+        handler.flush()
+        log_output = stream.getvalue()
+
+        # Only remove the handler, don't close the stream yet
+        logger = logging.getLogger()
+        logger.removeHandler(handler)


The log handler is added in pytest_runtest_setup, but removed (and log_streams is popped) in pytest_runtest_makereport for the first phase that runs (typically when == 'setup'). This means the handler is no longer attached during the actual test body (when == 'call'), so call-phase logs won’t be captured in the HTML report. A concrete fix is to only remove/pop the handler on the last report phase (e.g., when == 'teardown'), or to attach/remove handlers in pytest_runtest_call/pytest_runtest_teardown so the handler spans the call phase.

Copilot · 2026-04-15T05:29:51Z

+        # Clean up references
+        log_streams.pop(item.nodeid, None)


The log handler is added in pytest_runtest_setup, but removed (and log_streams is popped) in pytest_runtest_makereport for the first phase that runs (typically when == 'setup'). This means the handler is no longer attached during the actual test body (when == 'call'), so call-phase logs won’t be captured in the HTML report. A concrete fix is to only remove/pop the handler on the last report phase (e.g., when == 'teardown'), or to attach/remove handlers in pytest_runtest_call/pytest_runtest_teardown so the handler spans the call phase.

Copilot · 2026-04-15T05:29:52Z

+                    # Use relative path from report.html location
+                    relative_path = os.path.relpath(
+                        screenshot_path,
+                        os.path.dirname(os.path.abspath("report.html"))
+                    )


Both the screenshot link generation and rename_duration_column() assume the report is written to ./report.html, but the workflow runs pytest with --html=report/report.html. As a result, the duration column rename will never run, and screenshot links may be computed relative to the wrong directory. Use the actual pytest-html output path (e.g., from pytest config/options) or make the report location a single constant/env var used by both the workflow command and these helpers.

Copilot · 2026-04-15T05:29:52Z

+def rename_duration_column():
+    """Rename Duration column to Execution Time in HTML report"""
+    report_path = os.path.abspath("report.html")
+    if not os.path.exists(report_path):
+        print("Report file not found, skipping column rename.")
+        return


Both the screenshot link generation and rename_duration_column() assume the report is written to ./report.html, but the workflow runs pytest with --html=report/report.html. As a result, the duration column rename will never run, and screenshot links may be computed relative to the wrong directory. Use the actual pytest-html output path (e.g., from pytest config/options) or make the report location a single constant/env var used by both the workflow command and these helpers.

Copilot · 2026-04-15T05:29:53Z

+        browser = playwright_instance.chromium.launch(
+            headless=False,
+            args=["--start-maximized"]
+        )


Launching the browser as headless=False unconditionally makes CI execution more fragile and slow (and tightly couples it to Xvfb). Make headless/headed configurable via an env var/pytest option (default to headless in CI), so local debugging can still opt into headed runs.

Copilot · 2026-04-15T05:29:54Z

+[pytest]
+log_cli = true
+log_cli_level = INFO
+log_file = logs/tests.log


Pytest’s file logging will fail if the logs/ directory doesn’t exist. This suite creates tests/screenshots/ but not logs/. Create the logs directory at startup (e.g., in conftest.py) or change log_file to a path that is guaranteed to exist.

Suggested change

log_file = logs/tests.log

log_file = tests.log

Copilot · 2026-04-15T05:29:54Z

+pytest-playwright
+pytest-reporter-html1
+python-dotenv
+pytest-check
+pytest-html
+py
+beautifulsoup4


All dependencies are unpinned, which makes CI runs non-reproducible and increases the chance of sudden breakages (especially for browser automation stacks like Playwright). Pin versions (or at least constrain major versions) and consider using a lockfile approach to keep the E2E environment stable over time.

Suggested change

pytest-playwright

pytest-reporter-html1

python-dotenv

pytest-check

pytest-html

py

beautifulsoup4

pytest-playwright==0.*

pytest-reporter-html1==1.*

python-dotenv==1.*

pytest-check==2.*

pytest-html==4.*

py==1.*

beautifulsoup4==4.*

github-actions · 2026-04-20T10:02:28Z

🎉 This PR is included in version 1.14.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Vamshi-Microsoft added 7 commits April 7, 2026 11:43

Refcator Pipeline

31c478a

rename pipeline

348e9d6

feat: Add runner_os input option for deployment environment in workflow

7b362d0

Integrated Smoke testing automation

022ff8b

fix: Remove --headed option from pytest commands and update screensho…

6a20bde

…t paths

Merge branch 'dev' into psl-smoketesting

1247ab6

Resolve pylint issue

92f9b50

Vamshi-Microsoft requested review from Avijit-Microsoft, Prajwal-Microsoft, Roopan-Microsoft, Vinay-Microsoft, aniaroramsft, dgp10801, nchandhi and toherman-msft as code owners April 14, 2026 14:14

rename deploy-linux to deploy-v2

8d0dc7e

Vamshi-Microsoft changed the title ~~ci: Refactor pipeline. Add oidc auth and integrate smoke testing automation~~ ci: Refactor pipeline, add oidc auth and integrate smoke testing automation Apr 14, 2026

Prajwal-Microsoft requested a review from Copilot April 15, 2026 05:22

Prajwal-Microsoft approved these changes Apr 15, 2026

View reviewed changes

Prajwal-Microsoft merged commit 4d78576 into dev Apr 15, 2026
5 checks passed

Copilot AI reviewed Apr 15, 2026

View reviewed changes

Copilot started reviewing on behalf of Prajwal-Microsoft April 15, 2026 06:26 View session

github-actions bot added the released label Apr 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: Refactor pipeline, add oidc auth and integrate smoke testing automation#541

ci: Refactor pipeline, add oidc auth and integrate smoke testing automation#541
Prajwal-Microsoft merged 8 commits intodevfrom
psl-sm-integration

Vamshi-Microsoft commented Apr 14, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

github-actions bot commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Vamshi-Microsoft commented Apr 14, 2026

Purpose

Does this introduce a breaking change?

Golden Path Validation

Deployment Validation

What to Check

Other Information

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants