Skip to content

Fix broken CI -- update python command for Linux Helix work items#5179

Merged
DrewScoggins merged 1 commit intodotnet:mainfrom
danmoseley:fix-python3-helix
Mar 30, 2026
Merged

Fix broken CI -- update python command for Linux Helix work items#5179
DrewScoggins merged 1 commit intodotnet:mainfrom
danmoseley:fix-python3-helix

Conversation

@danmoseley
Copy link
Copy Markdown
Member

@danmoseley danmoseley commented Mar 29, 2026

Note

This PR description was AI/Copilot-generated.

Fix python: not found on Linux Helix work items

The updated Ubuntu 22.04 Helix image (Helix-Ubuntu-2204-2026-03-23) no longer provides a python command — only python3 is available. This broke all Linux Helix work items in public CI with exit code 127.

Main branch has been red since March 26 (build 1354108 onward; last green was 1353009 on March 25).

Changes

  • get_work_item_command(): Change the non-Windows command from python to python3 (line 598). The Windows path (python, line 593) is correct as-is.
  • PerfSendToHelixArgs: Use the existing agent_python variable (python3 on Linux, py -3 on Windows) for the $(Python) env var passed to .proj files, instead of unconditionally using "python".

Internal builds were unaffected because they create a Python venv first, which provides a python symlink.

Why this is safe

$(Python) is consumed by all the scenario .proj files (pre.py, test.py, post.py, etc.). The new values (python3 on Linux, py -3 on Windows) are the same values already used by the agent itself — see line 1157 (os.environ["Python"] = agent_python) and run-performance-job-script-step.yml line 6. Python venvs create both python and python3 symlinks, so internal builds (which use a venv) are also unaffected. The hardcoded "python" was the oddball; this change makes the Helix work item value consistent with the rest of the infrastructure.

The Ubuntu 22.04 Helix image was updated and no longer provides a
'python' command (only 'python3'). This broke all Linux Helix work
items in public CI with 'python: not found' (exit code 127).

- Change the non-Windows work item command from 'python' to 'python3'
- Use the existing 'agent_python' variable ('python3' on Linux,
  'py -3' on Windows) for the Python env var passed to .proj files,
  instead of unconditionally using 'python'

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@danmoseley danmoseley requested a review from caaavik-msft March 29, 2026 20:07
@danmoseley danmoseley changed the title Fix python command for Linux Helix work items Fix broken CI -- update python command for Linux Helix work items Mar 29, 2026
@danmoseley danmoseley added pipeline blocker issue blocks perf pipeline stays green. This be considered high priority. and removed blocking-clean-ci-optional labels Mar 29, 2026
@danmoseley danmoseley requested a review from Copilot March 29, 2026 20:37
@danmoseley
Copy link
Copy Markdown
Member Author

@caaavik-msft as an aside, how about enabling copilot review on new PR's (repo settings). either on first push only, or all. but I think first push only is a definite good thing.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Helix work item Python invocation to match modern Linux images where python is no longer available, restoring Linux Helix CI reliability.

Changes:

  • Switch non-Windows Helix work item command execution from python to python3.
  • Pass the platform-correct agent_python value into PerfSendToHelixArgs.python so .proj scenario work items use the correct Python launcher via $(Python).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@danmoseley
Copy link
Copy Markdown
Member Author

Note

This comment was AI/Copilot-generated.

Why is Windows OK with just python?
Windows Helix images install Python via the windows-python-3 artifact, which puts python.exe on PATH. On Windows, python has always meant Python 3 (there's no Python 2 legacy issue). Ubuntu dropped the unversioned python symlink years ago — it only ships python3.

What is agent_python?
Defined at lines 1015-1018 of run_performance_job.py:

if args.os_group == "windows":
    agent_python = "py -3"
else:
    agent_python = "python3"

py -3 is the Windows Python Launcher — it finds the installed Python 3 automatically. It was originally meant for commands running on the AzDO agent machine (e.g., line 1157: os.environ["Python"] = agent_python), but the same values are correct for Helix machines too, since both Windows Helix images and AzDO agents have the Python launcher installed.

@danmoseley
Copy link
Copy Markdown
Member Author

CI passed!

@DrewScoggins
Copy link
Copy Markdown
Member

Going to do a quick test to make sure that all of our internal machines have the python3 alias. Once that looks good we can get this merged.

Copy link
Copy Markdown
Member

@LoopedBard3 LoopedBard3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me once we have a regular test run .👍

@DrewScoggins DrewScoggins merged commit 5f6f7e9 into dotnet:main Mar 30, 2026
78 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pipeline blocker issue blocks perf pipeline stays green. This be considered high priority.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants