Python no longer found on windows-2022 in 20240714.1.0 image release

[akoeplinger]

Description

With the latest image release I'm seeing builds break that rely on python being in the PATH.

  'python' is not recognized as an internal or external command,
  operable program or batch file.
D:\a\_work\1\s\cpython.proj(214,5): error MSB3073: The command "python D:\a\_work\1\s\PC\layout -vv -b D:\a\_work\1\s\artifacts\obj\Windows_NT.x64.Release\cpython\\amd64 --arch amd64 --catalog D:\a\_work\1\s\artifacts\obj\Windows_NT.x64.Release\cpython\\python.cdf --include-tools --include-dev --include-stable --include-symbols" exited with code 9009.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

20240714.1.0
https://dev.azure.com/dnceng-public/public/_build/results?buildId=750103&view=results

Is it regression?

Yes, it was working in 20240707.1.0

Expected behavior

Python is found in PATH

Actual behavior

Python is not found in PATH

Repro steps

Try calling python from a batch script.

[Prabhatkumar59]

Hi @akoeplinger - Thank you for bringing this issue to us. We are looking into this issue and will update you shortly on this issue after investigating.

[xchen218]

Hi @Prabhatkumar59 , we are experiencing similar issue in our runs after 20240714.1.0. How could we prevent these things from happening without any regression? Could we target image version in the YAML? For example, adding another optional parameter say "LKG" or something in the same level with pool name and os to use latest known good version to keep consistency?

[BrianMouncer]

Hi @akoeplinger - Thank you for bringing this issue to us. We are looking into this issue and will update you shortly on this issue after investigating.

Same issue here. I compared the logged agent runner image version between passing and failing builds, and it did not change, so what ever casued this is likely some tool install after then agent image first starts.

I was able to work around the issue be just explicitly selecting a version of Python, and having the task add it back to the system path, rather than depending on the version the agent previously had set in the system path be default.

  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.10'
      addToPath: true
    displayName: "make Python 3.10 the default."

[gangzhang-ms]

We also met the same issue from last Friday or Saturday and the walkaround @BrianMouncer mentioned can works well.

[Prabhatkumar59]

Hi @xchen218 @BrianMouncer @gangzhang-ms we are checking on this and will update you shortly, thanks:)

[akoeplinger]

I just reran the build that was failing and it started working again even though it has the same image version.

[Prabhatkumar59]

Hi @akoeplinger - Thank you for the update. I’m glad to hear that the build is working now.

To ensure smooth operations, you might also consider the following steps:-

Monitor for Recurrence: You can keep an eye on the build process to see if the issue reoccurs. If it does, please let us know so we can investigate further.

Review Logs: If you experience any further issues, reviewing the build logs might help pinpoint any recurring patterns or anomalies.

Retry Mechanism: Implementing retry logic for critical build steps can sometimes help mitigate temporary failures.

If you encounter any more issues or have any questions, feel free to reach out.
Hope, your issue will be resolved, so we can close this story...thanks:)

[tilumi]

@Prabhatkumar59 can you just revert everything & release reverted version asap? it has blocking us for 2 days

[xchen218]

Sadly we were not as lucky as the author. The issue still persists. @Prabhatkumar59 . Could you please provide an ETA to the fix?

[xchen218]

Gently ping on the issue. Are there any updates? @Prabhatkumar59

[Prabhatkumar59]

@xchen218 - Thanks for your response:)
As I can see in the above comment, @akoeplinger has already reran the build in the same image version and it's working for him.
So, could you also please try to rerun the build in the same image version. I am also checking into this issue and will update.