The first command invocation on Windows is extraordinarily slow

[ofek]

Description

Between 8-25s to run a help command

Note:

• bash is faster but still affected, powershell is fastest
• Rust-compiled tools like Cargo and Rustup are significantly slower than Git or Python

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 10.15
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

https://github.com/ofek/vector/actions/runs/3456312722

Is it regression?

Unknown

Expected behavior

Minimal start-up penalty

Actual behavior

Extreme start-up penalty

Repro steps

name: Test

on: [push, pull_request]

jobs:
  bash:
    name: ${{ matrix.os }} / bash / ${{ matrix.command }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: ["windows-2019", "windows-2022"]
        command: ["cargo --help", "rustup --help", "python --help", "git --help"]

    steps:
    - run: ${{ matrix.command }}
      shell: bash
    - run: ${{ matrix.command }}
      shell: bash

  pwsh:
    name: ${{ matrix.os }} / pwsh / ${{ matrix.command }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: ["windows-2019", "windows-2022"]
        command: ["cargo --help", "rustup --help", "python --help", "git --help"]

    steps:
    - run: ${{ matrix.command }}
      shell: pwsh
    - run: ${{ matrix.command }}
      shell: pwsh

  powershell:
    name: ${{ matrix.os }} / powershell / ${{ matrix.command }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: ["windows-2019", "windows-2022"]
        command: ["cargo --help", "rustup --help", "python --help", "git --help"]

    steps:
    - run: ${{ matrix.command }}
      shell: powershell
    - run: ${{ matrix.command }}
      shell: powershell

[erik-bershel]

Hello @ofek,
We'll take a look at it.

[ofek]

Any update on this?

[erik-bershel]

@ofek we are working on this issue. I will share all information after I get something interesting.

[ofek]

Do you have any theories? I'm curious 🙂

[ilia-shipitsin]

Dear @ofek , I suppose that Windows Defender might be responsible for build delay.

I created sample repo with your pipeline definition.
After I added Set-MpPreference -DisableRealtimeMonitoring $true before your build steps, seems, timings are good

[ofek]

oh wow nice! note that shell: powershell is required rather than using the default shell or else it is as slow or slower than other commands running for the first time

[ofek]

Actually it looks like the machines are already set up that way https://github.com/al-cheb/virtual-environments/blob/main/images/win/scripts/Installers/Configure-Antivirus.ps1

It would be nice if we could get an official follow-up on this issue as it is globally wasting countless minutes

[myitcv]

FYI @ofek I independently raised #7320. See #7320 (comment) in particular, which points to action/investigation over at actions/checkout#1186.

It would be nice if we could get an official follow-up on this issue as it is globally wasting countless minutes

Note my response is not an official response, just a signpost to others issues.