[WIP] Add the Helix Job Monitor by premun · Pull Request #127749 · dotnet/runtime

premun · 2026-05-04T13:35:18Z

dotnet-policy-service · 2026-05-04T13:36:49Z

Tagging subscribers to this area: @akoeplinger, @matouskozak, @simonrozsival
See info in area-owners.md if you want to be subscribed.

Copilot

Pull request overview

This PR introduces a new Azure Pipelines “Helix Job Monitor” stage/job template under eng/common/core-templates and wires it into the runtime pipeline so a dedicated job can monitor Helix work across stages.

Changes:

Adds a new stage template (helix-job-monitor.yml) that wraps a job template to run the monitor tool.
Adds a new job template (helix-job-monitor.yml) intended to acquire/install the monitor tool and run it with pipeline/Helix context.
Integrates the stage into eng/pipelines/runtime.yml with monitorAllStages: true.

Reviewed changes

Copilot reviewed 1 out of 3 changed files in this pull request and generated 9 comments.

File	Description
eng/pipelines/runtime.yml	Adds the new Helix Job Monitor stage template to the runtime pipeline.
eng/common/core-templates/stages/helix-job-monitor.yml	New stage wrapper for the Helix Job Monitor job template.
eng/common/core-templates/job/helix-job-monitor.yml	New job template that downloads/installs and runs the Helix Job Monitor tool.

+      echo "Tool DLL: $toolDll"
+      echo "##vso[task.setvariable variable=HelixJobMonitorDll]$toolDll"
+    displayName: Install Helix Job Monitor
+
+  - bash: |
+      set -euo pipefail
+


+      # Tool was installed from a local nupkg; run the DLL via the repo-local dotnet.
+      export DOTNET_ROOT="$(Build.SourcesDirectory)/.dotnet"
+      ./eng/common/dotnet.sh exec "$(HelixJobMonitorDll)" "${toolArgs[@]}"
+    displayName: Monitor Helix Jobs
+    env:
+      SYSTEM_ACCESSTOKEN: $(System.AccessToken)


+# Optional explicit tool version. Only honored when 'toolNupkgArtifactName' is set; in the
+# default code path the version is taken from the consuming repo's .config/dotnet-tools.json.
+- name: toolVersion
+  type: string
+  default: '11.0.0-ci'


+# Pool override. When empty the template selects a default azurelinux pool based on the team project.
+- name: pool
+  type: object
+  default: {}
+


+  - task: DownloadPipelineArtifact@2
+    displayName: Download Helix Job Monitor artifact
+    inputs:
+      buildType: specific
+      project: public
+      pipeline: arcade-pr
+      runId: 1407423
+      artifactName: Artifacts_Windows_NT_Release
+      itemPattern: '${{ parameters.toolNupkgArtifactSubPath }}/${{ parameters.toolPackageId }}.*.nupkg'
+      targetPath: $(Agent.TempDirectory)/helix-job-monitor-nupkg


+# NuGet package id of the Helix job monitor tool.
+- name: toolPackageId
+  type: string
+  default: Microsoft.DotNet.Helix.JobMonitor
+
+# Console command exposed by the installed tool package.
+- name: toolCommand
+  type: string
+  default: dotnet-helix-job-monitor
+
+# Optional explicit tool version. When empty, the latest available version is installed.
+- name: toolVersion
+  type: string
+  default: ''
+
+# Optional NuGet feed used as an additional source when installing the tool.
+- name: toolSource
+  type: string
+  default: ''
+
+# JobMonitorOptions: --helix-base-uri.
+- name: helixBaseUri
+  type: string
+  default: https://helix.dot.net/
+
+# Helix API access token forwarded via the HELIX_ACCESSTOKEN environment variable.
+- name: helixAccessToken
+  type: string
+  default: ''
+
+# JobMonitorOptions: --polling-interval-seconds.
+- name: pollingIntervalSeconds
+  type: number
+  default: 30


+
+stages:
+- stage: ${{ parameters.stageName }}
+  dependsOn: ${{ parameters.dependsOn }}


+# Optional NuGet feed used as an additional source when installing the tool. Only honored
+# when 'toolNupkgArtifactName' is set; in the default code path the tool is restored from
+# the consuming repo's .config/dotnet-tools.json manifest and no extra feeds are consulted.
+- name: toolSource
+  type: string
+  default: ''
+


+  - bash: |
+      set -euo pipefail
+
+      toolArgs=(


Copilot

Pull request overview

Copilot reviewed 4 out of 6 changed files in this pull request and generated 7 comments.

                eq(variables['isRollingBuild'], true))
+
+    - template: /eng/common/core-templates/stages/helix-job-monitor.yml
+      parameters:
+        monitorAllStages: true


+  - template: /eng/common/core-templates/job/helix-job-monitor.yml
+    parameters:
+      jobName: ${{ parameters.jobName }}
+      helixBaseUri: ${{ parameters.helixBaseUri }}
+      helixAccessToken: ${{ parameters.helixAccessToken }}
+      timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
+      organization: ${{ parameters.organization }}
+      repository: ${{ parameters.repository }}
+      prNumber: ${{ parameters.prNumber }}
+      monitorAllStages: ${{ parameters.monitorAllStages }}


+parameters:
+# Stage identifier.
+- name: stageName
+  type: string
+  default: Helix_Job_Monitor
+
+# Optional list of stages this stage depends on.
+- name: dependsOn
+  type: object
+  default: []
+
+# Optional stage condition expression.
+- name: condition
+  type: string
+  default: ''
+
+# Job identifier produced inside the stage.
+- name: jobName
+  type: string
+  default: HelixJobMonitor
+
+# NuGet package id of the Helix job monitor tool.
+- name: toolPackageId
+  type: string
+  default: Microsoft.DotNet.Helix.JobMonitor
+
+# Console command exposed by the installed tool package.
+- name: toolCommand
+  type: string
+  default: dotnet-helix-job-monitor
+
+# Optional explicit tool version. When empty, the latest available version is installed.
+- name: toolVersion
+  type: string
+  default: ''
+
+# Optional NuGet feed used as an additional source when installing the tool.
+- name: toolSource
+  type: string
+  default: ''
+
+# JobMonitorOptions: --helix-base-uri.
+- name: helixBaseUri
+  type: string
+  default: https://helix.dot.net/
+
+# Helix API access token forwarded via the HELIX_ACCESSTOKEN environment variable.
+- name: helixAccessToken
+  type: string
+  default: ''
+
+# JobMonitorOptions: --polling-interval-seconds.
+- name: pollingIntervalSeconds
+  type: number
+  default: 30
+
+# JobMonitorOptions: --max-wait-minutes. Also used as the job/stage timeout.
+- name: timeoutInMinutes
+  type: number
+  default: 360
+
+# JobMonitorOptions: --organization (owner segment of the source repository).
+- name: organization
+  type: string
+  default: ''
+
+# JobMonitorOptions: --repository (name of the source repository).
+- name: repository
+  type: string
+  default: ''
+
+# JobMonitorOptions: --pr-number. Required for PR validation pipelines.
+- name: prNumber
+  type: string
+  default: ''
+
+# When true, the monitor tracks Helix jobs and pipeline jobs across every stage of the
+# build. When false, the monitor only tracks jobs that belong to the same Azure DevOps stage as
+# the monitor job itself (i.e. this stage).
+- name: monitorAllStages
+  type: boolean
+  default: false
+
+stages:
+- stage: ${{ parameters.stageName }}
+  dependsOn: ${{ parameters.dependsOn }}
+  ${{ if ne(parameters.condition, '') }}:
+    condition: ${{ parameters.condition }}
+  jobs:
+  - template: /eng/common/core-templates/job/helix-job-monitor.yml
+    parameters:
+      jobName: ${{ parameters.jobName }}
+      helixBaseUri: ${{ parameters.helixBaseUri }}
+      helixAccessToken: ${{ parameters.helixAccessToken }}
+      timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
+      organization: ${{ parameters.organization }}
+      repository: ${{ parameters.repository }}
+      prNumber: ${{ parameters.prNumber }}
+      monitorAllStages: ${{ parameters.monitorAllStages }}


+# Pool override. When empty the template selects a default azurelinux pool based on the team project.
+- name: pool
+  type: object
+  default: {}
+


+# Optional explicit tool version. Only honored when 'toolNupkgArtifactName' is set; in the
+# default code path the version is taken from the consuming repo's .config/dotnet-tools.json.
+- name: toolVersion
+  type: string
+  default: '11.0.0-ci'
+
+# Optional NuGet feed used as an additional source when installing the tool. Only honored
+# when 'toolNupkgArtifactName' is set; in the default code path the tool is restored from
+# the consuming repo's .config/dotnet-tools.json manifest and no extra feeds are consulted.


+  - task: DownloadPipelineArtifact@2
+    displayName: Download Helix Job Monitor artifact
+    inputs:
+      buildType: specific
+      project: public
+      pipeline: arcade-pr
+      runId: 1407423
+      artifactName: Artifacts_Windows_NT_Release
+      itemPattern: '${{ parameters.toolNupkgArtifactSubPath }}/${{ parameters.toolPackageId }}.*.nupkg'
+      targetPath: $(Agent.TempDirectory)/helix-job-monitor-nupkg


+      pushd "$(Build.SourcesDirectory)" > /dev/null
+
+      # Update .NET SDK version in global.json
+      if [ -f "global.json" ]; then
+        sed -i 's/"11.0.100-preview.3.26170.106"/"11.0.0-preview.4.26210.111"/g' global.json
+      fi
+
+      ./eng/common/dotnet.sh tool install \


Copilot

Pull request overview

Copilot reviewed 7 out of 11 changed files in this pull request and generated 6 comments.

@@ -14,7 +14,7 @@
  },
  "msbuild-sdks": {
    "Microsoft.DotNet.Arcade.Sdk": "11.0.0-beta.26211.102",


    <add key="dotnet11-transport" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11-transport/nuget/v3/index.json" />
    <add key="dotnet-diagnostics-tests" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-diagnostics-tests/nuget/v3/index.json" />
+
+    <!-- TODO: Remove -->


+parameters:
+# Azure DevOps job identifier.
+- name: jobName
+  type: string
+  default: HelixJobMonitor
+


+# Pool override. When empty the template selects a default azurelinux pool based on the team project.
+- name: pool
+  type: object
+  default: {}
+


+# Optional NuGet feed used as an additional source when installing the tool. Only honored
+# when 'toolNupkgArtifactName' is set; in the default code path the tool is restored from
+# the consuming repo's .config/dotnet-tools.json manifest and no extra feeds are consulted.
+- name: toolSource
+  type: string
+  default: ''
+


+# Console command exposed by the installed tool package.
+- name: toolCommand
+  type: string
+  default: dotnet-helix-job-monitor
+
+# Optional explicit tool version. When empty, the latest available version is installed.
+- name: toolVersion
+  type: string
+  default: ''
+
+# Optional NuGet feed used as an additional source when installing the tool.
+- name: toolSource
+  type: string
+  default: ''
+


Copilot

Pull request overview

Copilot reviewed 7 out of 11 changed files in this pull request and generated 3 comments.

  "msbuild-sdks": {
    "Microsoft.DotNet.Arcade.Sdk": "11.0.0-beta.26211.102",
-    "Microsoft.DotNet.Helix.Sdk": "11.0.0-beta.26211.102",
+    "Microsoft.DotNet.Helix.Sdk": "11.0.0-beta.26254.2",
    "Microsoft.DotNet.SharedFramework.Sdk": "11.0.0-beta.26211.102",


+# Pool override. When empty the template selects a default azurelinux pool based on the team project.
+- name: pool
+  type: object
+  default: {}
+


    <add key="dotnet11-transport" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11-transport/nuget/v3/index.json" />
    <add key="dotnet-diagnostics-tests" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-diagnostics-tests/nuget/v3/index.json" />
+
+    <!-- TODO: Remove -->


Copilot

Pull request overview

Copilot reviewed 7 out of 61 changed files in this pull request and generated 7 comments.

    <add key="dotnet11" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11/nuget/v3/index.json" />
    <add key="dotnet11-transport" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11-transport/nuget/v3/index.json" />
    <add key="dotnet-diagnostics-tests" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-diagnostics-tests/nuget/v3/index.json" />
+    <!-- TODO: Remove -->


+  pool:
+    ${{ if eq(variables['System.TeamProject'], 'public') }}:
+      name: $(DncEngPublicBuildPool)
+      demands: ImageOverride -equals build.azurelinux.3.amd64.open
+    ${{ else }}:
+      name: $(DncEngInternalBuildPool)
+      demands: ImageOverride -equals build.azurelinux.3.amd64


+    - bash: |
+        globalJsonPath="$(Build.SourcesDirectory)/global.json"
+        # Update existing global.json to set sdk.version and tools.dotnet
+        cat "$globalJsonPath" | jq '.sdk.version = "11.0.100-preview.3.26170.106" | .tools.dotnet = "11.0.100-preview.3.26170.106"' > "$globalJsonPath.tmp" && mv "$globalJsonPath.tmp" "$globalJsonPath"
+      displayName: Prepare global.json
+
+    - bash: ./eng/common/dotnet.sh tool restore
+      displayName: Restore Helix Job Monitor


 # Returns a full path to an Arcade SDK task project file.
 function GetSdkTaskProject {
-  local taskName=$1
-  local toolsetDir
-  toolsetDir="$(dirname "$_InitializeToolset")"
-  local proj="$toolsetDir/$taskName.proj"
-  if [[ -a "$proj" ]]; then
-    echo "$proj"
-    return
-  fi
-  # TODO: Remove this fallback once all supported versions use the new layout.
-  local legacyProj="$toolsetDir/SdkTasks/$taskName.proj"
-  if [[ -a "$legacyProj" ]]; then
-    echo "$legacyProj"
-    return
-  fi
-  Write-PipelineTelemetryError -category 'Build' "Unable to find $taskName.proj in toolset at: $toolsetDir"
-  ExitWithExitCode 3
+  taskName=$1
+  echo "$(dirname $_InitializeToolset)/SdkTasks/$taskName.proj"


+      - ${{ if eq(parameters.publishingInfraVersion, 4) }}:
+        - task: DownloadPipelineArtifact@2
+          displayName: Download Pipeline Artifacts (V4)
+          inputs:


+      - ${{ if eq(parameters.publishingInfraVersion, 4) }}:
+        - task: DownloadPipelineArtifact@2
+          displayName: Download Pipeline Artifacts (V4)
+          inputs:


          arguments: >
              -BuildId $(BARBuildId)
-              -PublishingInfraVersion ${{ parameters.publishingInfraVersion }}
+              -PublishingInfraVersion 3


Copilot

Pull request overview

Copilot reviewed 7 out of 61 changed files in this pull request and generated 11 comments.

    <add key="dotnet11" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11/nuget/v3/index.json" />
    <add key="dotnet11-transport" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11-transport/nuget/v3/index.json" />
    <add key="dotnet-diagnostics-tests" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-diagnostics-tests/nuget/v3/index.json" />
+    <!-- TODO: Remove -->


+          /p:TestScope=${{ parameters.testScope }}
+          /p:TestRunNamePrefixSuffix=${{ parameters.testRunNamePrefixSuffix }}
+          /p:HelixBuild=$(Build.BuildNumber)
+          /p:EnableHelixJobMonitor=${{ parameters.useHelixMonitor }}
+          ${{ parameters.extraHelixArguments }}


 # Returns a full path to an Arcade SDK task project file.
 function GetSdkTaskProject {
-  local taskName=$1
-  local toolsetDir
-  toolsetDir="$(dirname "$_InitializeToolset")"
-  local proj="$toolsetDir/$taskName.proj"
-  if [[ -a "$proj" ]]; then
-    echo "$proj"
-    return
-  fi
-  # TODO: Remove this fallback once all supported versions use the new layout.
-  local legacyProj="$toolsetDir/SdkTasks/$taskName.proj"
-  if [[ -a "$legacyProj" ]]; then
-    echo "$legacyProj"
-    return
-  fi
-  Write-PipelineTelemetryError -category 'Build' "Unable to find $taskName.proj in toolset at: $toolsetDir"
-  ExitWithExitCode 3
+  taskName=$1
+  echo "$(dirname $_InitializeToolset)/SdkTasks/$taskName.proj"
 }


 # Returns a full path to an Arcade SDK task project file.
 function GetSdkTaskProject([string]$taskName) {
-  $toolsetDir = Split-Path (InitializeToolset) -Parent
-  $proj = Join-Path $toolsetDir "$taskName.proj"
-  if (Test-Path $proj) {
-    return $proj
-  }
-  # TODO: Remove this fallback once all supported versions use the new layout.
-  $legacyProj = Join-Path $toolsetDir "SdkTasks\$taskName.proj"
-  if (Test-Path $legacyProj) {
-    return $legacyProj
-  }
-  throw "Unable to find $taskName.proj in toolset at: $toolsetDir"
+  return Join-Path (Split-Path (InitializeToolset) -Parent) "SdkTasks\$taskName.proj"
 }


          arguments: >
              -BuildId $(BARBuildId)
-              -PublishingInfraVersion ${{ parameters.publishingInfraVersion }}
+              -PublishingInfraVersion 3


+# Optional NuGet feed used as an additional source when installing the tool. Only honored
+# when 'toolNupkgArtifactName' is set; in the default code path the tool is restored from
+# the consuming repo's .config/dotnet-tools.json manifest and no extra feeds are consulted.
+- name: toolSource
+  type: string
+  default: ''
+


+    - bash: |
+        globalJsonPath="$(Build.SourcesDirectory)/global.json"
+        # Update existing global.json to set sdk.version and tools.dotnet
+        sed -i 's/"sdk":{[^}]*}/"sdk":{"version":"11.0.100-preview.3.26170.106"}/g' "$globalJsonPath"
+        sed -i 's/"tools":{[^}]*}/"tools":{"dotnet":"11.0.100-preview.3.26170.106"}/g' "$globalJsonPath"
+      displayName: Prepare global.json
+


+  if [[ -a "$toolset_location_file" ]]; then
+    local path=`cat "$toolset_location_file"`
+    if [[ -a "$path" ]]; then
+      # return value
+      _InitializeToolset="$path"


+  local toolset_build_proj=`cat "$toolset_location_file"`

-  if [[ -a "$toolset_tools_dir/Build.proj" ]]; then
-    toolset_build_proj="$toolset_tools_dir/Build.proj"
-  else
-    Write-PipelineTelemetryError -category 'Build' "Unable to find Build.proj in toolset at: $toolset_tools_dir"
+  if [[ ! -a "$toolset_build_proj" ]]; then
+    Write-PipelineTelemetryError -category 'Build' "Invalid toolset path: $toolset_build_proj"
    ExitWithExitCode 3


+              targetPath: '$(Build.ArtifactStagingDirectory)/artifacts'
              artifactName: ${{ coalesce(parameters.artifacts.publish.artifacts.name , 'Artifacts_$(Agent.Os)_$(_BuildConfig)') }}
              continueOnError: true
-              condition: always()
-              retryCountOnTaskFailure: 10 # for any logs being locked
+              condition: succeeded()
+              retryCountOnTaskFailure: 10 # for any files being locked


Copilot AI review requested due to automatic review settings May 4, 2026 13:35

premun added NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it labels May 4, 2026

Copilot started reviewing on behalf of premun May 4, 2026 13:35 View session

github-actions Bot added the area-Infrastructure-mono label May 4, 2026

dotnet-policy-service Bot assigned premun May 4, 2026

Copilot AI reviewed May 4, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings May 4, 2026 14:16

Copilot started reviewing on behalf of premun May 4, 2026 14:16 View session

Copilot AI reviewed May 4, 2026

View reviewed changes

premun force-pushed the prvysoky/helix-job-monitor branch from de93db3 to 63f15a7 Compare May 4, 2026 14:35

Flow a test Arcade build

ebc6b06

Copilot AI review requested due to automatic review settings May 5, 2026 09:16

premun force-pushed the prvysoky/helix-job-monitor branch from 63f15a7 to b11ee5f Compare May 5, 2026 09:16

Copilot started reviewing on behalf of premun May 5, 2026 09:16 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

Turn on the Helix Monitor Job for the runtime pipeline

6e31c0d

premun force-pushed the prvysoky/helix-job-monitor branch from b11ee5f to 6e31c0d Compare May 5, 2026 09:40

Revert extra changes

7efdcec

Copilot AI review requested due to automatic review settings May 5, 2026 09:41

Copilot started reviewing on behalf of premun May 5, 2026 09:42 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

premun added 3 commits May 5, 2026 12:42

Update global.json with the preview 3 before running the tool

bd8ee71

Flow another Arcade build

16d7bd7

Update global.json with the preview 3 before running the tool

03df6e5

Copilot AI review requested due to automatic review settings May 5, 2026 12:09

Copilot started reviewing on behalf of premun May 5, 2026 12:10 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

premun added 2 commits May 5, 2026 14:29

Remove jq

a7f78dc

Use a compatible version of the tool

4dee09e

Copilot AI review requested due to automatic review settings May 5, 2026 13:00

Copilot started reviewing on behalf of premun May 5, 2026 13:00 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add the Helix Job Monitor#127749

[WIP] Add the Helix Job Monitor#127749
premun wants to merge 8 commits intodotnet:mainfrom
premun:prvysoky/helix-job-monitor

premun commented May 4, 2026

Uh oh!

dotnet-policy-service Bot commented May 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

premun commented May 4, 2026

Uh oh!

dotnet-policy-service Bot commented May 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants