ghcr.io docker login is forced for JIT based self hosted enterprise runners

[aron-muon]

Summary
original bug: #1199
attempted fix: #1291
what they missed: the IsHostedServer bool isn't configured properly for JIT configs.
open source proposed fix: #3326

Describe the bug
When using GitHub Enterprise Server (GHES) with self-hosted runners managed by Actions Runner Controller (ARC) in a Just-in-Time (JIT) configuration (specifically runner agent v2.323.0), the runner agent incorrectly forces an authenticated docker login to ghcr.io when a workflow job specifies a public image from ghcr.io in its container: block. This login attempt uses GITHUB_ACTOR as the username and attempts to use GITHUB_TOKEN via --password-stdin. This occurs even when no credentials are specified in the container: block, and an anonymous pull is expected for a public image.

The root cause appears to be related to how the IsHostedServer boolean is determined within the runner agent for JIT configurations originating from GHES. The logic in ContainerOperationProvider.cs (specifically the UpdateRegistryAuthForGitHubToken method) defaults to injecting GITHUB_ACTOR and GITHUB_TOKEN if registryCredentialsNotSupplied is true and isFallbackTokenFromHostedGithub is true. However, isFallbackTokenFromHostedGithub (derived from HostContext.GetService().GetSettings().IsHostedServer) seems to be incorrectly evaluating to true for ghcr.io interactions in a GHES JIT context, leading the runner to believe it's interacting with a hosted GitHub service that should use the GITHUB_TOKEN.

This issue is a recurrence or a missed edge case from the original bug report #1199 , which was addressed by PR #1291. The fix in #1291 relied on the IsHostedServer property from settings, which, as noted in subsequent discussions (e.g., related to potential fix #3326 ), is not correctly set by the GHES endpoint for JIT configurations.  

To Reproduce
Steps to reproduce the behavior:

1. Configure GitHub Enterprise Server (GHES) with Actions Runner Controller (ARC) using Just-in-Time (JIT) runners. Ensure runner pods utilize a Docker-in-Docker (DinD) setup.

template:
  spec:
    initContainers:
      - name: init-dind-externals
        image: ghcr.io/actions/actions-runner:latest
        command: ["cp", "-r", "/home/runner/externals/.", "/home/runner/tmpDir/"]
        volumeMounts:
          - name: dind-externals
            mountPath: /home/runner/tmpDir
      - name: dind
        image: docker:dind
        restartPolicy: Always
        args:
          - dockerd
          - --host=unix:///var/run/docker.sock
          - --group=$(DOCKER_GROUP_GID)
        env:
          - name: DOCKER_GROUP_GID
            value: "123"
        startupProbe:
          exec:
            command:
            - docker
            - info
          initialDelaySeconds: 0
          failureThreshold: 24
          periodSeconds: 5    
        securityContext:
          privileged: true
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: dind-sock
            mountPath: /var/run
          - name: dind-externals
            mountPath: /home/runner/externals
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        env:
          - name: DOCKER_HOST
            value: unix:///var/run/docker.sock
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: dind-sock
            mountPath: /var/run
        securityContext:
          runAsUser: 1001
    volumes:
      - name: work
        emptyDir: {}
      - name: dind-sock
        emptyDir: {}
      - name: dind-externals
        emptyDir: {}

2. Use GitHub Actions runner version v2.323.0.
3. Create a GitHub Actions workflow with a job that defines a container: block specifying a public image from ghcr.io (e.g., image: ghcr.io/actions/actions-runner:latest or any other public ghcr.io image). Do not provide a credentials: block for the container.

jobs:
  test:
    name: Expose bug
    runs-on: [ actions-runner ]
    container:
      image: ghcr.io/actions/actions-runner:latest
      options: --user root

4. Trigger the workflow.
5. Observe the job logs during the "Starting job container" phase. You will see the runner agent attempting a docker login ghcr.io -u <GITHUB_ACTOR> --password-stdin command, which may fail.

Expected behavior
The GitHub Actions runner agent should recognize that a public image is being requested from ghcr.io without explicit credentials. It should then proceed with an anonymous docker pull operation. No docker login attempt using GITHUB_ACTOR and GITHUB_TOKEN should be made for public images from ghcr.io when no credentials are provided in the container: block.

Runner Version and Platform

Runner Version: v2.323.0
OS of the machine running the runner: Linux (as is typical for ARC-managed Kubernetes pods)

What's not working?

The runner agent incorrectly determines that authentication is required for pulling a public image from ghcr.io when running in a GHES JIT context.
The logic in ContainerOperationProvider.cs around line 520, specifically the UpdateRegistryAuthForGitHubToken method, injects GITHUB_ACTOR and GITHUB_TOKEN as credentials because isFallbackTokenFromHostedGithub (derived from HostContext.GetService<IConfigurationStore>().GetSettings().IsHostedServer) evaluates to true.

Relevant code snippet from ContainerOperationProvider.cs

        private void UpdateRegistryAuthForGitHubToken(IExecutionContext executionContext, ContainerInfo container)
        {
            var registryIsTokenCompatible = container.RegistryServer.Equals("ghcr.io", StringComparison.OrdinalIgnoreCase) || container.RegistryServer.Equals("containers.pkg.github.com", StringComparison.OrdinalIgnoreCase);
            var isFallbackTokenFromHostedGithub = HostContext.GetService<IConfigurationStore>().GetSettings().IsHostedServer;
            if (!registryIsTokenCompatible || !isFallbackTokenFromHostedGithub)
            {
                return;
            }

            var registryCredentialsNotSupplied = string.IsNullOrEmpty(container.RegistryAuthUsername) && string.IsNullOrEmpty(container.RegistryAuthPassword);
            if (registryCredentialsNotSupplied)
            {
                container.RegistryAuthUsername = executionContext.GetGitHubContext("actor");
                container.RegistryAuthPassword = executionContext.GetGitHubContext("token");
            }
        }

The IsHostedServer logic, as defined in Runner.Sdk/Util/UrlUtil.cs, does not seem to correctly account for the GHES JIT runner context when the target registry is ghcr.io.

public static bool IsHostedServer(UriBuilder gitHubUrl)
{
    if (StringUtil.ConvertToBoolean(Environment.GetEnvironmentVariable("GITHUB_ACTIONS_RUNNER_FORCE_GHES"))) // This is not hit for JIT configs
    {
        return false;
    }

    return
        string.Equals(gitHubUrl.Host, "github.com", StringComparison.OrdinalIgnoreCase) ||
        string.Equals(gitHubUrl.Host, "[www.github.com](https://www.github.com)", StringComparison.OrdinalIgnoreCase) ||
        string.Equals(gitHubUrl.Host, "github.localhost", StringComparison.OrdinalIgnoreCase) ||
        gitHubUrl.Host.EndsWith(".ghe.localhost", StringComparison.OrdinalIgnoreCase) ||
        gitHubUrl.Host.EndsWith(".ghe.com", StringComparison.OrdinalIgnoreCase);
}

For JIT configurations, HostContext.GetService<IConfigurationStore>().GetSettings().IsHostedServer is not being correctly set to false when the runner is on GHES but interacting with ghcr.io. This leads to the forced login attempt.  

Job Log Output

Checking docker version
Clean up resources from previous jobs
Create local container network
Starting job container
/usr/bin/docker --config /home/runner/_work/_temp/.docker_e765f1ac-14bb-40dd-973b-5d17a1584615 login ghcr.io -u <GITHUB_ACTOR> --password-stdin
Warning: Docker login for 'ghcr.io' failed with exit code 1, back off X.XXX seconds before retry.
/usr/bin/docker --config /home/runner/_work/_temp/.docker_e765f1ac-14bb-40dd-973b-5d17a1584615 login ghcr.io -u <GITHUB_ACTOR> --password-stdin
Warning: Docker login for 'ghcr.io' failed with exit code 1, back off Y.YYY seconds before retry.
/usr/bin/docker --config /home/runner/_work/_temp/.docker_e765f1ac-14bb-40dd-973b-5d17a1584615 login ghcr.io -u <GITHUB_ACTOR> --password-stdin
Error: Docker login for 'ghcr.io' failed with exit code 1

(Note: <GITHUB_ACTOR> will be the username of the user who triggered the workflow, e.g., "myusername" or "theirusername". The temporary docker config path will vary.)

Runner and Worker's Diagnostic Logs

Runner and Worker diagnostic logs (_diag folder) would show:

The ContainerOperationProvider invoking UpdateRegistryAuthForGitHubToken.
The subsequent call to _dockerManager.DockerLogin with the GITHUB_ACTOR and a temporary config path.
The IsHostedServer property likely evaluating to true within the UpdateRegistryAuthForGitHubToken method, despite the runner being registered to a GHES instance, because the container.RegistryServer is ghcr.io.
Logs from the DockerCommandManager executing the docker login command and its failure.

This behavior is consistent with the missed case from issue #1199 where JIT configurations do not correctly set the context for IsHostedServer when interacting with ghcr.io. The proposed community fix in #3326 aims to address this by more accurately computing IsHostedServer based on the GitHub server URL from the JIT configuration.

[alixinne]

Hello, author of #3326 here. Thanks for adding more context to this issue! This is exactly the problem we've had while hosting runners on EC2 using the philips-labs autoscaling solution in JIT mode against GHES.

We've been running a fork of this repository with the patch in #3326 for a year without issues on our side but it'd definitely be easier if it was upstreamed. Looking at recent contributions, maybe @TingluoHuang could take a look at this?

[aron-muon]

Hello, author of #3326 here. Thanks for adding more context to this issue! This is exactly the problem we've had while hosting runners on EC2 using the philips-labs autoscaling solution in JIT mode against GHES.

We've been running a fork of this repository with the patch in #3326 for a year without issues on our side but it'd definitely be easier if it was upstreamed. Looking at recent contributions, maybe @TingluoHuang could take a look at this?

Hello! Thanks for putting the PR out. I had an enterprise support ticket open regarding this issue and they said that they have an internal support ticket open and waiting to be resolved/deployed on any upcoming runner releases. They did not give me a timeline, but noted that it is being tracked. They referred me back to this GitHub issue I wrote for tracking, or to check new runner releases to see when it'd be released.

Ideally the new changes are tagged back to this repo so we can see when this gets resolved @TingluoHuang