The ssh-key mode will be failed in checking out code from AWS EC2 with Agents started by SSM

[charlie-fox]

Hi Team,

I found an issue when we try to checkout code with SSH key in Github actions and AWS Windows EC2 as Runner ( the Actions Agent is started from SSM).

The checkout process responded us :

  C:\Windows\system32\icacls.exe C:\actions-runner\_work\_temp\bce01f23-14da-4a3f-bd52-5e95cc5f0517 /grant:r WORKGROUP\EC2AMAZ-XXXXX$:F
  WORKGROUP\EC2AMAZ-XXXXX$: No mapping between account names and security IDs was done.
  Successfully processed 0 files; Failed processing 1 files
Removing auth
Error: The process 'C:\Windows\system32\icacls.exe' failed with exit code 1332

it is very strange, because in Linux EC2 and normal windows runner without SSM (user normal windows GUI login), no such issues.

check the github Action process, found in Windows checkout, it has a special activity to grant permission:

https://github.com/actions/checkout/blob/2541b1294d2704b0964813337f33b291d3f8596b/src/git-auth-helper.ts#L221

It will use "USERNAME" and "USERDOMAIN" to locate current user and grant permission.

Check normal windows login shell (from Gui) and , the env list is :

USERDOMAIN                     EC2AMAZ-XXXXXX
USERNAME                       Administrator

They are correct.

But if you try to get such list from SSM shell :
it would be :

USERDOMAIN                     WORKGROUP
USERNAME                       EC2AMAZ-XXXXXX$

So in SSM Session manager mode, to make ssh key checkout work, we need overwrite USERDOMAIN and USERNAME to the correct one with job-> env, and then , the checkout process would work.

special tips in using SSM run-command:
in SSM run-command mode (we create runner automatically by AWS SSM run command, with ephemeral way) , the USERNAME should be "system" , but even I overwrite the USERNAME with system, it would still failed. because looks the system is a special username, it can be only work-around by overwriting the USERDOMAIN to blank, and overwrite the username to system.

I think it was caused by AWS issue in SSM agent, and I have filed case to AWS EC2 team.

but since AWS windows EC2 is widely used and more and more users are building autoscaled runner with automatic way like SSM, also suggest to use more stable way to obtain current USERNAME instead of just environment var.

Thanks.

[robpomeroy]

I've had what looks like an identical problem with a self-hosted Windows Server 2022 runner:

  C:\Windows\system32\icacls.exe C:\actions-runner\_work\_temp\f637cea8-35b0-4dd3-b600-3d83bdbb9ac8 /grant:r DR\GH-RUNNER-WIN-1$:F
  DR\GH-RUNNER-WIN-1$: Successfully processed 0 files; Failed processing 1 files
  No mapping between account names and security IDs was done.
  Error: The process 'C:\Windows\system32\icacls.exe' failed with exit code 1332

It occurred to me that the GitHub Actions service is installed as the "NETWORK SERVICE" user. My fix is to use the following overrides for USERNAME and USERDOMAIN (which I discovered through this issue raised by @charlie-fox):

      - name: Checkout the branch (remote)
        uses: actions/checkout@v3
        env:
          SSH_KEY: ${{ secrets.SSH_KEY }}
          USERDOMAIN: "NT AUTHORITY"
          USERNAME: "NetworkService"
        if: env.SSH_KEY
        with:
          repository: MyOrg/MyRepo
          ref: ${{ github.ref_name }}
          ssh-key: ${{ env.SSH_KEY }}

Context: this workflow is being called from a workflow in another repo. Not sure if that has a bearing on this issue. Secrets are set to inherited in the calling workflow. The if test is here because this workflow may also be called locally, in which case a different step performs the checkout without the SSH key.

[trend-charlie-cao]

Yes, overwrite the USERNAME and USERDOMAIN ENV can workaround, but looks not good solution.
Since it is very popular to run Github Windows Runner Agent as service, is it possible to have optimization to here

checkout/src/git-auth-helper.ts

Line 221 in 2541b12

// Remove inherited permissions on Windows

to make user need not to do overwriting?

[Gargaj]

Is there a known workaround for this that doesn't involve hardwiring the username into the YML? (We have multiple self-hosted runners so doing that would probably result in only working for one and not the other.)

[robpomeroy]

You could use a "whoami" step to capture the running username and domain? In PowerShell you'd want [System.Environment]::Domain and [System.Environment]::UserDomainName.

[Gargaj]

That doesn't seem to work well for me, I get the following values:

[System.Environment]::Domain:         ''
[System.Environment]::UserDomainName: 'WORKGROUP'

[robpomeroy]

@Gargaj 🤦 - sorry, I meant to mention [System.Environment]::UserName.

So you'd need a step that did something like:

      - name: Write username and domain name to environment variables
        run: |
          chcp 65001 # Set code page to utf-8
          Write-Output ( `
            "USERNAME=" + ([System.Environment]::UserName) `
          ) >> $env:GITHUB_ENV
          Write-Output ( `
            "USERDOMAIN=" + ([System.Environment]::UserDomainName) `
          ) >> $env:GITHUB_ENV

[Gargaj]

Ultimately I decided the easier method to go around is to just install it as a service but to use one of the existing users on the machine. That seems to have fixed everything.

[trend-charlie-cao]

Hi @Gargaj , Could you please elaborate on how you handled it? Are there any code snippets for the registration service with existing users that we can refer to? Thank you.

[Gargaj]

So this was happening on our own self-hosted machine, not on AWS, but what I did was just reinstall the runner as a service, but instead picking my own user account as the service owner instead of NetworkService.

[charlie-fox]

To Github checkout plugin team: I still think if this location can be opitimized to meet the scenario of "start Runner Agent through SSM in AWS Windows EC2", that would be best:

checkout/src/git-auth-helper.ts

Line 226 in b32f140

`"${icacls}" "${this.sshKeyPath}" /grant:r "${process.env['USERDOMAIN']}\\${process.env['USERNAME']}:F"`

After all, AWS is very popular public cloud, and SSM is a very secure way to operate EC2 in AWS instead of using password or token to avoid leakage. In a world with a lot cybersecurity threads, it can be encouraged to use more secure way to maintain Github runners.