Git LFS with a token doesn't work with GitHub Enterprise Server without subdomain isolation

We have found an issue regarding GitHub checkout action v2, Git LFS and GitHub Enterprise Server (on Azure).

We have a very simply workflow which doesn't work:

- uses: actions/checkout@v2
  with:
    lfs: 'true'

All git lfs request will be rejected with following error: HTTP/1.1 400 Bad Request
By enabling GIT_TRACE=1 and GIT_CURL_VERBOSE=1 we see following:

> GET /storage/lfs/3/objects/d5c5871801d62c64f453462558c3a4697ac162730e49d48461ce87bafa83684c HTTP/1.1
> Host: ***.westeurope.cloudapp.azure.com
> Authorization: RemoteAuth AAAAAF72C****
> Authorization: Basic * * * * *
> User-Agent: git-lfs/2.12.1 (GitHub; windows amd64; go 1.14.10; git 85b28e06)
....
> HTTP/1.1 400 Bad Request
> Content-Length: 150
> Content-Type: text/html
> Date: Thu, 17 Dec 2020 10:25:15 GMT
> Server: GitHub.com

My current understanding:
The GitHub checkout action is using the extraheader option with basic authorization in the local git config.
So for each request this basic authorization header will be used.

In additional a remoteAuth authorization header will be added as result of the git lfs batch api reponse /info/lfs/objects/batch:

{
    "objects": [
        {
            "oid": "fcc622faad3b44962e9211cc2fd478e7c0480d516098fab011ccdb1d29fbde81",
            "size": 4612119,
            "actions": {
                "download": {
                    "href": "...",
                    "header": {
                        "Authorization": "RemoteAuth AAAAAMHDN3KJWR****"
                    }
                }
            }
        }
     ]
 }

Now we have two authorization headers.

In this comment git-lfs/git-lfs#4031 (comment) one of the git lfs maintainer mentions that the git lfs server only allows request with one authorization header.
So all requests with two authorization headers will be rejected.

[dassencio]

@Roda83: Today I tried to reproduce this issue without success. This is what I did (on both GitHub Enterprise Cloud and GitHub Enterprise Server):

1. Created a private repository with files stored via Git LFS (about 30 files / 200MB in total).
2. Created a workflow file (contents below).
3. Set up a self-hosted runner with Windows 2019 Server => workflow succeeded.
4. Added 10000 small files (1KB each) to the repository (all stored via Git LFS) => workflow succeeded.

Contents of workflow file:

name: Git LFS test
on:
 push:
   branches: [ main ]
jobs:
 checkout:
   name: Checkout
   runs-on: self-hosted
   steps:
   - name: Checkout with Git LFS
     uses: actions/checkout@v2
     with:
       token: ${{ github.token }}
       lfs: 'true'
       clean: 'true'
       fetch-depth: 0
   - name: List files
     run: ls

Hi @dassencio, could you set the environment variable GIT_CURL_VERBOSE=1 on the action runner and check whether you also see two authorization header in the git lfs request like I have described in my comment. Second stuff: Do you have also enabled the protection mode for the GitHub Enterprises Server so that public repositories can't be accessed from outside? If it works in your case I expect that you have only one authorization header. So it would be interesting to see what will be the response of an /info/lfs/objects/batch request again your server. If you like we can also do a short session together to take a look on that.

[larsxschneider]

@Roda83 Hmm. We still cannot reproduce your results.

We used the following workflow:

    - name: Checkout with Git LFS
      uses: actions/checkout@v2
      env:
        GIT_TRACE: 1
        GIT_CURL_VERBOSE: 1
      with:
        token: ${{ github.token }}
        lfs: 'true'
        clean: 'true'
        fetch-depth: 0

Which generated the following debug output:

2021-01-15T13:23:28.6435698Z 13:23:28.640518 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
2021-01-15T13:23:28.6686330Z 13:23:28.667774 trace git-lfs: tq: running as batched queue, batch size of 100
2021-01-15T13:23:28.6686872Z 13:23:28.667774 trace git-lfs: fetch lars.bin [4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865]
2021-01-15T13:23:28.6687148Z 13:23:28.667774 trace git-lfs: tq: sending batch of size 1
2021-01-15T13:23:28.6887698Z 13:23:28.687779 trace git-lfs: api: batch 1 files
2021-01-15T13:23:28.7286412Z 13:23:28.727788 trace git-lfs: HTTP: POST https://testserver.com/test-user/test-repo.git/info/lfs/objects/batch
2021-01-15T13:23:28.7328524Z > POST /test-user/test-repo.git/info/lfs/objects/batch HTTP/1.1
2021-01-15T13:23:28.7328998Z > Host: testserver.com
2021-01-15T13:23:28.7329378Z > Accept: application/vnd.git-lfs+json; charset=utf-8
2021-01-15T13:23:28.7330177Z > Authorization: Basic * * * * *
2021-01-15T13:23:28.7330410Z > Content-Length: 186
2021-01-15T13:23:28.7330636Z > Content-Type: application/vnd.git-lfs+json; charset=utf-8
2021-01-15T13:23:28.7330876Z > User-Agent: git-lfs/2.13.2 (GitHub; windows amd64; go 1.14.13; git fc664697)
2021-01-15T13:23:28.7331100Z > 
2021-01-15T13:23:29.2899927Z {"operation":"download","objects":[{"oid":"4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865","size":2}],"transfers":["lfs-standalone-file","basic"],"ref":{"name":"HEAD"}}13:23:29.286957 trace git-lfs: HTTP: 200
2021-01-15T13:23:29.2900762Z 
2021-01-15T13:23:29.2901126Z 
2021-01-15T13:23:29.2901502Z < HTTP/1.1 200 OK
2021-01-15T13:23:29.2901845Z < Access-Control-Allow-Origin: *
2021-01-15T13:23:29.2902337Z < Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
2021-01-15T13:23:29.2924609Z < Cache-Control: private, max-age=60, s-maxage=60
2021-01-15T13:23:29.2925974Z < Content-Security-Policy: default-src 'none'
2021-01-15T13:23:29.2927479Z < Content-Type: application/json; charset=utf-8
2021-01-15T13:23:29.2932545Z < Date: Fri, 15 Jan 2021 13:23:29 GMT
2021-01-15T13:23:29.2942316Z < Etag: W/"00220fe66fdc3b2c509b8e0b07843c81"
2021-01-15T13:23:29.2945146Z < Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
2021-01-15T13:23:29.2945602Z < Server: GitHub.com
2021-01-15T13:23:29.2946226Z < Status: 200 OK
2021-01-15T13:23:29.2946589Z < Strict-Transport-Security: max-age=31536000; includeSubdomains
2021-01-15T13:23:29.2946957Z < Vary: Accept, Authorization, Cookie, X-GitHub-OTP
2021-01-15T13:23:29.2947304Z < X-Content-Type-Options: nosniff
2021-01-15T13:23:29.2947647Z < X-Frame-Options: deny
2021-01-15T13:23:29.2947989Z < X-Github-Enterprise-Version: 2.22.5
2021-01-15T13:23:29.2948448Z < X-Github-Media-Type: unknown
2021-01-15T13:23:29.2949822Z < X-Github-Request-Id: 9a6ae8eb-240d-421d-8098-8d53210d01eb
2021-01-15T13:23:29.2950328Z < X-Ratelimit-Limit: 3000
2021-01-15T13:23:29.2950693Z < X-Ratelimit-Remaining: 2999
2021-01-15T13:23:29.2951037Z < X-Ratelimit-Reset: 1610717069
2021-01-15T13:23:29.2951512Z < X-Runtime-Rack: 0.067881
2021-01-15T13:23:29.3040373Z < X-Xss-Protection: 1; mode=block
2021-01-15T13:23:29.3040800Z < 
2021-01-15T13:23:29.3049922Z 13:23:29.289960 trace git-lfs: HTTP: {"objects":[{"oid":"4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865","size":2,"actions":{"download":{"href":"https://media.testserver.com/lfs/2220/objects/4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865","header":{"Authorization":"RemoteAuth *****"}}}}]}
2021-01-15T13:23:29.3050696Z {"objects":[{"oid":"4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865","size":2,"actions":{"download":{"href":"https://media.testserver.com/lfs/2220/objects/4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865","header":{"Authorization":"RemoteAuth *****"}}}}]}13:23:29.289960 trace git-lfs: tq: starting transfer adapter "basic"
2021-01-15T13:23:29.3051247Z 13:23:29.289960 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
2021-01-15T13:23:29.3051505Z 13:23:29.290960 trace git-lfs: xfer: adapter "basic" started
2021-01-15T13:23:29.3051777Z 13:23:29.290960 trace git-lfs: xfer: adapter "basic" worker 0 starting
2021-01-15T13:23:29.3052056Z 13:23:29.290960 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865"
2021-01-15T13:23:29.3052377Z 13:23:29.301955 trace git-lfs: HTTP: GET https://media.testserver.com/lfs/2220/objects/4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865
2021-01-15T13:23:29.3052685Z > GET /lfs/2220/objects/4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865 HTTP/1.1
2021-01-15T13:23:29.3052906Z > Host: media.testserver.com
2021-01-15T13:23:29.3053164Z > Authorization: RemoteAuth *****
2021-01-15T13:23:29.3053430Z > User-Agent: git-lfs/2.13.2 (GitHub; windows amd64; go 1.14.13; git fc664697)
2021-01-15T13:23:29.3053641Z > 
2021-01-15T13:23:29.3053955Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 1 starting
2021-01-15T13:23:29.3054186Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
2021-01-15T13:23:29.3054427Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 2 starting
2021-01-15T13:23:29.3054653Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
2021-01-15T13:23:29.3054874Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 3 starting
2021-01-15T13:23:29.3055091Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
2021-01-15T13:23:29.3055313Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 4 starting
2021-01-15T13:23:29.3055536Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
2021-01-15T13:23:29.3055755Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 5 starting
2021-01-15T13:23:29.3055974Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
2021-01-15T13:23:29.3056374Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 6 starting
2021-01-15T13:23:29.3056616Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
2021-01-15T13:23:29.3056837Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 7 starting
2021-01-15T13:23:29.3057137Z 13:23:29.303959 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
2021-01-15T13:23:29.7233081Z 13:23:29.718335 trace git-lfs: HTTP: 200
2021-01-15T13:23:29.7234314Z 
2021-01-15T13:23:29.7235195Z < HTTP/1.1 200 OK
2021-01-15T13:23:29.7314250Z < Content-Length: 2
2021-01-15T13:23:29.7316031Z < Accept-Ranges: bytes
2021-01-15T13:23:29.7316613Z < Access-Control-Allow-Origin: https://render.testserver.com
2021-01-15T13:23:29.7317693Z < Content-Security-Policy: default-src 'none'
2021-01-15T13:23:29.7317941Z < Content-Type: application/octet-stream
2021-01-15T13:23:29.7318194Z < Date: Fri, 15 Jan 2021 13:23:29 GMT
2021-01-15T13:23:29.7318461Z < Etag: "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865"
2021-01-15T13:23:29.7334566Z < Last-Modified: Fri, 15 Jan 2021 13:23:29 GMT
2021-01-15T13:23:29.7338301Z < Server: GitHub.com
2021-01-15T13:23:29.7382090Z < Strict-Transport-Security: max-age=31557600
2021-01-15T13:23:29.7387285Z < Timing-Allow-Origin: https://testserver.com
2021-01-15T13:23:29.7458045Z < X-Content-Type-Options: nosniff
2021-01-15T13:23:29.7469308Z < X-Frame-Options: deny
2021-01-15T13:23:29.7475441Z < X-Github-Request-Id: db6ebe81-5734-11eb-8ddc-b7119db16db3
2021-01-15T13:23:29.7482420Z < X-Xss-Protection: 1; mode=block

We've run our test on GHES 2.22.5. What puzzles me is, that your LFS files are downloaded from GET /storage/ wheras in our test they are downloaded from GET /lfs/. I'll research that.

Can you tell us:

• What GHES version are you running on?
• Do you have subdomain isolation enabled?

[dassencio]

Do you have also enabled the protection mode for the GitHub Enterprises Server so that public repositories can't be accessed from outside?

Yes, the instance is also in private mode (like yours).

What GHES version are you running on?

2.22.3

Do you have subdomain isolation enabled?

I don't know but I will try to find it out

2021-01-15T13:23:29.3053164Z > Authorization: RemoteAuth *****

So interesting: In your case only the RemoteAuth authorization header will be set!
It would be interesting to check whether your GitHub checkout action is also setting the extraheader in your local git config??
I have checked the code of the checkout action in the master branch and normally that should be the case.
So for some reason your git lfs client is ignoring the extraheader setting in the git config.

[larsxschneider]

@Roda83 I have a suspicion what is going on:

On your machine subdomain isolation is not enabled. Therefore, the Git Authorization header is automatically applied since it is the very same domain as the Git clone URL. In my test case subdomain isolation is enabled. Therefore, the LFS files are downloaded from a different domain (media.foo.bar) and no additional header is applied.

In general it is highly recommended to enable subdomain isolation as this is an important GHES security feature. More info here: https://docs.github.com/en/enterprise-server@2.22/admin/configuration/enabling-subdomain-isolation

We will try to fix this problem. Can you try to enable subdomain isolation in the meantime?

@larsxschneider, subdomain isolation is not enabled on our system.

Do we can expect any side effects by enabling this? Or why is it not enabled by default?
Or is it enabled per default and for some reasons we have disabled it?
We would like to avoid to do any changes on Friday evening. So we will enable the subdomain isolation on Monday morning. I will give you feedback if this solves the issue.

THX for the help and the efforts!

[dassencio]

@Roda83: Subdomain isolation is not enabled by default. While we strongly recommend enabling it to prevent cross-site scripting attacks, it requires additional DNS configuration (in your case, on Azure) to work.

@larsxschneider: Thank you very much for sharing your expertise on this topic with us!

[larsxschneider]

Thanks @dassencio !

@Roda83 I 💯 agree with @dassencio . One more note: you likely need a new TLS certificate if you enable subdomain isolation.

I proposed a change in Git LFS to handle that situation better in the future: git-lfs/git-lfs#4369

[bk2204]

I'd like to point out that the correct way to handle this is by Actions using a custom credential helper as outlined in #162. If Apple Git doesn't work properly, please detect that and reject using that version instead of using the extra header.

The behavior of Git LFS in this case is that it sends two Authorization headers and the GitHub LFS server rejects that. While not compliant with the HTTP spec, this is also the behavior that Git has, but the GitHub Git server happens to silently accept that, which is a bug and will likely change soon. Git LFS in this case is behaving in a bug-for-bug compatible way with Git, and this problem really needs to be fixed correctly on the Actions side.

[ericsciple]

Thanks all!

Wondering if there is an easy workaround for customers in this situation.

For example, something like perform checkout action with lfs: false and persist-credentials: false and then script the LFS checkout.

SSH might also be an easy workaround. Albeit some up-front configuration, but also might more closely resemble how customers work locally.

[ericsciple]

Also of course other workaround is to enable subdomain isolation, which is highly recommended anyway per this doc :)