'actions/checkout@v4' with LFS fails because of double auth header

[AngelKing86]

I am encountering an issue with the Git LFS integration in my Workflow. The problem involves the sending of double Authorization headers in HTTP requests, which results in a 400 Bad Request error. This issue is preventing successful data transfers and impacting the workflow execution. The problem mainly occurs in my submodules, since these require Git LFS.

The first LFS batch request works without problem and I see only one basic authorization.

> POST /***/***.git/info/lfs/objects/batch HTTP/1.1
> Host: repo.tld
> Accept: application/vnd.git-lfs+json
> Authorization: Basic * * * * *
> Content-Length: 1036
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.5.1 (GitHub; linux amd64; go 1.21.8)
>
{"operation":"download","objects":[{"oid":"c4055d651a6be1c2c1b6d03d96b0ce2c8485d53e1aae0aef6fe481c5868159e8","size":685307408},{"oid":"7d17b1e89a63bd6c590ad45fab7507c815881d2d2931fb64c35cefff731c[82](https://repo.tld/***/***/actions/runs/100#jobstep-6-82)4e","size":182695584},{"oid":"da34949f6fdcf578d6c63655aff797c02ca6b7c19d16a42063e5448648d6f050","size":151829144},{"oid":"ca80e4ff16714bf22ae2f05973f65b1a0e147f2bf04358a3e716ae51cf5be149","size":17343208},{"oid":"34e54dfbde73b0251882520be384539019ee6655b547c34e0b758d534396f011","size":11541008},{"oid":"a77fa64f194f0462e46049[83](https://repo.tld/***/***/actions/runs/100#jobstep-6-83)977f32b68c17fd5e4798bf1407e7b0512c05ac69","size":5719752},{"oid":"8e2c383551607e0a2862ac774cc5fc806c63fef7397528ab1a300c62dc6aa302","size":4808592},{"oid":"96567cfc911eecc60b8c32fa53d9c7ff232366dc6d7703ad71714645c78a872f","size":3122960},{"oid":"f7a5df0ce4ce747f8801eb4ec90de09e2f4fea6fec4c916c95e87b65204c3bdc","size":2338536},{"oid":"8[84](https://repo.tld/***/***/actions/runs/100#jobstep-6-84)163897dd3a8315f1984aa14862421f59e96e9ab7e11bb088124c7aa00989f","size":798976}],"transfers":["lfs-standalone-file","basic","ssh"],"ref":{"name":"HEAD"},"hash_algo":"sha256"}08:03:07.159[87](https://repo.tld/***/***/actions/runs/100#jobstep-6-87)6 trace git-lfs: HTTP: 200
< HTTP/2.0 200 OK
< Connection: close

Error Log
Here is a snippet from the workflow logs that illustrates the issue:

[...]
08:03:07.160465 trace git-lfs: tq: starting transfer adapter "basic"
08:03:07.160483 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
08:03:07.160499 trace git-lfs: xfer: adapter "basic" started
08:03:07.160529 trace git-lfs: xfer: adapter "basic" worker 0 starting
08:03:07.160551 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "c4055d651a6be1c2c1b6d03d96b0ce2c8485d53e1aae0aef6fe481c5868159e8"
[...]
08:03:07.160700 trace git-lfs: xfer: adapter "basic" worker 7 starting
08:03:07.160731 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
08:03:07.164080 trace git-lfs: HTTP: GET https://repo.tld.de/***/***.git/info/lfs/objects/c4055d651a6be1c2c1b6d03d96b0ce2c8485d53e1aae0aef6fe481c5868159e8
> GET /***/***/info/lfs/objects/c4055d651a6be1c2c1b6d03d96b0ce2c8485d53e1aae0aef6fe481c5868159e8 HTTP/1.1
> Host: repo.tld
> Authorization: Basic * * * * *
> Authorization: Basic * * * * *
> User-Agent: git-lfs/3.5.1 (GitHub; linux amd64; go 1.21.8)
> 
08:03:07.166248 trace git-lfs: HTTP: 400

Steps to Reproduce

• Configure checkout@v4 in a Workflow and enable LFS support
• Enable debug GIT_TRACE: 1, GIT_TRANSFER_TRACE: 1, GIT_CURL_VERBOSE: 1
• Observe the workflow execution.
• Check the logs to see the duplicated Authorization headers and the resulting 400 Bad Request error.

Expected Behavior
The HTTP requests made by Git LFS should include only a single Authorization header.

Actual Behavior
The HTTP requests include two Authorization headers, leading to a 400 Bad Request error.

More information about this issue:
https://gitea.com/gitea/act_runner/issues/164#issue-194072
https://gitea.com/gitea/act_runner/issues/164#issuecomment-739866
https://gitea.com/gitea/act_runner/issues/164#issuecomment-836693

[Mai-Lapyst]

For anyone interested here's a workaround to get this to work:

- uses: actions/checkout@v4
  with:
    persist-credentials: 'true' # Optional; should be the default

- name: Checkout lfs
  run: |
    git lfs install --local
    AUTH=$(git config --local http.${{ github.server_url }}/.extraheader)
    git config --local --unset http.${{ github.server_url }}/.extraheader
    git config --local http.${{ github.server_url }}/${{ github.repository }}.git/info/lfs/objects/batch.extraheader "$AUTH"
    git lfs pull

[joshmgross]

Can you share a workflow reproducing this issue?

[devhaozi]

Can you share a workflow reproducing this issue?

name: Generate Checksums
on:
  push:
    branches:
      - main
  pull_request:
permissions:
  contents: write
jobs:
  checksums:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          ref: ${{ github.head_ref }}
          lfs: true
      - name: Generate
        run: |
          set -e
          IFS=$'\n'
          for file in $(git ls-files | grep "^panel/" | grep -v ".sha256$"); do
            filename=$(basename "$file");
            sha256sum "$file" | sed "s|$file|$filename|" > "${file}.sha256";
          done
          unset IFS
      - name: Commit
        uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: "chore: update checksums"
          file_pattern: "*.sha256"

[bk2204]

As a former Git LFS maintainer and a Git contributor, the proper solution here is to use a credential helper. The Git FAQ outlines how to do this with a small shell script.

Using http.extraheader is a direction to Git and Git LFS to always, no matter what, add the extra header. The problem is that in some cases, the Git LFS protocol includes an Authorization header that has to be used, and then there's two headers, which the GitHub LFS server rejects. There is simply no way to use http.extraheader with GitHub's LFS server and have it work properly in all cases, and Git LFS can't change the behaviour because of compatibility with Git. (Notably, if you specify multiple of the same header with http.extraheader, Git will send all of them, even if that violates the HTTP spec.)

If you use a credential helper, then Git LFS will only call it when it doesn't have other credentials, and so it won't add a second Authorization header, and things will work. That's really the only viable approach that's going to work here in all situations.

If you need Bearer or other types of authentication, Git 2.46 supports extensions to the credential helper protocol to handle that case, and I added it specifically to get people to stop using http.extraheader (notably Azure DevOps, which requires Bearer auth). Git LFS 3.6.0 also supports those extensions.

Similarly, if you need to send the credentials on the first go and not wait for a 401 like Git normally does, then Git 2.47 has http.proactiveAuth, which can be set to do that. I tested it against GitHub's Git server when I implemented it, so I know it works.

I know there was some discussion elsewhere of broken versions of Apple Git that performed inappropriate caching, but I am happy as a Git contributor to simply say that changes that don't honour the original protocol are bugs. If you can provide an example of that problem to the Git list (and CC the personal email I have in the Git .mailmap), I'll CC the contributors from Apple and ask them to fix it so that's no longer a problem.

[iacs]

How does one enable the credential helper in the github action?

[iacs]

@bk2204 What is the recommended workflow to use these credential helpers?
If a command to config the credential helper is run before the checkout action, it will fail because we are not in a git directory yet. And obviously after the checkout action can't be done either, as it already failed the LFS step

[leonbohmann]

For anyone interested here's a workaround to get this to work:

• uses: actions/checkout@v4
  with:
  persist-credentials: 'true' # Optional; should be the default

• name: Checkout lfs
  run: |
  git lfs install --local
  AUTH=$(git config --local http.${{ github.server_url }}/.extraheader)
  git config --local --unset http.${{ github.server_url }}/.extraheader
  git config --local http.${{ github.server_url }}/${{ github.repository }}.git/info/lfs/objects/batch.extraheader "$AUTH"
  git lfs pull

This is the only thing that worked for me. Credential Helpers did nothing, double Auth Headers were still there.

[iacs]

@leonbohmann Thanks for the workaround. Do you know what is the equivalent of github.server_url for non-github but compatible runners (e.g. the one used by gitea). Should it be the url of the gitea instance?

[devhaozi]

@leonbohmann Thanks for the workaround. Do you know what is the equivalent of github.server_url for non-github but compatible runners (e.g. the one used by gitea). Should it be the url of the gitea instance?

For me, I use these commands directly instead of actions/checkout@v4

git clone --depth 1 ${{ github.server_url }}/${{ github.repository }}.git ${{ github.workspace }}/repo
cd ${{ github.workspace }}/repo
git checkout ${{ github.ref_name }}
git lfs install --local
git config lfs.${{ github.server_url }}/${{ github.repository }}.git/info/lfs.locksverify true
git lfs pull

[leonbohmann]

Not my workaround, just quoted an answer from above: #1830 (comment)

The github.server_url can remain unchanged. The gitea act runner picks it up just fine. I literally copy pasted the code into a workflow step. Just add a $ in front of auth, to make it correct Powershell.

[vikhik]

Adding a comment just to state that my team (Game Development) is currently considering options between GitLab Premium and GitHub Enterprise, and these kinds of issues make it more likely that we would go towards GitLab.

The workaround does work for us, but these kinds of hacks are... subpar. I would expect that a product using modern git would follow the git auth steps correctly (referencing this comment, above).