Git LFS with a token doesn't work with GitHub Enterprise Server without subdomain isolation #415
Comments
ghost
changed the title
|
@Roda83: Today I tried to reproduce this issue without success. This is what I did (on both GitHub Enterprise Cloud and GitHub Enterprise Server):
Contents of workflow file: |
|
Hi @dassencio, could you set the environment variable GIT_CURL_VERBOSE=1 on the action runner and check whether you also see two authorization header in the git lfs request like I have described in my comment. Second stuff: Do you have also enabled the protection mode for the GitHub Enterprises Server so that public repositories can't be accessed from outside? If it works in your case I expect that you have only one authorization header. So it would be interesting to see what will be the response of an /info/lfs/objects/batch request again your server. If you like we can also do a short session together to take a look on that. |
|
@Roda83 Hmm. We still cannot reproduce your results. We used the following workflow: Which generated the following debug output: We've run our test on GHES 2.22.5. What puzzles me is, that your LFS files are downloaded from Can you tell us:
|
Yes, the instance is also in private mode (like yours). |
2.22.3
I don't know but I will try to find it out
So interesting: In your case only the RemoteAuth authorization header will be set! |
|
@Roda83 I have a suspicion what is going on: On your machine subdomain isolation is not enabled. Therefore, the Git In general it is highly recommended to enable subdomain isolation as this is an important GHES security feature. More info here: https://docs.github.com/en/enterprise-server@2.22/admin/configuration/enabling-subdomain-isolation We will try to fix this problem. Can you try to enable subdomain isolation in the meantime? |
|
@larsxschneider, subdomain isolation is not enabled on our system. Do we can expect any side effects by enabling this? Or why is it not enabled by default? THX for the help and the efforts! |
|
@Roda83: Subdomain isolation is not enabled by default. While we strongly recommend enabling it to prevent cross-site scripting attacks, it requires additional DNS configuration (in your case, on Azure) to work. @larsxschneider: Thank you very much for sharing your expertise on this topic with us! |
|
Thanks @dassencio ! @Roda83 I 💯 agree with @dassencio . One more note: you likely need a new TLS certificate if you enable subdomain isolation. I proposed a change in Git LFS to handle that situation better in the future: git-lfs/git-lfs#4369 |
|
I'd like to point out that the correct way to handle this is by Actions using a custom credential helper as outlined in #162. If Apple Git doesn't work properly, please detect that and reject using that version instead of using the extra header. The behavior of Git LFS in this case is that it sends two Authorization headers and the GitHub LFS server rejects that. While not compliant with the HTTP spec, this is also the behavior that Git has, but the GitHub Git server happens to silently accept that, which is a bug and will likely change soon. Git LFS in this case is behaving in a bug-for-bug compatible way with Git, and this problem really needs to be fixed correctly on the Actions side. |
|
Thanks all! Wondering if there is an easy workaround for customers in this situation. For example, something like perform checkout action with SSH might also be an easy workaround. Albeit some up-front configuration, but also might more closely resemble how customers work locally. |
ericsciple
changed the title
|
Also of course other workaround is to enable subdomain isolation, which is highly recommended anyway per this doc :) |
|
@ericsciple Yes currently we are using the checkout-action v2 with |
|
Not sure if it helps here, but it might help others finding this issue: When the Git repo and Git LFS URLs at least differ at some point, sending the authorization header can be limited to the regular git repo requests with this bash line as a workaround: #!/bin/bash
git ${GIT_HTTP_ACCESS_TOKEN:+-c http.${repository_url%/*}/.extraHeader="Authorization: Bearer $GIT_HTTP_ACCESS_TOKEN"} clone ${repository_url}Example Git repo: will result in #!/bin/bash
git -c http.https://some.bitbucket.server/bitbucket/scm/project/.extraHeader="Authorization: Bearer <GIT_HTTP_ACCESS_TOKEN>" clone https://some.bitbucket.server/bitbucket/scm/project/repository.git |
|
An obvious solution is to simply stop using |
|
Hello all, Could someone please further explain what "git-lfs pull over SSH" means here?
When I set Thanks! |
|
workaround does not seem to work I have a little doubt regarding what you exactly meant with git lfs pull over SSH. I used the following two actions but I still get the same LFS: Client error on GitHub actions. Could you please kindly confirm, if this is the setup you were referring to? Is there anything else to add in order to use git lfs over SSH? |
|
I think , the workaround means setup ssh config to connect GHES.
work flow |
|
+1 |
|
I used a workaround I've made purely for Python, even before I had a chance to use actions. Any args are optional, but the key feature is to use a tokenized URL like this:
For my luck GitHub actions always use the actual token, while in Python I had to refresh it each time I ran this command. |
|
Any update here? We are facing the same issue with GHES without subdomain isolation. No checkout with LFS possible. It looks like a bug to me as it seems to violate RFC with the duplicate authorization header |
|
I found a workaround for this issue: - uses: actions/checkout@v3
with:
# actions/checkout + LFS doesn't work on GhES without subdomain isolation (actions/checkout#415)
lfs: false
- name: LFS checkout
run: |
git lfs install --local
# disable credentials by actions/checkout for LFS endpoints
git -c "http.${GITHUB_SERVER_URL}/storage/lfs/.extraHeader=" lfs pull |
|
#415 (comment) This workaround worked for me, thank you! I had attempted a bunch of similar config combinations but hadn't found that specific one. Should this bug be something that is addressed in the checkout action, or should it be reported to git-LFS? |
Great workaround! And if you have submodules with LFS, you can do something like: It's still a bummer that a workaround is needed at all, though. |
|
It really is wired that there is no fix for this. We now changed our infra to support subdomain isolation which was quite a hassle as GHES does not support Letsencrypt DNS-01 and our server is not publicly reachable |
|
FYI even after enabling subdomain isolation this does not work properly. It works if you just check out the repository the workflow is running in without specifying the repository. - name: Checkout repository with repo and path, no ssh, SomeOrg/my_little_repo
uses: actions/checkout@v4
with:
lfs: true
repository: Platform/my_little_repo
path: my_little_repo_explicitIt does not work if you specify a path to a repository. It looks like it is then not making use of the subdomain isolation and falls back into the initial problem of duplicated headers. - name: Checkout repository with repo and path, SomeOrg/my_little_repo
uses: actions/checkout@v4
with:
lfs: true
repository: SomeOrg/my_little_repo
path: my_little_repo_explicit
ssh-key: ${{ secrets.XXXX }}
ssh-known-hosts: ${{ vars.YYYY }} |
We have found an issue regarding GitHub checkout action v2, Git LFS and GitHub Enterprise Server (on Azure).
We have a very simply workflow which doesn't work:
All git lfs request will be rejected with following error:
HTTP/1.1 400 Bad RequestBy enabling
GIT_TRACE=1 and GIT_CURL_VERBOSE=1we see following:My current understanding:
The GitHub checkout action is using the extraheader option with basic authorization in the local git config.
So for each request this basic authorization header will be used.
In additional a remoteAuth authorization header will be added as result of the git lfs batch api reponse
/info/lfs/objects/batch:{ "objects": [ { "oid": "fcc622faad3b44962e9211cc2fd478e7c0480d516098fab011ccdb1d29fbde81", "size": 4612119, "actions": { "download": { "href": "...", "header": { "Authorization": "RemoteAuth AAAAAMHDN3KJWR****" } } } } ] }Now we have two authorization headers.
In this comment git-lfs/git-lfs#4031 (comment) one of the git lfs maintainer mentions that the git lfs server only allows request with one authorization header.
So all requests with two authorization headers will be rejected.
The text was updated successfully, but these errors were encountered: