private submodule checkout fails @v2 #116
Comments
|
Sub-modules support was removed in v2, I'm not sure why though |
I understood that it refers to another syntax |
tvainika
changed the title
|
I also need to have sub-module support. When I use v2, there's no |
|
I have the exact same issue but I just made it work using the Checkout multiple repos (private) example This is working just as a workaround and might not be practical for a project with a lot of submodules, but I hope it helps until #72 is resolved |
|
@rodrigorn It works for some use cases. Even with a single submodule, it can become inconsistent when the master branch |
|
This is actually blocking us from using GitHub actions. I hope it can be prioritized.. |
|
@tvainika the default token embedded in the main repo does not have access to fetch other private repositories. Instead if you supply |
|
@neutrinog no .git folder indicates git 2.18 or higher is not in your PATH. I updated the readme earlier today to make that more clear. |
@ericsciplec @rodrigorn is there a better alternative? I would like to avoid maintaining submodule hashes/branches in two different places. This is very error prone.. |
|
I'm having the same issue, came here from the readme where it has a section on "Checkout Submodules" that does not work at all for me (I can't checkout any submodule, public or private). |
I agree completely but haven't found any better solution yet |
|
FYI this is what I have been using: - name: Checkout submodule
run: |
git config --file=.gitmodules submodule.lib/YOUR_SUBMODULE.url https://${{ secrets.CI_PAT }}:${{ secrets.CI_PAT }}@github.com/ORG/YOUR_SUBMODULE.git
git submodule sync
git submodule update --init --recursive |
|
@Lauszus is the CI_TOKEN a personal access token? |
|
@ashwinvis yes it is. I have updated my comment to make it more clear. |
|
mm.. I am tempted to make a python script that does what @Lauszus suggests automatically for for all repositories defined in .gitmodules |
|
@jleni here's a version that will checkout all repositories defined in It works for both - name: Checkout submodules using a PAT
run: |
git config --file .gitmodules --get-regexp url | while read url; do
git config --file=.gitmodules $(echo "$url" | sed -E "s/git@github.com:|https:\/\/github.com\//https:\/\/${{ secrets.CI_PAT }}:${{ secrets.CI_PAT }}@github.com\//")
done
git submodule sync
git submodule update --init --recursive |
|
oh wow! that's awesome!! thanks!! |
The Example Example name: Test
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
token: ${{ secrets.MY_GITHUB_PAT }}
- name: Checkout submodules
shell: bash
run: |
auth_header="$(git config --local --get http.https://github.com/.extraheader)"
git submodule sync --recursive
git -c "http.extraheader=$auth_header" -c protocol.version=2 submodule update --init --force --recursive --depth=1 |
|
@samkit-jain yeah, that is why I ended up using my code above: #116 (comment), as my private repositories use |
|
@Lauszus +1 good point. i think adding inputs for ssh support + submodule support would create a generally frictionless experience for folks looking to checkout submodules. |
|
As an alternative to @Lauszus solution to rewrite the https://github.com/webfactory/ssh-agent You can generate an SSH keypair and add the public key to the repository you need access to (a normal Deploy Key) and then add the private key as a secret in your repo that's running the GH Action I'm not suggesting this is great, but until there's native support for at least something resembling same-org mutual authentication for actions, this may be as good as it gets. |
|
Similar to some of the comments above, I got it working with the following, where GITHUB_ACCESS_TOKEN is a personal access token in the format username:token, that is base64 encoded. Would be great for submodules and private submodules to be supported more directly. |
|
I got frustrated switching my submodules from SSH to HTTPS and dealing with PATs so I decided to build my own action for those wishing to checkout private or public submodules via SSH in their workflows: submodule-checkout |
|
thanks to @alicia 's work, we also got private ssh submodules working! However, we ended up with a slightly different result: on: push
jobs:
check-elm:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Checkout submodules
shell: bash
run: |
# From https://github.com/actions/checkout/issues/116#issuecomment-583221947
git config --global url."https://github.com/".insteadOf "git@github.com:"
git submodule sync --recursive
git -c "http.extraheader=Authorization: basic ${{secrets.GITHUB_ACCESS_TOKEN}}" -c protocol.version=2 submodule update --init --force --recursive --depth=1
- uses: actions/setup-node@v1
with:
node-version: '8.16.0'
- run: npm run testThe main difference being |
|
Please see #81 (comment) for some background on why we chose to not implement support for submodules in the initial version of the v2 action. And keep an eye on the pull requests for a document discussing what features we are looking at adding. |
|
I tried steps:
- uses: actions/checkout@v2
with:
token: ${{ secrets.RUNNER_TOKEN }}
lfs: true
- name: Checkout submodules
shell: bash
run: |
auth_header="$(git config --local --get http.https://github.com/.extraheader)"
git submodule sync --recursive
git -c "http.extraheader=$auth_header" -c protocol.version=2 submodule update --init --force --recursive --depth=1but it fails due to LFS stuff:
Any ideas? EDIT: Actually, the result is identical after removing the |
This solution seems to work perfectly for linux runners. Is this a problem with powershell formatting the commands incorrectly? |
|
@766F6964 it does not work because it uses piping, |
|
For me, the issue was that the default token used by the github action didn't have access to clone the submodule repository. I provided access by creating a new secret in my private repo that contains a copy of my SSH github-private key. This private key provides access to both repositories. For more security, you could also use a more restrictive PAT and the token a keyword. Note that the key used needs to have access to both repositories cos it will be used for cloning the parent repository and the submodule repository. |
Did you succeed to add the same SSH public key into 2 repos? In my case github throws an error when I try to add it to the 2nd repo. |
This worked for me!! Thanks @ashwinvis |
|
@vavsab when using shaunco/ssh-agent@git-repo-mapping |
@TiagoGalvaoChange no, I haven't. My .gitmodules only contain links to specific commits. So you may fork and add more logic to it to cover your case. |
|
Maybe it will help somebody. This solution works when you want to keep flexibility of URL repos and still use GitHub Actions with Deploy Keys to access private submodules: |
@arolus , that doesn't seem to work. The yaml actions file doesn't understand the |
How did this work for you, @iamsushi10 ? I am getting this error.
|
I get this error when trying to use your script... Run git config --file .gitmodules --get-regexp url | while read url; do
sed: -e expression #1, char 86: unterminated `s' commandThe script look fine though. Do you know what the issue is @Lauszus ? |
From your error output it looks like you are running Windows. I don't think my script works with Windows. |
Need to checkout submodules in CI. actions/checkout#116
|
For anyone who gets here in the future, adding to @fbernaly's comment, when creating a PAT:
|
|
It's a little concerning that the only solution here is to create a PAT. The point of a Github Action is to centralise the deployment process for an organisation, that organisation may have multiple private repositories, for example defining graphql or protobuf contracts. They need importing into each repo during the build process as submodules. To rely on PAT for this process is a security concern as well as an availability risk. If the person who creates the PAT leaves the company, the whole company will now be unable to deploy due to a quite deep down and hidden error within this process. What is the recommended solution for allowing an organisation to deploy code that references submodules from within their own organisation? |
Oh that is true, @richw-kai. I didn't think of that issue with the PAT. I was so focussed on getting it to work in the first place! |
Because of that, I use GitHub App Installation access tokens which are not directly tied to a user but instead tied to the GitHub App installed in your organization. I use this action to use a previously configured and installed GitHub App to get a special token with more privileges (like access to other organization repos). This other action also seems to do the same. Then, my action would look like: name: Some test
on:
pull_request:
jobs:
some_job:
runs-on: ubuntu-22.04
name: Some action
steps:
- name: Get token
id: get_token
uses: machine-learning-apps/actions-app-token@master
with:
APP_PEM: ${{ secrets.AUTH_APP_PEM }}
APP_ID: ${{ secrets.AUTH_APP_ID }}
- name: Checkout repository and submodules
uses: actions/checkout@v3
with:
submodules: recursive
token: ${{ steps.get_token.outputs.app_token }} |
This worked for me and probably the best solution so far to not tie the submodule checkout permissions to a specific user. The only requirement is the GitHub App needs the read-only to the "Contents" permission. |
|
Can't believe there's still no better way to do this |
|
how can i checkout submodules on pull reqeust ??? ${{ github.token }} -> can't checkout submodules -> pull-request can use github.token |
|
This worked for me. After checking out the main repository, my pipeline runs this action to checkout any submodules. It's a little bit hacky, but it has worked for me. - name: Checkout the repo
uses: actions/checkout@v3.1.0
with:
persist-credentials: false
- name: Checkout submodule
run: |
git submodule sync --recursive
git -c protocol.version=2 submodule update --init --force --depth=1 --recursive |
and others
I've private git submodule, which contains our shared i18n resouces, with same organization owner as my main repo
I've configured git submodule in main repository as
Then I added this one from README to my github action workflow.
I expected this to work.
Instead I got
The text was updated successfully, but these errors were encountered: