Document How To Install Python Package from a Private Repo

[dthomas-sensonix]

I have a number of Python Packages in private (company) repos and I am using GitHub Actions to run pytest on commits. One of the repos depends on packages from other repos. When pip runs from the Action, I see the following error:

Collecting pyconfig@ git+ssh://git@github.com/sxi/pyconfig@master
Running command git clone -q 'ssh://****@github.com/sxi/pyconfig' /tmp/pip-install-wglwufhp/pyconfig
Cloning ssh://****@github.com/sxi/pyconfig (to revision master) to /tmp/pip-install-wglwufhp/pyconfig
Warning: Permanently added the RSA host key for IP address '140.82.114.4' to the list of known hosts.
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com/sxi/pyconfig' /tmp/pip-install-wglwufhp/pyconfig Check the logs for full command output.
##[error]Process completed with exit code 1.

Please document how the user can grant access to private repos to the Action. For example, I solved the problem using the following:

1. For every Python package, create a step to check out the private repo

    - name: Checkout pyconfig from a private repo
      uses: actions/checkout@v2
      with:
        repository: <company>/pyconfig
        token: ${{ secrets.ACCESS_TOKEN }}
        path: pyconfig

2. Modify the “Install dependencies” stop to pip install each Python package sourced from a private repo

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install /home/runner/work/<path>/pyconfig

While this works, it is a little tedious. Tech Support suggested I use HTTPS with a username and password to check out the packages from the private repositories. I would prefer to not use this method. It would likely require me to create and maintain a "fake" user account just for checking repositories in GitHub Actions. I would much prefer to use a personal access token (like I did above), but in a more simplified manner.

I want to work on #104

[ctoth]

Is there a canonical way to do this now in 2022?

[SJCaldwell]

I would also love to know if there was a canonical way to do this in 2022.

[ctoth]

So far the best way I've found to do this is by creating a Machine User and inviting it to the organization.

Then, in requirements.txt you can add lines like:

git+ssh://git@github.com/<org>/<package>.git

Generate and add an SSH key for the machine user, then use something like

shimataro/ssh-key-action

to set the key in your workflow from a secret.

There is more information at Using organization Python package in Github actions without Python repository although this only seems to work for one package and so that's why I ended up having to use the machine user approach as usually if you depend on one dependency in an organization you will depend on others.

Hope somebody finds this useful.

[victorsevero]

Not really canonical but this workaround might be a little better:

Just add
git config --global url."https://${{ secrets.GH_TOKEN }}@github".insteadOf https://github
at the beggining of your run step (before installing dependencies).

[fuji44]

I tackled the same issue a long time ago. My conclusion at the time was much the same as the one described by croth.

Then I learned that GitHub recommends authentication via HTTPS, and I felt the need to reconsider this issue, but I neglected it.

Recently, I had to do some maintenance on an old project, so I thought about my own best practices at this point. It is a combination of the methods described by croth and victorsevero.

In other words, do the following.

• Define requirements.txt and pyproject.toml with the https schema.
• Add the following steps at the beginning of the CI
  • Register private key with ssh-agent.
  • Add URL rewrite setting to Git config
    • ex. git config --global url."ssh://git@github.com/".insteadOf https://github.com/

I checked and found that for GitHub Actions, this can be easily accomplished by using webfactory/ssh-agent.
The README is very carefully written, so I recommend reading it.

[tyriis]

I have the following up and running successful:

    steps:
      # https://github.com/actions/checkout
      - name: checkout
        uses: actions/checkout@v3.2.0

      # https://github.com/marketplace/actions/setup-python
      - name: Setup Python
        uses: actions/setup-python@v4.3.1
        with:
          python-version: "3.10"

      - name: replace requirements
        run: sed -i "s/ssh:\/\/git@github\.com/https:\/\/${{ secrets.GH_USERNAME }}:${{ secrets.GH_PAT_WITH_ACCESS_TO_OTHER_REPOS }}@github\.com/g" requirements.txt

      - name: install
        run: pip install -e . && pip install -r requirements.txt

      - name: test
        run: python test.py

will try out another variant with a gh app token.

[blieusong]

Same issue, and had to resort to @victorsevero (and others') solution

[aripollak]

There's another option now that I believe is a bit more secure than managing a separate machine user and its associated credentials: Create a new Github App with "Contents" permissions, add it to the private repositories that the Github Action needs to access, and use something like https://github.com/marketplace/actions/action-github-app-token to get an access token that can be used during git checkout. Unfortunately this still requires the app's private key to be saved as a secret for the Action.