Authentication for private git repository

[venaturum]

I've seen examples of how to add a git repo as dependency

[tool.poetry.dependencies]
myrepo = { git = "https://github.com/me/myrepo.git" }

and I've seen many different ways of providing credentials for private pypi repositories.

But I'm still yet to understand how to provide credentials for a private git repository.

I think I can add username and token via:

[tool.poetry.dependencies]
myrepo = { git = "https://username:token@github.com/me/myrepo.git" }

but I don't want to store the token in pyproject.toml

It's also unclear if I could use an environment variable, or setup keyring somehow.

Does anybody know either way what is possible?

[venaturum]

As a workaround for a build pipeline I'm currently using

poetry remove myrepo
poetry add git+https://username:TOKEN@github.com/me/myrepo.git

in the build script, where TOKEN is a secret environment variable, so that presumably the value of TOKEN gets hardcoded into the pyproject file, but since the pyproject file is temporary, it's not such of an issue.

[dnlopes]

This almost worked 100% for me. However in my case, I need to specify a tag/branch in that dependency dynamically, so it's not so trivial.

[sriharsha-tnf]

@dnlopes just add #branch or #v0.x.x at the end for branch and tag respectively

[btjones-me]

+1 to this question

[lochsh]

I use SSH for Git authentication. So, my SSH keys are automatically used for poetry dependencies specified like poetry add git+ssh://git@github.com/account/repo. ssh-agent makes it convenient as it avoids having to type your SSH key password on every command requiring git authentication.

This makes local development on poetry projects with private GitHub repo dependencies easy. However, I had issues setting up CI with GitHub Actions.

I settled on configuring git to replace ssh://git@github.com URLs with ones using a PAT:

git config --global url."${user}:${PAT}@github.com".insteadOf ssh://git@github.com

I don't know if I'd recommend this for local development as it would mean your PAT is sitting unsecured in your git config file. (Whereas for GHA it's thrown away after the CI run. The PAT would be stored as a Secret that is accessed via ${{secret.SECRET_NAME}} in the GHA yaml.)

It took me a while to figure out exactly how to do this in GitHub Actions (there are good docs for doing it with GitLab CI), so I wanted to post my solution here after finding this question during my search.

[danseely]

Sorry for the zombie reply! I'm having trouble getting poetry to respect this git global config in Github actions, and it sounds like you were able to get it working. Mind peeking at what I have to see if it lines up with what you came up with?

Minor difference with my setup: we're using ssh pointers in the pyproject.toml file, and trying to rewrite those to https pointers. However, I think the logic behind the git config should hold.

Here is how we point to our private repos in the pyproject.toml file:

package-name = {git = "ssh://git@github.com/our-org/package-name.git", rev = "main"}

And here's the block in my Github action config:

- name: Install dependencies
  run: |
    apt-get update && apt-get install -y build-essential gcc git libpq-dev
    pip install --upgrade pip
    pip install poetry
    poetry --version
    git config --global url."https://${{ secrets.GH_TOKEN }}".insteadOf "ssh://git"
    poetry config virtualenvs.create false
    poetry install --without dev,scripts

To confirm, we do have GH_TOKEN defined in our org as a valid Github access token. And I've tried using one of the install-poetry actions instead of doing it manually with pip, with no difference.

When the action runs, I can see that poetry is trying to install the packages with the ssh pointer:

Command '['git', 'clone', '--recurse-submodules', '--', 'ssh://git@github.com/our-org/package-name.git', '/home/our-user/.cache/pypoetry/virtualenvs/outer-project-name-hash-py3.10/src/package-name']' returned non-zero exit status 128.

[radekska]

Hey @danseely,

I had the same issue on CI ( exit code 128). It seems that below git config fixed the issue, take a look at oauth2 prefix.

git config --global url."https://oauth2:${GITHUB_TOKEN}@github.com/Organization/repo".insteadOf "https://github.com/Organization/repo"

[danseely]

Thanks! I don't think that was it for me though. I was having trouble with the git config insteadOf directive even being followed at all; it turns out it wasn't making it into the git config of the github runner context.

[Congee]

Well, I also use ssh but unfortunately I have multiple accounts with different ssh keys. pip or poetry does not pick the correct private ssh key as the plain git command does. And, ssh in this environment does not resolve a host alias in ~/.ssh/config

ssh: Could not resolve hostname github-xxx:repo: Name or service not known

[Congee]

I solved this problem by prefixing an environment variable GIT_SSH_COMMAND="ssh -i ~/.ssh/id_ed25519_xxx@work -o IdentitiesOnly=yes"

[campanelli-sunpower]

I recently came up with a .git-credentials-based method for HTTPS auth to Github using a token, except that I am using setuptools instead of poetry (with a pyproject.toml and without any setup.py). The trick here was working around the inability to easily interpolate environment variables in the setup.cfg file. Because the solution was in the git config rather than in the build system, it may work for poetry too, but I'm not 100% sure. I'm happy to provide details if requested.

[Congee]

You successfully made me interested in how this might work with poetry

[campanelli-sunpower]

In the setup.cfg of the "application" repo, the private "dependency" repo (with optional extras) can refer to a branch or tag using this syntax (syntax differs from requirements.txt and pip command line):

...
[options]
install_requires =
    private_dependency_repo[extra1,extra2] @ git+https://github.com/org/repo_name.git@branch_or_tag
    ...

Then in the build script (say, in an AWS EC2 instance or Dockerfile), I first echo the HTTPS token from the environment (here, GH_TOKEN) into the .git-credentials file in the home directory (git's default location, but customizable) before pip installing the application (and then removing the cred's file for security). For example, in a Dockerfile with the application repo copied over:

RUN echo "https://${GH_TOKEN}:@github.com" > ${HOME}/.git-credentials \
  && git config --global credential.helper store \
  && python -m pip install . \
  && rm -rf ${HOME}/.git-credentials

Note the colon (:) in the line https://${GH_TOKEN}:@github.com and the --global flag in the subsequent line. Also, the .git-credentials is created first to allow headless operation on the first run (instead of otherwise prompting for username and password).

I also note that on my Mac, with credential.helper=osxkeychain, I do not need to do any of this because in my case both private repo's (the application and the dependency) are in the same org and it "just works" :). It may be that multiple org's would need to be called out separately in the URLs in the .git-credentials with their respective tokens but I have not tried that.

Reference: https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage

[Congee]

This approach is useful in CI runners. Thank you!

[bitphage]

Thanks, works for me, resulting action step is:

      - name: Install dependencies
        run: |
          source "$HOME/.poetry/env"
          echo "https://${{ secrets.SERVICE_ACCOUNT_PAT }}:@github.com" > $HOME/.git-credentials
          git config --global credential.helper store
          poetry config virtualenvs.in-project true
          poetry install
          rm -f $HOME/.git-credentials

[Sricharan-H]

I am having facing a issue w.r.t organization wide fine grained tokens.
I'm able to do this: poetry add git+https://username:<PAT>@github.com/me/myrepo.git where PAT is my personal github account's Personal Access Token .
What I'm not able to do is poetry add git+https://username:<fine grained PAT>@github.com/me/myrepo.git. It throws the error Max retries exceeded with url:
Note that I'm able to git clone https://username:<fine grained PAT>@github.com/me/myrepo.git which means my fine grained PAT for the org is working fine.

[PhilippeGalvan]

Answer completing this comment:
#3794 (reply in thread)

If locally you are storing your credentials in your git config file, you can try a simple elegant solution that works for gitlab but should have an equivalent for github:

Let's say you have the following custom package in pyproject.toml:

my-project = {git = "https://gitlab.custom_domain.com/path/to/my/repo/my-project.git", tag = "v1.0.0"}

in the CI you would add a git URL substitution that you would need to adapt to your gitlab domain (if not gitlab.com)

script:
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.custom_domain.io/".insteadOf "https://gitlab.custom_domain.com/"
    - poetry install

CI_JOB_TOKEN is a feature that might exist in github too!
https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html

Event without it, another substitution or directly configuring credentials in your store should work fine because poetry is evaluating git thus it will apply your config.

[tugot17]

For Github the way to configure it is:

- git config --global url."https://${GITHUB_TOKEN}@github.com/Organization/repo-name".insteadOf "https://github.com/Organization/repo-name"

[svagier]

If you are using Bitbucket, you can grant access to a repository using Repository Access Tokens. What's great about them is that you can set permissions very granularly - also "read only". You can revoke these tokens anytime.
To use them (assuming your git repository that you want to include as a dependency is called myproject), run these in CI/command line:

poetry config repositories.myproject https://github.com/org/myproject.git
poetry config http-basic.myproject x-token-auth ${{ YOUR_BITBUCKET_REPOSITORY_ACCESS_TOKEN }}

and then you can simply include it in pyproject.toml:

[tool.poetry.dependencies]
myproject = {git = "https://bitbucket.org/org/myproject.git", branch = "main" }

[abn]

For more information on using tokens, see the documentation.

You can also use gitcredentials. However in this case, you will use the system git client. Tokens are preferred for now.

[JPHutchins]

This did not work for me on GitHub, I had to use the git config hack to get it going:

      - name: poetry install
        run: |
          git config --global url."https://x-token-auth:${{ secrets.PAT_TOKEN }}@github.com".insteadOf "ssh://git@github.com"
          poetry install

I guess there's an advantage that local users can still use SSH. 🤷‍♀️

[abn]

The config above will only work for cases where the vcs dependency is specified using https in your pyproject.toml.

[JPHutchins]

Yes, I tried it first in that manner, and it did not work, yet the git config hack did.

[ferreteleco]

Hi @dlahyani where can I find documentation on the use of the http-basic auth for poetry? In their documentation I could only find the option of username and password, not PAT based authentication. Am I missing something?

My dependency is specified in the pyproject.toml file as git over HTTPS and I cannot make it work. Is there anything I could try?

Thanks in advance