Authentication for HTTPS cloning

[SebastianBalle]

I am currently investigating replacing our current git submodules with Gitman, however all our repositories are private hence authentication is required. It is not clear to me how Gitman configures authentication when cloning private repositories using HTTPS. All HTTPS examples are using public repositories on Github.

I would like to avoid using SSH.

Example of error when trying to access private repository using HTTPS with Azure DevOps:

Installing dependencies...

$ mkdir -p /Users/***/***/***/.gitman
$ cd /Users/***/***/***/.gitman

  $ git clone --reference-if-able /***/***/.gitcache/*** https://***@dev.azure.com/*** ***
  Cloning into '***'...
  fatal: Authentication failed for 'https://dev.azure.com/***/***/***/'
Traceback (most recent call last):
  File "/opt/homebrew/bin/gitman", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/lib/python3.10/site-packages/gitman/cli.py", line 239, in main
    _run_command(function, args, kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/gitman/cli.py", line 325, in _run_command
    success = function(*args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/gitman/decorators.py", line 9, in wrapped
    result = function(*args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/gitman/commands.py", line 101, in install
    _count = config.install_dependencies(
  File "/opt/homebrew/lib/python3.10/site-packages/gitman/models/config.py", line 109, in install_dependencies
    source.update_files(
  File "/opt/homebrew/lib/python3.10/site-packages/gitman/models/source.py", line 133, in update_files
    git.clone(
  File "/opt/homebrew/lib/python3.10/site-packages/gitman/git.py", line 79, in clone
    git("clone", "--reference-if-able", reference, repo, normpath, *user_params)
  File "/opt/homebrew/lib/python3.10/site-packages/gitman/git.py", line 16, in git
    return call("git", *args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/gitman/shell.py", line 94, in call
    raise ShellError(message, program=program, output=complete_output)
gitman.exceptions.ShellError: An external program call failed.

In working directory: /***/***/***/***/.gitman

The following command produced a non-zero return code:

$ git clone --reference-if-able /***/***/.gitcache/***.reference https://***@dev.azure.com/***/***/*** *** Cloning into '***'...
fatal: Authentication failed for 'https://dev.azure.com/***/***/_git/***/'

I am using Gitman v3.3.1

[jacebrowning]

Gitman calls git as a subprocess. You emulate this in a Python shell:

python
>>> import subprocess
>>> subprocess.run(["git", "clone", "https://dev.azure.com/***/***/***/"])

Does that work?

Here are the setup docs in case you missed them: https://gitman.readthedocs.io/en/latest/setup/git/

[daniel-brosche]

As mentioned try at first to clone the repository via calling git as subprocess to investigate the problem without gitman.
Have found some tipps that are dealing with this issue, may this help: https://stackoverflow.com/questions/35834117/authentication-failed-for-https-xxx-visualstudio-com-defaultcollection-git-pr

[SebastianBalle]

I am currently investigating this as I would like to find a solution that works both for local development and CI. I would prefer to configure git using command line for automatic build to be authenticated as well. FYI I am working with Azure Pipelines.

It seems like if you would like to use the System.AccessToken for authenticating using the identity of the pipeline. This will be done using the folllowing git command git clone -c http.extraheader="AUTHORIZATION: bearer <System_AccessToken>" [some-repo-url]. Is it possible to somehow inject this header into the git command invoked by the Python subprocess implemented in Gitman?

[jacebrowning]

You could try passing the extra parameter using the params setting in the Gitman configuration file: https://gitman.readthedocs.io/en/latest/setup/configuration/#gitman.models.source.Source--params

Unfortunately that will mean keeping your secret in Git, but let's see if it clones first, then work on alternative approaches.

[SebastianBalle]

I manage to authenticate to the repository using the system token in Azure DevOps like so.
git config --global http.extraheader "AUTHORIZATION: bearer $(System.AccessToken)". This command was executed before any Gitman commands. Alternatively I could've executed the Gitman command using the params configuration but this would imply that I use a system token for authenticating locally.

In addition I had to disable Protect access to repositories in YAML pipelines in Azure DevOps. After this I could clone the repositories in Azure Pipelines using HTTPS. In Azure DevOps it is not possible to clone other repositories using command line git if this setting is enabled. More information can found here.

Locally I simply used the Git credential manager.

This setup allows me to have the same Gitman configuration file for both local development and automated builds in Azure DevOps using HTTPS.

[jacebrowning]

@SebastianBalle thanks for sharing!

Would that be worth adding to the setup docs or do you think it's mostly specific to your situation?

[SebastianBalle]

@jacebrowning I think this would be valuable to add if you are considering having some documentation on CI implementations. The documentation only describes the local development which, I would say, is only half of the work.

You could of course simply redirect the user to the official documentation page from the different CI providers but it would be cool if you could have a small compact guide for how to implement the tool in the different systems.

I can help draft such a documentation for Azure DevOps at least.