Add option to force usage of HTTP basic auth for HTTP connections to Git

[jannfis]

Problem statement

Currently, Argo CD does not work with Git repositories hosted on Azure DevOps Server on-premise installations and accessed via HTTPS using a personal access token. When trying to connect such a repository from Argo CD, the authentication will always fail.

After some investigation using a trial installation of Azure DevOps Server on-premise, I came to the conclusion that Git refuses to perform HTTP basic authentication against Azure DevOps. The root cause for this is that Git is using libcurl under the hood, and configures [1] it to "automatically select the one it finds most secure" [2]. The on-premise installation of Azure DevOps provides multiple HTTP authentication schemes for requests to its Git server, namely NTLM, SPNEGO and HTTP basic authentication.

In this scenario, Git will pick NTLM as the authentication scheme for the HTTP request on negotation, but Azure DevOps is not able to authenticate a personal access token using NTLM as it expects the PAT to be sent encoded as HTTP basic auth request. The NTLM authentication will only work with Windows users, and only by using a username and password combination.

Microsoft's official documentation [3] suggests to users that want to use PAT to explicitly pass the basic auth HTTP header when executing Git, thereby bypassing Git's authentication scheme negotiation. This works because authentication negotiation is only performed when the server returns a 401 error on the first (unauthenticated) request, and along with the response the server sends the list of supported authentication methods. If the basic auth HTTP header is passed as part of this initial request, the server will never return a 401 but happily accept the submitted credentials.

There's some ongoing problems with Azure DevOps Server within the community:

• Use Azure DevOps Git Artefacts with PAT #6289
• Unable to create application from a connected Azure DevOps server repository #6362

Possible solutions

I see the following options to make Argo CD work with Git repositories hosted on Azure DevOps Server and accessed via HTTPS:

1. Submit a feature PR (or feature request) to Git that allows to override the preferred usage of HTTP authentication scheme
2. Disable the NTLM authentication provider on the Azure DevOps server
3. Add an option to Argo CD to send the HTTP basic auth header with each request when connecting to Azure DevOps repository

Option 1 would IMHO be the correct fix for this issue and also definitely a fun contribution. However, it's uncertain whether such a change would be accepted by the Git project. And even when such a contribution would be accepted and eventually end up in a Git release, it would also be uncertain when a version containing this feature would land in in the distribution used by Argo CD's container images.

Option 2 is not supported by Microsoft, and would possibly lock-out legitimate Azure DevOps Server users from the repositories on the server, if they chose to authenticate via username and password (thereby requiring NTLM). Worse yet, their SSH implementation lags behind a couple of years and doesn't work with more recent OpenSSH versions out-of-the-box. Making Azure DevOps work with Argo CD via SSH would require enabling deprecated ciphers in the SSH client [4], which is also considered pretty insecure. They are working on a fix [5], but the issue is open since 2018 [sic].

Option 3 would be a implementing a change to Argo CD, and could be implemented as an option on the repository configuration, e.g. "Force HTTP basic auth for this connection". However, passing the credentials as a HTTP header on the command line is insecure - not only could a malicious user snoop on the process table to retrieve the credentials, the credentials could end up in some log files if not implemented carefully enough. However, this seems to be the only viable option to have Argo CD and Azure DevOps Server play nicely together.

Proposed change to Argo CD

I propose that a new option for repository configuration should be introduced, which allows to force the use of HTTP basic auth instead of having Git/libcurl negotiate the preferred authentication method.

In declarative configuration, this option could be named forceBasicAuth and accept a boolean value, with the default being false.

In the UI's repository configuration page, there should be a corresponding checkbox with a label reading Force HTTP basic auth for this connection (insecure) or something around these lines.

In the CLI's argocd repo commands, a new switch to enable or disable this setting should exist. The switch could be named --force-basic-auth, and also accept a boolean value.

If this option is set to true for a repository, then the HTTP basic auth string will be computed by Argo CD (i.e. the base64 encoded value of the <username>:<password> string) and the Git client's execution environment modified to include the additional header on the command line with each call, e.g. git -c http.extraHeader="Authorization: Basic ${computed_string}" clone ...

Since command execution is logged in Argo CD, we would also have to adapt the log redactor to redact the additional header on the command line before it's being logged, or when an error message is passed back to the user.

References

[1] https://github.com/git/git/blob/a38d39a4c50d1275833aba54c4dbdfce9e2e9ca1/http.c#L838
[2] https://curl.se/libcurl/c/CURLOPT_HTTPAUTH.html
[3] https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Linux#tabpanel_1_Linux
[4] https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops#q-what-do-i-do-if-im-still-prompted-for-my-password-and-git_ssh_commandssh--v-git-fetch-shows-no-mutual-signature-algorithm
[5] https://developercommunity.visualstudio.com/t/support-non-rsa-keys-for-ssh-authentication/365980

[jannfis]

cc @crenshaw-dev @34fathombelow

[crenshaw-dev]

Two thoughts:

1. Can we go ahead and file an enhancement request with git? Even if they/we can't move quickly enough to avoid Argo CD changes, we might help out any other projects which hit this problem.
2. Can we avoid passing creds on the CLI by setting the headers some other way? For example, could we write the http.extraHeader to a git config file, or could we use an mitm proxy to add the header?

[jannfis]

Good thinking.

After reading the manual page for git(1), I found the following option:

       --config-env=<name>=<envvar>
           Like -c <name>=<value>, give configuration variable <name> a value, where <envvar> is the name of an environment variable from which to retrieve the value. Unlike -c there is no shortcut for
           directly setting the value to an empty string, instead the environment variable itself must be set to the empty string. It is an error if the <envvar> does not exist in the environment.
           <envvar> may not contain an equals sign to avoid ambiguity with <name> containing one.

So, we might be able to use it as follows (not tested yet, though):

• Set environment ARGOCD_GIT_BASIC_AUTH to base64 encoded value of "Authorization: Basic $(base64 of <username>:<password>)" for the Git client's execution context
• Exec git with --config-env=http.extraHeader=ARGOCD_GIT_BASIC_AUTH

This way, no credentials would be passed on the command line but instead they would be taken from the environment.

Regardless of that, I agree that we should initiate the discussion with the Git project. They do not have a (public) issue tracker as it seems, but they do have a mailing list [1] to bring up such items.

[1] https://git-scm.com/community

[crenshaw-dev]

Yep, I like that option a lot better, and would feel comfortable using it while we wait for potential movement from the git folks.

Would you be willing to start the conversation on their mailing list?

[jannfis]

Would you be willing to start the conversation on their mailing list?

Yes, will do that.