Problem with bearer authorization with Azure DevOps on-premise

[mlbors]

Hi!

I decided to open this issue after I knocked my head against the wall for hours. I can't figure out if it is a problem related to Composer, or a problem with something located between the computer screen and the chair 😉. So feel free to close it. However, an explanation would be very appreciated.

We are using Azure DevOps on-premise, AKA TFS, and we have troubles to download our private packages. We are currently encountering a problem when we try to build a PHP project requiring internal Composer packages hosted on our private TFS on-premise repositories on a dedicated build machine through a TFS build pipeline.

In our composer.json file, external as well as internal packages are referenced. So, we have something like so:

{
    "repositories": [
    { 
    "type": "composer", 
    "url": "https://packagist.org/" 
    },
    { 
        "type": "git", 
        "url": "https://tfsonprem.com/organisation/project/_git/repository" 
    },
    "require": {
        ...
    }
}

Because we are behind a firewall, we have to set proxy information to get external packages. We do it like so:

export http_proxy="http://proxy-url"
export https_proxy="http://proxy-url"

However, to reach TFS, we also do something like so:

export no_proxy=tfsonprem.com

With this configuration, we can reach packagist.org and our TFS server. Unfortunately, we cannot pass the authentication. We tried the following things:

Creating an auth.json file at build time, trying various combinations:

1. Specifying repository:

cat >./auth.json <<EOF
{
    "repositories": [
        {
            "type": "composer",
            "url": "https://tfsonprem.com/organisation/project/_git/repository" 
            "options":  {
                "http": {
                    "header": [
                        "AUTHORIZATION: bearer $(System.AccessToken)"
                    ]
                }
            }
        }
    ]
}
EOF

2. Specifying bearer

cat >./auth.json <<EOF
{
    "bearer": {
        "tfsonprem.com": "$(System.AccessToken)"
    }
}
EOF

3. Specifying username and password

cat >./auth.json <<EOF
{
    "http-basic": {
        "tfsonprem.com": {
            "username": "userAccount",
            "password": "userPassword"
        }
    }
}
EOF

We also tried the following thing:

php ./composer.phar config repositories.1 '{"type": "vcs","url": "https://tfsonprem.com/organisation/project/_git/repository","options":  {"http": {"header": ["AUTHORIZATION: bearer $(System.AccessToken)"]}}}'

In every case in which we used "repositories", we tried to change the type for "git", "composer" or "vcs". We also tried to use a personal access token generated by TFS instead of System.AccessToken.

Unfortunately, the result is always the same as we get the following error:

[RuntimeException]
Failed to execute git clone --mirror -- 'https://***:***@tfsonprem.com/organisation/project/_git/repository' '/.config/composer/cache/vcs/https---tfsonprem.com-team-name--git-repo-name/'  
Cloning into bare repository '/.config/composer/cache/vcs/https---tfsonprem.com-team-name--git-repo-name'...
fatal: Authentication failed for 'https://***:***@tfsonprem.com/organisation/project/_git/repository/'

Our build job is set to allow scripts to access OAuth token, and it only fails when we use Composer. We tried with Composer 2.0.13 and 1.10.22. We also tried to play around with "--prefer-dist". We also cleared Composer cache each time.

For now (and since several months), as a workaround, we do the following thing: we have another composer.json file without our internal package that we use during the build, then we put the complete composer.json back. To install our private packages, for each one, we do the following thing:

REPO_URL=https://tfsonprem.com/organisation/project/_git/repository
EXTRAHEADER="AUTHORIZATION: bearer $(System.AccessToken)"
git -c http.extraheader="$EXTRAHEADER" clone $REPO_URL

It works fine, but this is not very clean and it does not allow us to switch the version of our packages easily.

The connection between our build server and TFS is fine and we can manually do the following thing:

git clone https://PAT@tfsonprem.com/organisation/project/_git/repository
...then it asks for password

Several months ago, we wrote the following posts: https://stackoverflow.com/questions/60741882/tfs-git-clone-private-repository-through-composer-fails and https://developercommunity.visualstudio.com/content/problem/968133/tfs-git-clone-private-repository-through-composer.html. But we were not able to go any further.

I read many posts and issues on the Internet, including the following ones:

• Using "bearer" auth config with a self-hosted bitbucket instance #9100
• WIP: API for self-hosted bitbucket instance #9134
• More explicit support for authorization/bearer header as authentication method #8642
• Bearer support. #8671
• Authentication does not work with for NTLM-authorized git repositories #7084

However, I am not able to conclude what the problem is.

So, is this a problem related to Composer and if it is, what could be the problem? Does Composer support bearer authorization? Or is this a problem with NTLM authentication? Is this the expected behaviour?

If the problem is not related to Composer, any idea what could that be?

Many thanks in advance!

[Seldaek]

I think one potential issue is that bearer auth is not supported by the GitDownloader, however if you see https://***:***@tfsonprem.com including the *** there it seems to indicate it did use your authentication (perhaps that was when trying with http-basic auth setup?

Anyway.. this is a pretty exotic setup and also pretty hard for me to even reproduce this. What I would suggest is that you have a look at Private Packagist as that may help proxy your tfsonprem.com stuff in a way that is more easily consumed by Composer.. There's also an on-prem offering if that's a requirement.

Aside from that I am not entirely sure how to help you further given the info I see here.

[mlbors]

Thanks for your answer!

The error message, with https://***:*** in it, appeared in every situation (if I am not wrong, logs have been cleaned now).

However, we haven't tried to connect over SSH yet. Maybe it could work.

I have to admit, as you said, that this configuration is pretty exotic, but TFS is the master tool in our workplace and we do not upload our packages on external servers. We took a look at Private Packagist, and that could be an answer to our problem.

[mlbors]

So, after many other attempts, it seems the problem is more related to Git. If we do something like so:

git config --global credential.helper store
git config --global credential.https://tfsonprem.com.username userName
git config --global credential.https://tfsonprem.com.integrated true
git config --global credential.https://tfsonprem.com.authority NTLM
git config --global http.https://tfsonprem.com "AUTHORIZATION: bearer $(System.AccessToken)"
git config --global user.name "userName"
git config --global user.password "userPassword"

...we can download our packages from our TFS server and packagist.org.

[stof]

Well, as said before, the GitDownloader of Composer does not support Bearer authentication. Your solution is to make git itself manage the credentials instead of relying on Composer to pass them to git.