Composer support: private packages

[swissspidy]

What would you like Renovate to be able to do?

Since Composer support is underway (see #1357), it would be great if it would also support private package repositories, e.g. packages that aren't on Packagist or public GitHub repositories. Without that, Renovate Bot won't be able to update all PHP dependencies when Composer can't access these packages.

I first raised this at renovatebot/config-help#73 (comment)

Describe the solution you'd like

Private repositories are usually protected using HTTP authentication, for which the credentials can be stored in ~/.composer/auth.json (see https://getcomposer.org/doc/articles/http-basic-authentication.md).

According to #1357 (comment) Renovate already creates a auth.json file for Composer.

What is now needed is a way to pass credentials for a private composer repository to Renovate that it then can add to auth.json.

[rarkins]

Currently, Renovate will set a token in a temporary auth.json for github.com only, if Renovate already has a token for it. Hence composer will be able to access the same list of repositories on github.com that the bot itself has been granted access to.

The next steps are:

1. Configuring a custom packagist registry URL
2. Setting a valid token in auth.json for that registry

@swissspidy where/when is the custom registry URL defined in your project? e.g. is it committed as part of composer.json ?

[swissspidy]

Yes, the custom registry is set in the composer.json file of the project in the repositories array.

Here's a simplified example:

composer.json

{
  "name": "acme/my-awesome-site",
  "description": "My WordPress site",
  "license": "GPL-2.0-or-later",
  "config": {
    "vendor-dir": "wordpress/vendor",
    "sort-packages": true
  },
  "repositories": [
    {
      "type": "composer",
      "url": "https://wpackagist.org"
    },
    {
      "type": "composer",
      "url": "https://composer.acme.com"
    },
    {
      "type": "git",
      "url": "git@github.com:acme/some-private-repository.git"
    }
  ],
  "require": {
    "php": ">=7.2",
    "acme/some-private-repository": "dev-master",
    "acme/foo-package": "^1.0",
    "acme/bar-package": "^1.0"
  },
  "minimum-stability": "dev",
  "prefer-stable": true
}

Explanation:

• https://wpackagist.org` is a public Composer repository for WordPress plugins
• https://composer.acme.com is the hidden repository
• git@github.com:acme/some-private-repository.git is a hidden GitHub repository
  Renovate will likely already have access to it as per your explanation
• The default, https://repo.packagist.org/, does not need to be listed

auth.json

{
  "http-basic": {
    "composer.acme.com": {
      "username": "johndoe",
      "password": "..."
    }
  },
  "github-oauth": {
    "github.com": "...."
  },
  "bitbucket-oauth": {
    "bitbucket.org": {

    }
  }
}

While some Composer repositories (like Packagist) just point to GitHub repositories, some use projects like Satis or Release Belt where simple ZIP files can be exposed as Composer packages.

[Jamesking56]

Its been a couple of months on this issue, I've now hit this issue with my own private satis repository.

I need to specify HTTP basic auth credentials for mine, does Renovate Docker take in an auth.json file?

[rarkins]

Can you specify exactly what the scenario is? Eg self hosted, GitLab, and what type of private composer host?

[Jamesking56]

I have a Satis repository for private composer packages which uses HTTP Basic Authentication. Composer would normally popup and ask for credentials but obviously instead I get an error because we are non-interactive.

I'm using the Docker repo and GitLab CI for this. So I just need some way I can pass in the authentication credentials OR alternatively provide the auth.json file that the container can copy into Composer's $COMPOSER_HOME.

[rarkins]

There will be two parts to this:

1. Making sure we do the lookup with auth using our own code
2. Making sure we write appropriate credentials to auth.json for composer

In fact I think there's one step missing at the start:

0. Ensure we look up from the correct host and not packagist (Support custom register for composer #2574)

For (0), are you looking up all packages from Satis, or just some? How is it specified in composer.json?

For (1), the credentials would need to be configured into hostRules and then the packagist datasource code extended to look up hostRules and use any auth found.

For (2), we already write auth.json with GitHub credentials next to composer.json(not COMPOSER_HOME) and it seems to work fine, so we just need to add logic to add the additional auth. I'm not sure whether to (a) add every single hostRules auth we find, or (b) add a new config option called hostAuth where you can specify which hosts we add auth for, but this would mean configuring each host in two places - hostRules and hostAuth. Maybe a better solution is that we look through all hostRules and use every single one that has http basic authentication.

[Jamesking56]

@rarkins

For 0, I only have 1 package which is from my private repo, the rest are from packagist. I specify the repository in my project's composer.json like so:

"repositories": [
    {
        "type": "composer",
        "url": "https://composer.momentum.studio"
    }
],

My repo above requires HTTP Basic Auth which what I need to give to Renovate. I think composer tries custom repositories first and then tries packagist if the package doesn't exist in the private repo.

Yes you can add auth.json per project by adding it next to composer.json.

[rarkins]

In that case:

• we should read and try that url automatically in our packagist data source (already a WIP)
• you will need to configure a hostRule for that host with auth set to username:password
• before updating composer.lock we should scan the list of “repositories” in composer.json and check if any have a hostRule with auth and if so then write it to auth.json, in addition to any auth we find for github.com

[Jamesking56]

@rarkins I don't think I'd like to commit secrets though, is there a way to specify hostRules on the CLI or using Environment Variables?

[rarkins]

@Jamesking56 you can supply --host-rules="[]" as a JSON-stringified CLI input. In theory we could support the same for env too but it's not enabled/tested yet. (ref)

For the hosted app, users would encrypt the field with Renovate's public key so that it's unreadable to anyone but the bot.

[renovate-bot]

🎉 This issue has been resolved in version 13.101.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[Jamesking56]

Thanks for this @rarkins , what is the documentation on how to configure the credentials for the Docker app? Is it specifying --hostRules?

[rarkins]

I forgot to add that to docs and will do. Configure it like:

hostRules: [{
endpoint: “https://your.endpoint”,
username: “foo”,
password: “bar”,
}]

[viceice]

@marijoo Please open a new issue in config-help repo for support and don't pollute old closed issues.