[with patch] requiring a platform package that is provided by another required package fails

[dzuelke]

First, this issue is not entirely trivial to reproduce, because everyone has different PHP versions and extensions installed. To reproduce the steps and behaviors below, please ensure that you run php -n $(which composer) instead of plainly composer, or that with composer show --platform | grep mongo, nothing shows up :)

If you're on Ubuntu or something, php -n will also prevent loading ext-phar, so nothing will work. In that case, just make sure you have no ext-mongo, and run composer directly instead of php -n $(which composer).

Let's say your code needs ext-mongo (via Doctrine ODM, for instance), but that's not available on PHP 7, so you want to use https://github.com/alcaeus/mongo-php-adapter, which in its composer.json says "provide": { "ext-mongo": "1.6.12" }, and which is a wrapper around ext-mongodb to provide an interface that's compatible with the old ext-mongo.

Start with a basic composer.json, to mock the existence of ext-mongodb (again, you do not want ext-mongo installed when following along, so always run with php -n later):

$ composer init -n
$ composer config platform.php "7.0.4"
$ composer config platform.ext-mhash "7.0.4"
$ composer config platform.ext-mongodb "1.1.3"

I will use the composer-provide branch of https://github.com/alcaeus/mongo-php-adapter explicitly because the master branch has changed between using provide and replace a few times recently. There is also a composer-replace branch that uses replace instead of provide for ext-mongo.

Remember, php -n will prevent ini loads, so if you have ext-mongo locally, it should not get loaded:

$ php -n $(which composer) require alcaeus/mongo-php-adapter:dev-composer-provide
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Installing mongodb/mongodb (1.0.1)
    Loading from cache

  - Installing alcaeus/mongo-php-adapter (dev-composer-provide 698d301)
    Cloning 698d3012c306af299491aa166dc9aa29a4bce5b8

Writing lock file
Generating autoload files

Now there is a package that provides "ext-mongo" installed, and that info is in the lock file. Which works as expected if you now require that extension:

$ php -n $(which composer) require ext-mongo:*
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Nothing to install or update
Writing lock file
Generating autoload files

A subsequent composer update also works. Great, so let's try a case that fails - having the requirements already in place, but no lock file:

$ rm -rf composer.lock vendor
$ php -n $(which composer) update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested PHP extension ext-mongo * is missing from your system. Install or enable PHP's mongo extension.

Weird, right? The same happens if you require both in one go obviously:

$ php -n $(which composer) remove ext-mongo
$ php -n $(which composer) remove alcaeus/mongo-php-adapter

$ php -n $(which composer) require alcaeus/mongo-php-adapter:dev-composer-provide ext-mongo:*
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested PHP extension ext-mongo * is missing from your system. Install or enable PHP's mongo extension.

Installation failed, reverting ./composer.json to its original content.

You could of course also manually put both packages into composer.json and run composer update, same effect (that's actually what happened when we removed lock file and vendor dir earlier and ran an update).

For the steps above, instead of requiring ext-mongo, you can also use doctrine/mongodb, which in turn requires ext-mongo. Same behavior.

There is a workaround: if you put the "alcaeus/mongo-php-adapter" package into a "package" repo inside composer.json, things work. Take this composer.json:

{
    "repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/alcaeus/mongo-php-adapter"
        }
    ],
    "require": {
        "alcaeus/mongo-php-adapter": "dev-composer-provide",
        "doctrine/mongodb": "^1.2"
    },
    "config": {
        "platform": {
            "php": "7.0.4",
            "ext-mhash": "7.0.4",
            "ext-mongodb": "1.1.3"
        }
    }
}

That works:

$ rm -rf composer.lock vendor
$ php -n $(which composer) update
Loading composer repositories with package information
Updating dependencies (including require-dev)     
…                         

This pointed to a problem with information from Packagist, and indeed, it's due to behavior in (and data available to) ComposerRepository.

There is no information from Packagist on provided platform packages such as ext-mongo, so without a lock file, the info that one of the other required packages provides ext-mongo is apparently not there, and that's why things fail.

Here is the fix:

diff --git a/src/Composer/Repository/ComposerRepository.php b/src/Composer/Repository/ComposerRepository.php
index 5bd4e8a..91d6d58 100644
--- a/src/Composer/Repository/ComposerRepository.php
+++ b/src/Composer/Repository/ComposerRepository.php
@@ -371,7 +371,9 @@ public function whatProvides(Pool $pool, $name)
                         // override provider with its alias so it can be expanded in the if block above
                         $this->providersByUid[$version['uid']] = $package;
                     } else {
-                        $this->providers[$name][$version['uid']] = $package;
+                        foreach($package->getNames() as $nameName) {
+                            $this->providers[$nameName][$version['uid']] = $package;
+                        }
                         $this->providersByUid[$version['uid']] = $package;
                     }

However, that feels like a band-aid treating a symptom of the real underlying issue, which is that Packagist does not have a list of provider packages for platform packages:

$ ls ~/.composer/cache/repo/https---packagist.org/provider-ext*
ls: .composer/cache/repo/https---packagist.org/provider-ext*: No such file or directory

/cc @alcaeus @holtkamp @stof

[dzuelke]

It's important to understand that in the case of a file like .composer/cache/repo/https---packagist.org/provider-psr$log-implementation.json, the "provider" part in the filename is not related to the "provide" feature, but instead a way of "real Composer" repositories (such as Packagist) to split up their data into smaller chunks.

However, as (maybe an unintentional?) a side effect, this allows dependencies on packages provided by others.

Check out ComposerRepository::whatProvides(). That basically loads .composer/cache/repo/https---packagist.org/provider-psr$log-implementation.json containing a list of all packages that provide psr/log-implementation, so if someone/something requires that package, and then a list of all packages providing that package are available to the solver, and that's why the whole thing works.

At least that's how I understand it. @Seldaek just told me "yeah this might all be accidental" ;)

[curry684]

I think the patch is fine and might as well be a PR.

[stof]

Check out ComposerRepository::whatProvides(). That basically loads .composer/cache/repo/https---packagist.org/provider-psr$log-implementation.json containing a list of all packages that provide psr/log-implementation, so if someone/something requires that package, and then a list of all packages providing that package are available to the solver, and that's why the whole thing works.

The fact that this file contains all packages providing it is necessary. Otherwise, composer would not be able to look for psr/log-implementation if it has not loaded the other packages already (and having to know who provides the package defeats the purpose of the feature). This would make the bugfix dependant on the order in which the loading of the alcaeus/mongo-php-adapter metadata and the call to ->whatProvides('ext-mongo') happen.

I think the right fix would be for Packagist to dump such files for any package referenced in provide and replace, even when they are platform packages, and then for composer to remove the code bypassing the search for platform packages.
The fact that registering a VCS repository fixes the issue is a good sign that this is the root cause: the VCS repository is always loading all the metadata instead of relying on server-side computation of provider names (something necessary for Packagist, due to the amount of metadata available in its repository)

[dzuelke]

Yeah I agree that this should be fixed on the Packagist side (I pointed that out at the end of my original report).

My second comment was just to point out that the "provider-…" files are not specifically for the "provide" feature, but for the "providers" functionality of Packagist type repos. In case anyone confused the two ;)

[stof]

@dzuelke would you mind opening a PR on the packagist repo to provide a fix ?

[stof]

My second comment was just to point out that the "provider-…" files are not specifically for the "provide" feature, but for the "providers" functionality of Packagist type repos. In case anyone confused the two

Except that these features are actually related. The way the repository protocol works is that the provider file for psr/log-implementation contains metadata of all packages providing this package on Packagist (be it because of their name or a provide rule).
The issue is that Packagist will not dump properly some of these provider files, because of https://github.com/composer/packagist/blob/e3cc5b3d6fec1d78d0088b2b9e33828801c0a583/src/Packagist/WebBundle/Package/SymlinkDumper.php#L229 which won't match for ext-mongo for instance.

[dzuelke]

The term "provides" in a Composer type repository refers to "packages provided by that repo", not "packages provided by other packages", @stof. It just happens that this feature is used for the provide keyword functionality as well. At least that's what the docs at https://getcomposer.org/doc/05-repositories.md#provider-includes-and-providers-url seem to indicate; correct me if I am wrong, @Seldaek.

[Seldaek]

I just don't think we should fix this at the packagist level right now, because it'll create even more legacy to remove later on. We want to clean this up in 2.0 and fix the providers handling so that the files on packagist do not have to include all providers (which would shrink some of them quite substantially). /cc @naderman

[stof]

@Seldaek but then, how would composer know which files it should load to find psr/log-implementation ?

And as I said, fixing the issue on Packagist is just a matter of updating the regex above so that it does not exclude package names like ext-mbstring

[Seldaek]

Composer does not and can not know which package to load for psr/log-implementation.. You should require an implementation, and that implementation providing psr/log-implementatoin should be enough for Composer to figure things out. Right now that last part fails if we remove the providers on packagist, but that's what we want to fix. The ext-mbstring issue in practice isn't an actual problem right now except for @dzuelke but AFAIK he solved it, so let's please not waste time discussing this and create more legacy for something that will be removed anyway.

[stof]

@Seldaek the patch provided by @dzuelke depends on the order in which packages are loaded. So it does not fix the issue fully (and would lead to WTF debugging where changing the order of requirements would magically make things break or work)

[Seldaek]

Yeah I never suggested we fix it with that patch for 2.0. There's a bigger refactoring planned that would fix this.

[stof]

OK, so which fix should we use for Composer 1.x ? 2.0 is not yet a thing, and this issue affects any package providing a polyfill for an extension. So this affects the ext-mongo polyfill mentioned by @dzuelke above (which is not solved as the only way to fix it he found was to register the package repository explicitly rather relying on packagist, which is not something easy for people affected by this issue) but would also affect the symfony polyfill once they tell composer that they are polyfill (see symfony/polyfill#54) instead of forcing any library to stop requiring ext-mbstring explicitly when they want to be compatible with these polyfills.

[dzuelke]

It's also not an issue for me mind you, since deploys to Heroku require a composer.lock. It's just that someone stumbled over it when deploying to Heroku, and I decided to investigate.

It's an issue for any user of a polyfill like the ext-mongo or Symfony one, since you can't composer update without an error if there is no composer.lock present.

[alcaeus]

It's an issue for me, since I have to run around telling people that they need to add provide: { "ext-mongo": "..." } in their composer.json at root level in order to get this to work. Even if it's a workaround, I'd appreciate a fix here.

[robfrawley]

What is the status of this? Trying to use any library that polyfills extension behavior is impossibly difficult and complex to get right. It shouldn't be. Correct me if I am wrong, but it looks like this issue is continually being pushed to the next release...but this should be a "blocker" as it quite literally describes a broken implementation for a feature that is documented involving the most important aspect of Composer: dependency resolution.

[Aeolun]

Any progress on this? 😢

[naderman]

This seems to be caused at least partially by #6753