composer requireing a provided package fails

[dzuelke]

There is a slightly expanded example further down that explains the why of this, but let's start with a simple case.

My composer.json:

{
    "require": {
        "heroku-sys/php": "8.1.*"
    },
    "repositories": [
        {
            "packagist": false
        },
        {
            "type": "package",
            "package": [
                {
                    "type": "metapackage",
                    "name": "heroku-sys/php",
                    "version": "8.1.1",
                    "require": {},
                    "replace": {},
                    "provide": {},
                    "conflict": {}
                },
                {
                    "type": "metapackage",
                    "name": "heroku-sys/ext-mbstring",
                    "version": "8.1.1",
                    "require": {
                        "heroku-sys/php": "8.1.1"
                    },
                    "replace": {},
                    "provide": {
                        "heroku-sys/ext-mbstring.native": "8.1.1"
                    },
                    "conflict": {}
                }
            ]
        }
    ]
}

This installs fine:

% composer install
No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking heroku-sys/php (8.1.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing heroku-sys/php (8.1.1)
Generating autoload files

Then I want that heroku-sys/ext-mbstring package, but that might be provided by a polyfill, so I want to try and install that package in a variant that provides heroku-sys/ext-mbstring.native, without changing any of the already locked dependencies, so I composer require them together, which gives an error:

% composer require heroku-sys/ext-mbstring.native heroku-sys/ext-mbstring 
                                                                               
  [InvalidArgumentException]                                                   
  Could not find a matching version of package heroku-sys/ext-mbstring.native  
  . Check the package spelling, your version constraint and that the package   
  is available in a stability which matches your minimum-stability (stable).   
                                                                               
…

However, changing composer.json as follows:

    "require": {
        "heroku-sys/php": "8.1.*",
        "heroku-sys/ext-mbstring": "*",
        "heroku-sys/ext-mbstring.native": "*"
    },

And then confining composer update to just those two packages works fine:

% composer update heroku-sys/ext-mbstring heroku-sys/ext-mbstring.native 
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking heroku-sys/ext-mbstring (8.1.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing heroku-sys/ext-mbstring (8.1.1)
Generating autoload files

Am I understanding composer require wrong, or is this a bug? :)

Second example

Take a fresh checkout of Composer itself. One of its dependencies is symfony/console, which provides the virtual package psr/log-implementation.

It's possible to add psr/log-implementation to composer.json and run composer update (or composer update psr/log-implementation), but composer require psr/log-implementation fails:

% composer require "psr/log-implementation:*"
                                                  
  [InvalidArgumentException]                      
  Could not find package psr/log-implementation.  
                                                  
  Did you mean this?                              
      xxnoobmanxx/psrlogger                       
                                                  
…

The reason is in InitCommand::determineRequirements() (RequireCommand extends InitCommand), where $this->findBestVersionAndNameForPackage() is called even when a version requirement is given.

Removing the else block in that method solves it:

% bin/composer require "psr/log-implementation:*"
./composer.json has been updated
Running composer update psr/log-implementation
Loading composer repositories with package information
Updating dependencies
Nothing to modify in lock file
Writing lock file
Installing dependencies from lock file (including require-dev)
Nothing to install, update or remove
Generating autoload files
22 packages you are using are looking for funding.
Use the `composer fund` command to find out more!

Why did I encounter this?

Heroku translates a project's composer.lock to a "platform" composer.json and uses that to install PHP and its dependencies. The README on it has all the details, but basically, if this operation succeeds, then the following composer install is guaranteed to have its platform dependencies satisfied.

There is an edge case: if a userland package declares a PHP extension as provided (a "polyfill"), then a potentially native version of that extension will not be selected by the solver, because the polyfill is explicitly required and satisfies this requirement.

Imagine this example:

• Package PHP is available in versions 7.1, 7.2, 7.3, 7.4, 8.0, 8.1.
• Package ext-mcrypt is available for PHP 7.1 only - it was removed in PHP 7.2.
• composer.json requirement for PHP is unbounded enough to allow the installation of 7.1 or later.
• composer.json requires ext-mcrypt, but also phpseclib/mcrypt_compat (which is a polyfill that provides ext-mcrypt), so that it's all installable on PHP versions 7.2 and later:

{
    "require": {
        "heroku-sys/php": "^7.1 | ^8.0",
        "heroku-sys/ext-mcrypt": "*",
        "phpseclib/mcrypt_compat": "^2.0"
    },
    "repositories": [
        {
            "packagist": false
        },
        {
            "type": "package",
            "package": [
                {
                    "type": "metapackage",
                    "name": "phpseclib/mcrypt_compat",
                    "version": "2.0.1",
                    "require": {
                        "heroku-sys/php": ">=5.6.1"
                    },
                    "replace": {},
                    "provide": {
                        "heroku-sys/ext-mcrypt": "5.6.40"
                    },
                    "conflict": {}
                },
                {
                    "type": "metapackage",
                    "name": "heroku-sys/php",
                    "version": "7.1.0",
                    "require": {},
                    "replace": {},
                    "provide": {},
                    "conflict": {}
                },
                {
                    "type": "metapackage",
                    "name": "heroku-sys/ext-mcrypt",
                    "version": "7.1.0",
                    "require": {
                        "heroku-sys/php": "7.1.0"
                    },
                    "replace": {},
                    "provide": {
                        "heroku-sys/ext-mcrypt.native": "7.1.0"
                    },
                    "conflict": {}
                },
                {
                    "type": "metapackage",
                    "name": "heroku-sys/php",
                    "version": "7.2.0",
                    "require": {},
                    "replace": {},
                    "provide": {},
                    "conflict": {}
                },
                {
                    "type": "metapackage",
                    "name": "heroku-sys/php",
                    "version": "7.3.0",
                    "require": {},
                    "replace": {},
                    "provide": {},
                    "conflict": {}
                },
                {
                    "type": "metapackage",
                    "name": "heroku-sys/php",
                    "version": "7.4.0",
                    "require": {},
                    "replace": {},
                    "provide": {},
                    "conflict": {}
                },
                {
                    "type": "metapackage",
                    "name": "heroku-sys/php",
                    "version": "8.0.0",
                    "require": {},
                    "replace": {},
                    "provide": {},
                    "conflict": {}
                },
                {
                    "type": "metapackage",
                    "name": "heroku-sys/php",
                    "version": "8.1.0",
                    "require": {},
                    "replace": {},
                    "provide": {},
                    "conflict": {}
                }
            ]
        }
    ]
}

Installing this gives PHP 8.1, as expected:

% composer install
No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
Loading composer repositories with package information
Updating dependencies
Lock file operations: 2 installs, 0 updates, 0 removals
  - Locking heroku-sys/php (8.1.0)
  - Locking phpseclib/mcrypt_compat (2.0.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 2 installs, 0 updates, 0 removals
  - Installing heroku-sys/php (8.1.0)
  - Installing phpseclib/mcrypt_compat (2.0.1)
Generating autoload files

But we want to try installing a native version of ext-mcrypt. We're a dumb piece of code that goes through all the userland provide declarations for extensions in order, and attempts to install a native variant. If that works, great, the polyfill will now fall back to faster, native code. If not, no harm done.

We have no idea that ext-mcrypt is old and not available on modern PHP versions anymore. We're generic code. We're also not in the business of solving dependency graphs. We just want to try and install the native variant without affecting the already installed set of packages.

So, since composer require doesn't work as reported in this issue, we instead add heroku-sys/ext-mcrypt.native to composer.json:

    "require": {
        "heroku-sys/php": "^7.1 | ^8.0",
        "heroku-sys/ext-mcrypt": "*",
        "heroku-sys/ext-mcrypt.native": "*",
        "phpseclib/mcrypt_compat": "^2.0"
    },

If we now simply run composer update, it will downgrade PHP to 7.1:

% composer update
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 1 update, 0 removals
  - Locking heroku-sys/ext-mcrypt (7.1.0)
  - Downgrading heroku-sys/php (8.1.0 => 7.1.0)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 1 update, 0 removals
  - Downgrading heroku-sys/php (8.1.0 => 7.1.0)
  - Installing heroku-sys/ext-mcrypt (7.1.0)
Generating autoload files

But as a user, you don't want that! You want the latest possible versions of everything, and the polyfill makes sure that your stuff works even though the extension isn't available for a more recent PHP.

Whereas composer update heroku-sys/ext-mcrypt heroku-sys/ext-mcrypt.native fails, as it should in this case:

% composer update heroku-sys/ext-mcrypt heroku-sys/ext-mcrypt.native
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires heroku-sys/ext-mcrypt.native * -> satisfiable by heroku-sys/ext-mcrypt[7.1.0].
    - heroku-sys/ext-mcrypt 7.1.0 requires heroku-sys/php 7.1.0 -> found heroku-sys/php[7.1.0] but the package is fixed to 8.1.0 (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.

Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.

This is great! We don't want to downgrade your dependencies in this case and just continue; but for other extensions, this might work - think all the symfony/polyfill packages for ext-mbstring and so forth, where this operation would succeed.

I suppose changing composer.json programmatically and running composer update is a viable workaround for the moment, but it would still be great if a simpler composer require were possible:

% composer require heroku-sys/ext-mcrypt heroku-sys/ext-mcrypt.native
Using version ^7.1 for heroku-sys/ext-mcrypt
                                                                               
  [InvalidArgumentException]                                                   
  Could not find a matching version of package heroku-sys/ext-mcrypt.native.   
  Check the package spelling, your version constraint and that the package is  
   available in a stability which matches your minimum-stability (stable).     
                                                                               
…

[stof]

Well, you say that you expect a failure when running a partial update, but you complain about an error when using require to do the same. So I don't understand your report. What do you actually expect to happen during your composer require ?

If your need is to get the solver report with the problem output, instead of a message saying Could not find a matching version of package heroku-sys/ext-mcrypt.native, a solution might be to specify an explicit version in your requirement instead of asking composer to guess the constraint (i.e. composer require heroku-sys/ext-mcrypt:* heroku-sys/ext-mcrypt.native:*

[dzuelke]

@stof the second example is just an elaboration on this setup, and gives an example where the install would fail (constraining versions etc does not change anything).

Check the first simple example again; there, the composer require should succeed, but it gives an error, while the constrained composer update works as expected.

[dzuelke]

Is the reason maybe that composer require treats the two listed packages separately and tries to find them, rather than as a "pool"?

[dzuelke]

Yup that's it, RequireCommand extends InitCommand and uses its determineRequirements for each given package separately:

composer/src/Composer/Command/InitCommand.php

Lines 517 to 550 in 3ae1111

	final protected function determineRequirements(InputInterface $input, OutputInterface $output, $requires = array(), PlatformRepository $platformRepo = null, $preferredStability = 'stable', $checkProvidedVersions = true, $fixed = false)
	{
	if ($requires) {
	$requires = $this->normalizeRequirements($requires);
	$result = array();
	$io = $this->getIO();
	foreach ($requires as $requirement) {
	if (!isset($requirement['version'])) {
	// determine the best version automatically
	list($name, $version) = $this->findBestVersionAndNameForPackage($input, $requirement['name'], $platformRepo, $preferredStability, null, null, $fixed);
	$requirement['version'] = $version;
	// replace package name from packagist.org
	$requirement['name'] = $name;
	$io->writeError(sprintf(
	'Using version <info>%s</info> for <info>%s</info>',
	$requirement['version'],
	$requirement['name']
	));
	} else {
	// check that the specified version/constraint exists before we proceed
	list($name) = $this->findBestVersionAndNameForPackage($input, $requirement['name'], $platformRepo, $preferredStability, $checkProvidedVersions ? $requirement['version'] : null, 'dev', $fixed);
	// replace package name from packagist.org
	$requirement['name'] = $name;
	}
	$result[] = $requirement['name'] . ' ' . $requirement['version'];
	}
	return $result;
	}

Not trivial to solve then I guess 😬

[Seldaek]

Indeed, you can workaround by bypassing the version guessing perhaps (i.e. specifically providing a constraint composer require foo/bar:* baz/qux:*, but I think the else at line 540 in your codeblock above will still screw with you.

I'm not so sure what else to offer here.

[dzuelke]

A constraint doesn't help; I don't think the code looks at the provide declarations.

I wonder... is it really necessary for this code to check, this early, if a package exists? At least in the case where an explicit version requirement is given, why check if it exists... that's done again later by the solver, no?

What I'm trying to do is attempt the installation in an atomic manner; that's why I'd like to use composer require. With the alternative of updating composer.json and running a composer update <packages…>, the change to composer.json then has to be reverted if the update wasn't successful.

Unfortunately, composer require --no-update also performs the check, so changing composer.json "by hand" (in code) is, I guess, my last hope?

[dzuelke]

I feel like this should, in principle, be possible. The composer require could be for any virtual package that's provided by another one; this would then also fail. I just realized that that's the even simpler description for this.

Just skipping the else block above entirely solves it, and the install succeeds.

Take just Composer itself as an example project... shouldn't a composer require "psr/log-implementation:*" succeed? It's "installable"; all its dependencies are satisfied. It's not entirely unreasonable to want to add that to the root composer.json, right?

(will update title and a bit of the description to reflect this)

[dzuelke]

It looks like that else was introduced in b4df2c9, but I'm afraid I don't quite understand how it relates to the fixing of #6821.

@Seldaek do you remember? And is that else even still needed to address that linked issue in 2.0?

[dzuelke]

Looked into this again. I think that else can now be removed (or at least be skipped for RequireCommand) because the case that b4df2c9 fixed (for #6821 and #3464) should now be fully covered by 759a3a9 (for #10118), no? /cc @nicolas-grekas @Seldaek