BC break in 2.2.8

[andrewbelcher]

Reproducible with Drupal's composer template: https://github.com/drupal/recommended-project/blob/9.1.x/composer.json with the addition of acquia BLT:

{
    "name": "drupal/recommended-project",
    "description": "Project template for Drupal 9 projects with a relocated document root",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "homepage": "https://www.drupal.org/project/drupal",
    "support": {
        "docs": "https://www.drupal.org/docs/user_guide/en/index.html",
        "chat": "https://www.drupal.org/node/314178"
    },
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ],
    "require": {
        "acquia/blt": "^13.3",
        "composer/installers": "^1.9",
        "drupal/core-composer-scaffold": "^9.1",
        "drupal/core-project-message": "^9.1",
        "drupal/core-recommended": "^9.1"
    },
    "require-dev": {
        "drupal/core-dev": "^9.1"
    },
    "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "sort-packages": true,
        "allow-plugins": {
            "composer/installers": true,
            "drupal/core-composer-scaffold": true,
            "drupal/core-project-message": true,
            "acquia/blt": true
        }
    },
    "extra": {
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            }
        },
        "installer-paths": {
            "web/core": ["type:drupal-core"],
            "web/libraries/{$name}": ["type:drupal-library"],
            "web/modules/contrib/{$name}": ["type:drupal-module"],
            "web/profiles/contrib/{$name}": ["type:drupal-profile"],
            "web/themes/contrib/{$name}": ["type:drupal-theme"],
            "drush/Commands/contrib/{$name}": ["type:drupal-drush"],
            "web/modules/custom/{$name}": ["type:drupal-custom-module"],
            "web/profiles/custom/{$name}": ["type:drupal-custom-profile"],
            "web/themes/custom/{$name}": ["type:drupal-custom-theme"]
        },
        "drupal-core-project-message": {
            "include-keys": ["homepage", "support"],
            "post-create-project-cmd-message": [
                "<bg=blue;fg=white>                                                         </>",
                "<bg=blue;fg=white>  Congratulations, you’ve installed the Drupal codebase  </>",
                "<bg=blue;fg=white>  from the drupal/recommended-project template!          </>",
                "<bg=blue;fg=white>                                                         </>",
                "",
                "<bg=yellow;fg=black>Next steps</>:",

                "  * Install the site: https://www.drupal.org/docs/8/install",
                "  * Read the user guide: https://www.drupal.org/docs/user_guide/en/index.html",
                "  * Get support: https://www.drupal.org/support",
                "  * Get involved with the Drupal community:",
                "      https://www.drupal.org/getting-involved",
                "  * Remove the plugin that prints this message:",
                "      composer remove drupal/core-project-message"
            ]
        }
    }
}

When I run any composer command (including diagnose) I get the following:

composer diagnose
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Component/DependencyInjection/Container.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Component/DependencyInjection/PhpArrayContainer.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Component/FileCache/FileCacheFactory.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Component/Utility/Timer.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Component/Utility/Unicode.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Cache/Cache.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Cache/CacheBackendInterface.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Cache/CacheTagsChecksumInterface.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Cache/CacheTagsChecksumTrait.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Cache/CacheTagsInvalidatorInterface.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Cache/DatabaseBackend.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Cache/DatabaseCacheTagsChecksum.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Database/Connection.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Database/Database.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Database/Driver/mysql/Connection.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Database/Driver/pgsql/Connection.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Database/Driver/sqlite/Connection.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Database/Statement.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Database/StatementInterface.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/DependencyInjection/Container.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/DrupalKernel.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/DrupalKernelInterface.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Installer/InstallerRedirectTrait.php" which does not appear to be a file nor a folder
Could not scan for classes inside "/var/www/html/vendor/drupal/core/lib/Drupal/Core/Site/Settings.php" which does not appear to be a file nor a folder

                                                                                                                     
  [ErrorException]                                                                                                   
  require(/var/www/html/vendor/drupal/core/includes/bootstrap.inc): failed to open stream: No such file or director  
  y

I believe this is probably the fault of acquia/blt having:

        "installer-paths": {
            "vendor/drupal/core": [
                "type:drupal-core"
            ]
        },

However, this is only an issue in 2.2.8 released this morning. Rolling back to 2.2.7 fixes the issue. Something has changed in the way it works out the eventual install path / autoload. From the release notes, I wonder if it is related to the change to alphabetical ordering.

[Seldaek]

Can reproduce, thanks, investigating.

[Seldaek]

Seems due to d679532 but I don't yet understand why that would cause such a messy outcome.

[Seldaek]

OK so the problem is this:

acquia/blt-13.3.0.0 requires drush/drush
drush/drush requires (in require-dev only tho) composer/installers

Now due to the commit above, require-dev is not taken into account anymore, as it really should not be because those transient dev dependencies are completely ignored for all other purposes.

This changes the sorted list of plugins to be loaded from:

dealerdirect/phpcodesniffer-composer-installer-0.7.2.0
composer/installers-1.12.0.0
acquia/blt-13.3.0.0
drupal/core-composer-scaffold-9.3.7.0
drupal/core-project-message-9.3.7.0

to:

dealerdirect/phpcodesniffer-composer-installer-0.7.2.0
acquia/blt-13.3.0.0
composer/installers-1.12.0.0
drupal/core-composer-scaffold-9.3.7.0
drupal/core-project-message-9.3.7.0

So acquia/blt now loads before composer/installers, and so Composer isn't yet aware of the custom install paths at the point where it loads BLT's dependencies.

This is ultimately a problem of drupal/core not requiring composer/installers, so that even tho blt requires drupal/core, Composer cannot figure out that composer/installers is a dependency and thus should be loaded first.

I am not sure what to do here. Ideally it would be resolved by adding the appropriate require in drupal/core (and other packages that do depend on custom types from composer/installers to be installed in the right location).

Another workaround would be for blt itself to require composer/installers, but I would only do that if blt is only ever used within projects that have composer/installers installed anyway (which I guess is the case but I am not 100% sure?).

The question for me is.. can this be resolved on your end in a way that does not cause lots of pain for the community or not? If not then I guess I can rollback the change, or add some custom hack for this in Composer to keep the existing ecosystem working, but I'd obviously rather not do that if I can avoid it. #

[andrewbelcher]

Thank you for the thorough investigation and response. I agree, this should be solved at BLT's (or it's dependencies' end) if we possibly can! Unfortunately this issue broke our CI which always runs the latest composer. We've locked to 2.2.7 which will be a good enough solution for now.

An upstream fix may take some time and will require everyone to update, but given the workaround is fairly easy and the problem is the actual packages, I am also inclined to not try and build in a custom hack for this.

However, is this actually a broader issue affecting composer installers? My (possibly incorrect) understanding is that at this point, an install has already happened and composer installers has already changed the install location. From that point on, the new locations need to have the top most priority, regardless of whether something depends on composer installers? Whilst this issue is related to Drupal's requirement of a different install location, I expect this could break any root project composer.json that chooses to change a path for a nested dependency?

[Seldaek]

I expect this could break any root project composer.json that chooses to change a path for a nested dependency?

IIRC you cannot just choose that, you can only override paths for package types supported by composer/installers, and those packages generally probably should be requiring the plugin.

Some other plugins may allow you to do crazy things tho, and could break Composer in various ways for sure :)

[mikemadison13]

i think the reason BLT doesn't currently require composer installers is that the project scaffold we assume you'll use (e.g. https://github.com/acquia/drupal-recommended-project/blob/master/composer.json) typically does. So... adding composer installers to BLT seems like a bit of a hacky solution, since composer installers should be coming into the project already due to the scaffolding in use.

[mikemadison13]

having said that, we could add composer installers to blt, i guess just worry that if it's truly alphabetical, are we kicking the can down the road and will run into this in the future if an "a" named package has a similar issue?

[andrewbelcher]

So my understanding is that it is alphabetical when not determined by a dependency. So package a can also solve it by adding a dependency. It can always be solved by a dependency on composer/installers.

I think the 'can down the road' question is whether a dependency can be legitimately moved if the dependency itself doesn't depend on composer/installers. E.g.:

Package A: Doesn't have a requirement to be in a particular directory.
Package B: Depends on package A.
Project: For it's own internal reasons, requires Package A to be in a particular directory.

In the above scenario, Package B breaks unless either it or Package A adds composer/installers as a dependency, even though it doesn't care about it.

[Seldaek]

It's not alphabetical, it's first sorted by dependency order, which is why adding the dependency would solve it. Alphabetical is only for packages with the same dependency "weight".

Anyway, I agree BLT should not have to require it. drupal/core definitely should though if it is expecting to be installed in a custom path always.

But anyway I think we should probably add some way for plugins like composer/installers to identify themselves as modifying the install path, and then those should be prioritized because if they are not loaded things can go very wrong. That would also fix the issue here.

I'll get a 2.2.9 out later today or tomorrow latest with this.

[andrewbelcher]

Apologies for the cross post above!

I think your suggestion for 2.2.9 is actually the most robust solution!

[danepowell]

I've removed the configuration from BLT in the latest release:

• https://github.com/acquia/blt/releases/tag/13.4.0
• Remove installer-paths acquia/blt#4482

[Seldaek]

Alright, https://github.com/composer/composer/releases/tag/2.2.9 is out - hopefully this fixes it for good!

[andrewbelcher]

Looks grand and works a charm. Thank you for investigating and fixing so quickly!