Merging files installed to vendor by other packages does not work on initial run

[schmunk42]

I am playing around with a setup where I want to include (merge) composer.assets.json files which reside in vendor and are therefore not available on the first run.

My composer.json.

First run...

composer update -vv
    1/4:	http://packagist.org/p/provider-latest$b3160f77308b723aeb638cf56dac6f48f1ec9d91cf4aeca4e5ec93d496292aa5.json
    2/4:	http://packagist.org/p/provider-2017-07$c6c63e6fc54ee7337b1c55dfc623b61eb3da483c06a624bd317e97b98b433533.json
    3/4:	http://packagist.org/p/provider-2015$a1fedc3267d58ba9f4377eec5b7e69bb19711787f86cbff939e9eab80b247152.json
    4/4:	http://packagist.org/p/provider-2018-01$31fbf249873f6ab209c38b30cbe8851d4a4dd4d708e5503c05a749d226ce392c.json
    Finished: success: 4, skipped: 0, failure: 0, total: 4
Loading composer repositories with package information
[...]
Updating dependencies (including require-dev)
Dependency resolution completed in 0.000 seconds
Analyzed 531 packages to resolve dependencies
Analyzed 172 rules to resolve dependencies
Package operations: 9 installs, 0 updates, 0 removals
Installs: yiisoft/yii2-composer:dev-master 58382e6, wikimedia/composer-merge-plugin:v1.4.1, psr/simple-cache:dev-master 408d5ea, psr/log:dev-master 4ebe3a8, psr/http-message:dev-master f6561bf, cebe/markdown:1.1.2, yiisoft/yii2-dev:2.1.x-dev 3e25001, yiisoft/yii2-jquery:dev-feature/nam c2bf8a8, codemix/yii2-streamlog:1.2.1
  - Installing yiisoft/yii2-composer (dev-master 58382e6): Cloning 58382e62e592b7cd800ab1e9f4a3cf54f3663cce from cache
  - Installing wikimedia/composer-merge-plugin (v1.4.1): Loading from cache
 Extracting archive    REASON: Required by the root package: Install command rule (install wikimedia/composer-merge-plugin v1.4.0|install wikimedia/composer-merge-plugin v1.4.1)

  [merge-plugin] composer-merge-plugin installed
  - Installing psr/simple-cache (dev-master 408d5ea): Cloning 408d5eafb83c57f6365a3ca330ff23aa4a5fa39b from cache
  - Installing psr/log (dev-master 4ebe3a8): Cloning 4ebe3a8bf773a19edfe0a84b6585ba3d401b724d from cache
  - Installing psr/http-message (dev-master f6561bf): Cloning f6561bf28d520154e4b0ec72be95418abe6d9363 from cache
  - Installing cebe/markdown (1.1.2): Loading from cache
 Extracting archive    REASON: yiisoft/yii2-dev 2.1.x-dev requires cebe/markdown ~1.0.0 | ~1.1.0 -> satisfiable by cebe/markdown[1.0.0, 1.0.0-rc, 1.0.1, 1.0.2, 1.0.3, 1.0.x-dev, 1.1.0, 1.1.1, 1.1.2].

  - Installing yiisoft/yii2-dev (2.1.x-dev 3e25001): Cloning 3e25001a6757c8c65dabb174b3d008a266cce544 from cache
    REASON: Required by the root package: Install command rule (install yiisoft/yii2-dev 2.1.x-dev)

  - Installing yiisoft/yii2-jquery (dev-feature/nam c2bf8a8): Cloning c2bf8a85f00061a3728403bac743de66c274f5b4 from cache
    REASON: Required by the root package: Install command rule (install yiisoft/yii2-jquery dev-feature/nam)

  - Installing codemix/yii2-streamlog (1.2.1): Loading from cache
 Extracting archive    REASON: Required by the root package: Install command rule (install codemix/yii2-streamlog 1.2.0|install codemix/yii2-streamlog 1.2.1)

yiisoft/yii2-dev suggests installing yiisoft/yii2-coding-standards (you can use this package to check for code style issues when contributing to yii)
yiisoft/yii2-dev suggests installing ezyang/htmlpurifier (required at `yii\helpers\HtmlPurifier` for 'html' data format support (e.g. `yii\i18n\Formatter:asHtml()`))
Writing lock file
Generating autoload files
  [merge-plugin] Loading vendor/yiisoft/yii2-jquery/composer.assets.json...
  [merge-plugin] Merging npm-asset/jquery
  [merge-plugin] Merging npm-asset/punycode
  [merge-plugin] Running additional update to apply merge settings
    1/3:	http://packagist.org/p/provider-latest$7bd67f74c323ff6601aa1ca39c2b1fbd1bc7cc3e5cafedd8ff0e7988ccb7f647.json
    2/3:	http://packagist.org/p/provider-2017-07$00a09e1025fd22008b01aba5fba08327ccb5a3b7a4fcf1d27f25033dec2e0cf9.json
    3/3:	http://packagist.org/p/provider-2018-01$1f72d71f41de15a06d4105ec1a277fa32cc295e41b7ed9519fecf3704de7ddd6.json
    Finished: success: 3, skipped: 0, failure: 0, total: 3
  [merge-plugin] Loading vendor/yiisoft/yii2-jquery/composer.assets.json...
  [merge-plugin] Merging npm-asset/jquery
  [merge-plugin] Merging npm-asset/punycode
  [merge-plugin] Loading -dev sections of vendor/yiisoft/yii2-jquery/composer.assets.json...
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Dependency resolution completed in 0.000 seconds
Analyzed 77 packages to resolve dependencies
Analyzed 137 rules to resolve dependencies
Nothing to install or update
Generating autoload files
  [merge-plugin] Already merged vendor/yiisoft/yii2-jquery/composer.assets.json completely

This actually looks good when reading the messages:

• default packages are installed
• additional composer.assets.json is found
• correct addtional packages are detected
• additional update is run to apply merge settings

For some reason the log says:

Installing dependencies (including require-dev) from lock file

Looks like an install, but that should be an update.

But here the additional packages are not installed, I need another run for that:

root@b620651b73b6:/app# composer update -vv
    1/3:	http://packagist.org/p/provider-latest$7bd67f74c323ff6601aa1ca39c2b1fbd1bc7cc3e5cafedd8ff0e7988ccb7f647.json
    2/3:	http://packagist.org/p/provider-2017-07$00a09e1025fd22008b01aba5fba08327ccb5a3b7a4fcf1d27f25033dec2e0cf9.json
    3/3:	http://packagist.org/p/provider-2018-01$1f72d71f41de15a06d4105ec1a277fa32cc295e41b7ed9519fecf3704de7ddd6.json
    Finished: success: 3, skipped: 0, failure: 0, total: 3
  [merge-plugin] Loading vendor/yiisoft/yii2-jquery/composer.assets.json...
  [merge-plugin] Merging npm-asset/jquery
  [merge-plugin] Merging npm-asset/punycode
  [merge-plugin] Loading -dev sections of vendor/yiisoft/yii2-jquery/composer.assets.json...
Loading composer repositories with package information
[...]
Updating dependencies (including require-dev)
Dependency resolution completed in 0.001 seconds
Analyzed 717 packages to resolve dependencies
Analyzed 504 rules to resolve dependencies
Package operations: 2 installs, 0 updates, 0 removals
Installs: npm-asset/jquery:3.2.1, npm-asset/punycode:1.3.2
  - Installing npm-asset/jquery (3.2.1): Loading from cache
 Extracting archive    REASON: Required by the root package: Install command rule (install npm-asset/jquery 3.2.1|install npm-asset/jquery 3.2.0|install npm-asset/jquery 3.1.1|install npm-asset/jquery 3.1.0|install npm-asset/jquery 2.2.4|install npm-asset/jquery 2.2.3|install npm-asset/jquery 2.2.2|install npm-asset/jquery 2.2.1|install npm-asset/jquery 2.2.0|install npm-asset/jquery 2.1.4|install npm-asset/jquery 2.1.3|install npm-asset/jquery 2.1.2|install npm-asset/jquery 2.1.1|install npm-asset/jquery 2.1.0|install npm-asset/jquery 1.12.4|install npm-asset/jquery 1.12.3|install npm-asset/jquery 1.12.2|install npm-asset/jquery 1.12.1|install npm-asset/jquery 1.12.0|install npm-asset/jquery 1.11.3|install npm-asset/jquery 1.11.2|install npm-asset/jquery 1.11.1|install npm-asset/jquery 1.11.0)

  - Installing npm-asset/punycode (1.3.2): Loading from cache
 Extracting archive    REASON: Required by the root package: Install command rule (install npm-asset/punycode 1.3.2|install npm-asset/punycode 1.3.1|install npm-asset/punycode 1.3.0)

Writing lock file
Generating autoload files
  [merge-plugin] Already merged vendor/yiisoft/yii2-jquery/composer.assets.json completely

Anything I have missed here?

CC: @cebe

[schmunk42]

Addon: It works as expected, when I have no (or an empy) composer.lock file.

I just need to run update, additional files are found, merged and deps installed 👍

$ composer update -vv

[...]

Writing lock file
Generating autoload files
  [merge-plugin] Loading vendor/yiisoft/yii2-jquery/composer.assets.json...
  [merge-plugin] Merging npm-asset/jquery
  [merge-plugin] Merging npm-asset/punycode
  [merge-plugin] Running additional update to apply merge settings
  [merge-plugin] Loading vendor/yiisoft/yii2-jquery/composer.assets.json...
  [merge-plugin] Merging npm-asset/jquery
  [merge-plugin] Merging npm-asset/punycode
  [merge-plugin] Loading -dev sections of vendor/yiisoft/yii2-jquery/composer.assets.json...
Loading composer repositories with package information
Reading composer.json of yiisoft/yii2-jquery (feature/nam)
Importing branch feature/nam (dev-feature/nam)
Reading composer.json of yiisoft/yii2-jquery (feature/remove-pjax)
Importing branch feature/remove-pjax (dev-feature/remove-pjax)
Reading composer.json of yiisoft/yii2-jquery (master)
Importing branch master (dev-master)
Updating dependencies (including require-dev)
Dependency resolution completed in 0.001 seconds
Analyzed 718 packages to resolve dependencies
Analyzed 504 rules to resolve dependencies
Package operations: 2 installs, 0 updates, 0 removals
Installs: npm-asset/jquery:3.2.1, npm-asset/punycode:1.3.2
  - Installing npm-asset/jquery (3.2.1): Loading from cache
 Extracting archive  - Installing npm-asset/punycode (1.3.2): Loading from cache
 Extracting archiveWriting lock file
Generating autoload files
  [merge-plugin] Already merged vendor/yiisoft/yii2-jquery/composer.assets.json completely

[schmunk42]

Halp! Can anyone shed some light on this?

[bd808]

Having composer-merge-plugin install packages specified by files which are provisioned by other packages is not supported. This is a "chicken and egg" problem similar to the problem that composer-merge-plugin works around for it's own installation (see #170).

Composer-merge-plugin adds additional data to Composer's internal dependency resolver at the beginning of any Composer install/update action. Composer takes over from that point and actually installs the resolved packages. Any data that is not available on the local disk at the start of the install/update action is not available for either Composer or composer-merge-plugin to examine. If the use case of gathering configuration from packages as they are provisioned were to be supported, composer-merge-plugin would need to at minimum recompute its configuration following every package install and then trigger an additional Composer run if that configuration was different before & after the prior run. In the most pathological case this could trigger infinite Composer runs if each package provisioned a new configuration file that matched the composer-merge-plugin include files pattern and that config specified another package which did the same. Ultimately the problem is that we cannot traverse and verify the internal consistency of a configuration graph until that entire graph is known.