New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installs plugins and dependencies before installation of other packages #3082

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
@francoispluchino
Contributor

francoispluchino commented Jun 29, 2014

Q A
Bug fix? yes (installation plugin order)
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? yes
Fixed tickets #2972, #2879, #2878, #1121, #3078
License MIT

This feature allows installing plugins and their dependencies before restarting the installation of other packages in the project. It allows the plugins to be added directly in the project and must not be installed in global.

With this feature it is for example possible to create a plugin adding automatically the repositories, without that Composer throws an error when the first installation.

It does not change the original behavior of the install, but at the same time, fixes a "strange" behavior of the order of the installation of plugins and their dependencies.

The dependencies plugins must be installed before the plugin, then the plugin must be installed, and then the rest of depedencies can be installed.

Test example:

{
    "repositories": [
        {
            "type": "package",
            "package": [
                { "name": "pkg", "version": "1.0.0" },
                { "name": "pkg2", "version": "1.0.0" },
                { "name": "inst", "version": "1.0.0", "type": "composer-plugin" },
                { "name": "inst2", "version": "1.0.0", "type": "composer-plugin", "require": { "pkg2": "*" } }
            ]
        }
    ],
    "require": {
        "pkg": "1.0.0",
        "inst": "1.0.0",
        "inst2": "1.0.0"
    }
}

The order should be:

Installing inst (1.0.0)
Installing pkg2 (1.0.0)
Installing inst2 (1.0.0)
Installing pkg (1.0.0)

and not (actually):

Installing inst (1.0.0)
Installing pkg (1.0.0)
Installing pkg2 (1.0.0)
Installing inst2 (1.0.0)
@nsams

This comment has been minimized.

Show comment
Hide comment
@nsams

nsams Jun 30, 2014

+1

This makes many things possible using plugins.

nsams commented Jun 30, 2014

+1

This makes many things possible using plugins.

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jul 1, 2014

Contributor

@Seldaek It would be really cool to have this improvement (and correction) for the plugin system, do you think this PR could be merged?

Contributor

francoispluchino commented Jul 1, 2014

@Seldaek It would be really cool to have this improvement (and correction) for the plugin system, do you think this PR could be merged?

@cebe

View changes

Show outdated Hide outdated src/Composer/Installer.php
@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jul 8, 2014

Contributor

@Seldaek, @naderman Do you think that this PR has a chance to be merged quickly?

This PR has no a break compatibility, the project with or without plugin retains the same running, but allows the plugins to be able to carry more thing in "project" mode.

Contributor

francoispluchino commented Jul 8, 2014

@Seldaek, @naderman Do you think that this PR has a chance to be merged quickly?

This PR has no a break compatibility, the project with or without plugin retains the same running, but allows the plugins to be able to carry more thing in "project" mode.

@naderman

This comment has been minimized.

Show comment
Hide comment
@naderman

naderman Jul 8, 2014

Member

Doesn't this entirely break the installation of plugins that depend on regular composer packages?

Member

naderman commented Jul 8, 2014

Doesn't this entirely break the installation of plugins that depend on regular composer packages?

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jul 8, 2014

Contributor

@naderman no break for this: the new version of the plugins installation, allows install the regular (composer) dependencies of plugins before the installation of plugins.

Contributor

francoispluchino commented Jul 8, 2014

@naderman no break for this: the new version of the plugins installation, allows install the regular (composer) dependencies of plugins before the installation of plugins.

@naderman

This comment has been minimized.

Show comment
Hide comment
@naderman

naderman Jul 8, 2014

Member

Guess this looks ok to me, @Seldaek what do you think about the multiple pool implementation of this? I can't really think of a better alternative either though.

Member

naderman commented Jul 8, 2014

Guess this looks ok to me, @Seldaek what do you think about the multiple pool implementation of this? I can't really think of a better alternative either though.

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jul 8, 2014

Contributor

@naderman Thanks!

Only the conditions for the display of informations are unnecessary, but it helps to keep the old display (and therefore, not having duplicate information in the console).

I made 4 attempts before proposing this PR, and this one, is the most optimized and uses maximum of the existing code. This allowed me to see the inner workings of Solver and Pool, and it's really a nice work!

Contributor

francoispluchino commented Jul 8, 2014

@naderman Thanks!

Only the conditions for the display of informations are unnecessary, but it helps to keep the old display (and therefore, not having duplicate information in the console).

I made 4 attempts before proposing this PR, and this one, is the most optimized and uses maximum of the existing code. This allowed me to see the inner workings of Solver and Pool, and it's really a nice work!

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jul 10, 2014

Contributor

Continuing to add new features to my plugin, I find that I come quickly to the limitations of the plugin system, same with this PR.

With the PR #3116, I propose to add new events to the Installer, allowing the plugins to do much more thing, while maintaining compatibility with the existing plugin system.

@naderman, @Seldaek Can you look the issue #3116, and tell me if I can add the feature to this PR? Or create another PR.

Contributor

francoispluchino commented Jul 10, 2014

Continuing to add new features to my plugin, I find that I come quickly to the limitations of the plugin system, same with this PR.

With the PR #3116, I propose to add new events to the Installer, allowing the plugins to do much more thing, while maintaining compatibility with the existing plugin system.

@naderman, @Seldaek Can you look the issue #3116, and tell me if I can add the feature to this PR? Or create another PR.

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jul 15, 2014

Contributor

@naderman, @Seldaek Do you have any news for this PR?

Contributor

francoispluchino commented Jul 15, 2014

@naderman, @Seldaek Do you have any news for this PR?

@naderman

This comment has been minimized.

Show comment
Hide comment
@naderman

naderman Jul 15, 2014

Member

Going to merge this I think, but that should certainly stay a separate PR.

Member

naderman commented Jul 15, 2014

Going to merge this I think, but that should certainly stay a separate PR.

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jul 15, 2014

Contributor

@naderman Ok, I do not edit this PR in this case.

Contributor

francoispluchino commented Jul 15, 2014

@naderman Ok, I do not edit this PR in this case.

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jul 18, 2014

Contributor

@Seldaek Is it possible to merge this PR, if it suits you?

Contributor

francoispluchino commented Jul 18, 2014

@Seldaek Is it possible to merge this PR, if it suits you?

@Seldaek Seldaek added this to the Backwards Compatible milestone Jul 19, 2014

@Seldaek Seldaek added the Feature label Jul 19, 2014

@nsams

This comment has been minimized.

Show comment
Hide comment
@nsams

nsams Jul 24, 2014

@Seldaek sorry to annoy, could please comment on this if you plan to merge it.

nsams commented Jul 24, 2014

@Seldaek sorry to annoy, could please comment on this if you plan to merge it.

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jul 24, 2014

Contributor

@nsams I spoke with @Seldaek last week, and he prefers to fix urgent problems and take the time to look a little closer this PR.

Contributor

francoispluchino commented Jul 24, 2014

@nsams I spoke with @Seldaek last week, and he prefers to fix urgent problems and take the time to look a little closer this PR.

@bishopb

This comment has been minimized.

Show comment
Hide comment
@bishopb

bishopb Jan 1, 2015

@samdark Yes, I follow. Both approaches I'm advocating handle that, and neither require this PR.

Suppose PHP code builds Javascript that requires jQuery 2.1.2 and jQuery UI 1.10.4.

In the first approach, the one that works today but is not ideal, composer.json contains:

"require": { "eloquent/composer-npm-bridge": "~2" }

The package.json file contains:

"dependencies": { "jquery": "2.1.2", "jquery-ui": "1.10.4" }

Running composer install pulls down composer-npm-bridge 2.* into vendor, which runs post install the npm installer to get jQuery 2.1.2 and jQuery UI 1.10.4. One can independently change the versions by editing these separate files. And upstream can happily release new versions without impacting the code. If one later add a PHP package that uses npm to require jQuery 1.11.2 (which is incompatible with 2.1.2) the post install step will complain and can be resolved appropriately.

If packagist specifically or satis generally were to add the features from this plugin, which the second approach I advocate, then one only needs a single composer.json to specify all dependencies:

"require": {
    "eloquent/composer-npm-bridge": "~2",
    "virtual/npm-jquery": "2.1.2",
    "virtual/npm-jquery-ui": "1.10.4"
}

The plugin wants to specify the syntax like above. That syntax is fantastic! I love it. I want it. But I do not feel the composer plugin approach is working. It's asking composer to do things it wasn't designed to do. It's reserving "npm-asset" and "bower-asset" without any teeth to enforce the reservation. It only works for those who happen upon the plugin.

The PHP world needs what this plugin offers. But not this way. Let packagist handle the proxying. Let composer continue to operate as it does.


@cebe I believe a blending of your experiment and this plugin is the best road to follow. Packagist would expose npm and bower packages via a reserved virtual vendor. Upon a request for a specific package within that vendor (eg, virtual/npm-jquery), packagist (via the plugin code) would scan the corresponding npm (or bower) repository for matching versions. That dependency graph would be packaged in composer-expected format and returned (as the plugin does now), and cached for subsequent fetches.

This approach is one of on-demand proxying in Packagist proper, rather than a separate Packagist-like repository that bootstraps npm and bower registries.

Analysis: Existing packages continue to work unaffected. If a package author decides he needs an npm- or bower-held asset, he can look them up in Packagist and add them directly to his package's composer.json. He doesn't have to install a special plugin. He doesn't have to do a composer update to get the revised version with this new restart feature. We're throwing away none of the plugin's great code: just moving where the work is done. True, the Packagist farm is doing more work (because there are now lots more packages), but the work done is just "more of the same" and done "just in time".

bishopb commented Jan 1, 2015

@samdark Yes, I follow. Both approaches I'm advocating handle that, and neither require this PR.

Suppose PHP code builds Javascript that requires jQuery 2.1.2 and jQuery UI 1.10.4.

In the first approach, the one that works today but is not ideal, composer.json contains:

"require": { "eloquent/composer-npm-bridge": "~2" }

The package.json file contains:

"dependencies": { "jquery": "2.1.2", "jquery-ui": "1.10.4" }

Running composer install pulls down composer-npm-bridge 2.* into vendor, which runs post install the npm installer to get jQuery 2.1.2 and jQuery UI 1.10.4. One can independently change the versions by editing these separate files. And upstream can happily release new versions without impacting the code. If one later add a PHP package that uses npm to require jQuery 1.11.2 (which is incompatible with 2.1.2) the post install step will complain and can be resolved appropriately.

If packagist specifically or satis generally were to add the features from this plugin, which the second approach I advocate, then one only needs a single composer.json to specify all dependencies:

"require": {
    "eloquent/composer-npm-bridge": "~2",
    "virtual/npm-jquery": "2.1.2",
    "virtual/npm-jquery-ui": "1.10.4"
}

The plugin wants to specify the syntax like above. That syntax is fantastic! I love it. I want it. But I do not feel the composer plugin approach is working. It's asking composer to do things it wasn't designed to do. It's reserving "npm-asset" and "bower-asset" without any teeth to enforce the reservation. It only works for those who happen upon the plugin.

The PHP world needs what this plugin offers. But not this way. Let packagist handle the proxying. Let composer continue to operate as it does.


@cebe I believe a blending of your experiment and this plugin is the best road to follow. Packagist would expose npm and bower packages via a reserved virtual vendor. Upon a request for a specific package within that vendor (eg, virtual/npm-jquery), packagist (via the plugin code) would scan the corresponding npm (or bower) repository for matching versions. That dependency graph would be packaged in composer-expected format and returned (as the plugin does now), and cached for subsequent fetches.

This approach is one of on-demand proxying in Packagist proper, rather than a separate Packagist-like repository that bootstraps npm and bower registries.

Analysis: Existing packages continue to work unaffected. If a package author decides he needs an npm- or bower-held asset, he can look them up in Packagist and add them directly to his package's composer.json. He doesn't have to install a special plugin. He doesn't have to do a composer update to get the revised version with this new restart feature. We're throwing away none of the plugin's great code: just moving where the work is done. True, the Packagist farm is doing more work (because there are now lots more packages), but the work done is just "more of the same" and done "just in time".

@damiandennis

This comment has been minimized.

Show comment
Hide comment
@damiandennis

damiandennis Jan 1, 2015

This plugin works great but I see the point being made, It sounds like a program needs to be made that can integrate composer, npm and bower into one package and check dependencies before running each of the package managers. I am not sure that however is even possible as it depends on how flexible each of the package managers. Also it does sound like a lot of work...

damiandennis commented Jan 1, 2015

This plugin works great but I see the point being made, It sounds like a program needs to be made that can integrate composer, npm and bower into one package and check dependencies before running each of the package managers. I am not sure that however is even possible as it depends on how flexible each of the package managers. Also it does sound like a lot of work...

@schmunk42

This comment has been minimized.

Show comment
Hide comment
@schmunk42

schmunk42 Jan 1, 2015

Contributor

@bishopb

Either (a) I deliberately choose packages that have compatible front-end assets or (b) handle the front-end entirely in the root package. For me, the bootstrapping requirement to "install npm and/or bower installers" trumps the requirement "install no dependencies if any of their assets conflict".

(c) I install a bundle of PHP, CSS and JS files like described here and without any manual installation or setup.

That's my everyday use-case and I see no better solution than using the asset plugin, which does a wonderful job here.

It's asking composer to do things it wasn't designed to do.

That's why we have the possiblity to create plugins, you can never foresee all possible use-cases or new ideas.

It's reserving "npm-asset" and "bower-asset" without any teeth to enforce the reservation. It only works for those who happen upon the plugin.

I agree that this could be improved, ie. these paths could be made configurable like the paths, where the npm or bower packages are installed, see here for an example.

If you use the virtual packages, you also have to require the plugin, the problem is, that you have to install it globally at the moment, that's what this PR is trying to solve.

As a sidenote: You can create packages under any namespace, there's no reservation like the user- or org-name on GitHub.

True, the Packagist farm is doing more work (because there are now lots more packages), but the work done is just "more of the same" and done "just in time".

This looks problematic (kindly speaking) to me, see Module Counts, for a comparision of the numbers:

  • packagist 46480
  • bower 24855
  • npm 115298

You're suggesting to proxy over the threefold number of packages than currently hosted on packagist - while we are already experiencing a lag of several minutes when creating a new version - and moreover a very large percentage of npm and bower packages are not of interest for PHP projects.

This would totally clutter the packagist search and is no option IMHO.

Contributor

schmunk42 commented Jan 1, 2015

@bishopb

Either (a) I deliberately choose packages that have compatible front-end assets or (b) handle the front-end entirely in the root package. For me, the bootstrapping requirement to "install npm and/or bower installers" trumps the requirement "install no dependencies if any of their assets conflict".

(c) I install a bundle of PHP, CSS and JS files like described here and without any manual installation or setup.

That's my everyday use-case and I see no better solution than using the asset plugin, which does a wonderful job here.

It's asking composer to do things it wasn't designed to do.

That's why we have the possiblity to create plugins, you can never foresee all possible use-cases or new ideas.

It's reserving "npm-asset" and "bower-asset" without any teeth to enforce the reservation. It only works for those who happen upon the plugin.

I agree that this could be improved, ie. these paths could be made configurable like the paths, where the npm or bower packages are installed, see here for an example.

If you use the virtual packages, you also have to require the plugin, the problem is, that you have to install it globally at the moment, that's what this PR is trying to solve.

As a sidenote: You can create packages under any namespace, there's no reservation like the user- or org-name on GitHub.

True, the Packagist farm is doing more work (because there are now lots more packages), but the work done is just "more of the same" and done "just in time".

This looks problematic (kindly speaking) to me, see Module Counts, for a comparision of the numbers:

  • packagist 46480
  • bower 24855
  • npm 115298

You're suggesting to proxy over the threefold number of packages than currently hosted on packagist - while we are already experiencing a lag of several minutes when creating a new version - and moreover a very large percentage of npm and bower packages are not of interest for PHP projects.

This would totally clutter the packagist search and is no option IMHO.

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jan 1, 2015

Contributor

@bishopb

But with all respect, I think the plugin's design of npm-asset and bower-asset as "virtual" vendors is misguided. That design decision is at the root of this issue. And other future issues: when someone creates an "npm-asset" (or "bower-asset") packagist category, then this plugin will essentially stop working, without some patching to map around the new category.

I agree with you, and that's why I chose to prefix the asset dependency with TYPE-asset. Nobody will use this prefix, but it is true that it can be problematic.

Regarding the design problem: thank you for your constructive comments, and it's always interesting to have external advice. I will not repeat all of my approach to my plugin (because it is not the subject), but the main idea was to use the maximum the native components of Composer (Installer, Sovler, Locker, Commands, etc...) and simply transformed the package information for Composer.
Personally I would have liked the possibility to prefix the asset dependency with "@bower" and "@npm" for example. In this way, Composer knows that dependency is a virtual dependency and does not take into account in its installation (by the Solver).

I also had the possibility to put the asset dependencies in the "extra" section, but in this case, we do not use the native components of Composer.

Your proposal to use "virtual/npm-" and "virtual/bower-" has the same problem as my implementation.

Your proposal to use a proxy, was my initial research: technically, it is possible, but unfortunately, the analysis is extremely heavy, and therefore, the limitation of 5000 request/hours to Github API would soon exploded just NPM has over 80 000 packages).

@bishopb; @cebe, @samdark The debate around my plugin is very interesting, but it not concern this PR directly. Indeed, this PR is not specific for my plugibut it allows to solve the problem of installation order of dependencies of plugins, and allows to the plugins to do the same action whatever the chosen mode of installation (projet or global mode).

What makes me happy, is to see the interest aroused by this PR when there are more than 6 months, and that I was ready, for the new year, to close this PR to choose another solution (but significantly less native).

I think about the management of assets (javascript/stylesheet) for servers languages other than JS, is a very interesting and important topic, because with any Framework used, management of assets is simply bypassed. For example I use a lot Symfony2, and the recommendation is to not put the assets in Bundles, but it no offers a technical solution to easily handle this case. And for other Frameworks (ZF2, Laravel4, etc...), from my initial tests, it's the same problem, except for Yii2 (but that may have changed since).

Another point, it is not because a project is not initially designed to manage something, that this tool can not evolve, if that something is a recurring problem. The management of assets is a recurring and pervasive problem for any project that manage the back-end and front-end. Given that Composer manages the PHP dependencies, I think it would be interesting it can manage the assets (without for all that use my solution).

For finish, you find that the principle is cool, but not the implementation, because for you, the plugin asks Composer to do something it does not. In this regard I do not agree with you. Did you see the Doc and FAQs? (see How does work the plugin?).

Contributor

francoispluchino commented Jan 1, 2015

@bishopb

But with all respect, I think the plugin's design of npm-asset and bower-asset as "virtual" vendors is misguided. That design decision is at the root of this issue. And other future issues: when someone creates an "npm-asset" (or "bower-asset") packagist category, then this plugin will essentially stop working, without some patching to map around the new category.

I agree with you, and that's why I chose to prefix the asset dependency with TYPE-asset. Nobody will use this prefix, but it is true that it can be problematic.

Regarding the design problem: thank you for your constructive comments, and it's always interesting to have external advice. I will not repeat all of my approach to my plugin (because it is not the subject), but the main idea was to use the maximum the native components of Composer (Installer, Sovler, Locker, Commands, etc...) and simply transformed the package information for Composer.
Personally I would have liked the possibility to prefix the asset dependency with "@bower" and "@npm" for example. In this way, Composer knows that dependency is a virtual dependency and does not take into account in its installation (by the Solver).

I also had the possibility to put the asset dependencies in the "extra" section, but in this case, we do not use the native components of Composer.

Your proposal to use "virtual/npm-" and "virtual/bower-" has the same problem as my implementation.

Your proposal to use a proxy, was my initial research: technically, it is possible, but unfortunately, the analysis is extremely heavy, and therefore, the limitation of 5000 request/hours to Github API would soon exploded just NPM has over 80 000 packages).

@bishopb; @cebe, @samdark The debate around my plugin is very interesting, but it not concern this PR directly. Indeed, this PR is not specific for my plugibut it allows to solve the problem of installation order of dependencies of plugins, and allows to the plugins to do the same action whatever the chosen mode of installation (projet or global mode).

What makes me happy, is to see the interest aroused by this PR when there are more than 6 months, and that I was ready, for the new year, to close this PR to choose another solution (but significantly less native).

I think about the management of assets (javascript/stylesheet) for servers languages other than JS, is a very interesting and important topic, because with any Framework used, management of assets is simply bypassed. For example I use a lot Symfony2, and the recommendation is to not put the assets in Bundles, but it no offers a technical solution to easily handle this case. And for other Frameworks (ZF2, Laravel4, etc...), from my initial tests, it's the same problem, except for Yii2 (but that may have changed since).

Another point, it is not because a project is not initially designed to manage something, that this tool can not evolve, if that something is a recurring problem. The management of assets is a recurring and pervasive problem for any project that manage the back-end and front-end. Given that Composer manages the PHP dependencies, I think it would be interesting it can manage the assets (without for all that use my solution).

For finish, you find that the principle is cool, but not the implementation, because for you, the plugin asks Composer to do something it does not. In this regard I do not agree with you. Did you see the Doc and FAQs? (see How does work the plugin?).

@bishopb

This comment has been minimized.

Show comment
Hide comment
@bishopb

bishopb Jan 2, 2015

@francoispluchino @schmunk42 I'm glad we're continuing to have this discussion, as I feel a comprehensive asset management solution would help the whole community. But, yes, we're off topic with this PR. Is there a mailing list, IRC channel, or other place we can move this larger conversation?

For the PR. I did read the plugin docs, and I did read the entire PR, and I know a little bit about the Composer code base. I am not an expert, though.

I am with @stof when he said "define a repository providing these packages to the solver, rather than trying to make the solver behave in weird ways". I would prefer that repository be up at Packagist. But, I am ok with the dependencies being in "extra", as I think you meant in your recent #3602 comment.

I am with @naderman when he said "This is really just not what the solver is made for at all" regarding the Solver.php change. I am following the logic, and it seems to work. But again I'm not an expert. I personally want to see more test coverage, because I do not want a "Composer debacle" (like the silly npm debacle). Some tests that I think are needed:

  • plugin-restart-always.test but with --no-dev
  • plugin-restart-always.test but with a/a in conflicting versions
  • plugin-restart-always.test but with a/a depending on b/b
  • plugin-restart-always.test but with a/a depending on b/b in conflicting versions
  • timing tests with real-world (ie "non-trivial") dependency graphs

bishopb commented Jan 2, 2015

@francoispluchino @schmunk42 I'm glad we're continuing to have this discussion, as I feel a comprehensive asset management solution would help the whole community. But, yes, we're off topic with this PR. Is there a mailing list, IRC channel, or other place we can move this larger conversation?

For the PR. I did read the plugin docs, and I did read the entire PR, and I know a little bit about the Composer code base. I am not an expert, though.

I am with @stof when he said "define a repository providing these packages to the solver, rather than trying to make the solver behave in weird ways". I would prefer that repository be up at Packagist. But, I am ok with the dependencies being in "extra", as I think you meant in your recent #3602 comment.

I am with @naderman when he said "This is really just not what the solver is made for at all" regarding the Solver.php change. I am following the logic, and it seems to work. But again I'm not an expert. I personally want to see more test coverage, because I do not want a "Composer debacle" (like the silly npm debacle). Some tests that I think are needed:

  • plugin-restart-always.test but with --no-dev
  • plugin-restart-always.test but with a/a in conflicting versions
  • plugin-restart-always.test but with a/a depending on b/b
  • plugin-restart-always.test but with a/a depending on b/b in conflicting versions
  • timing tests with real-world (ie "non-trivial") dependency graphs
@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jan 2, 2015

Contributor

@bishopb Ok for the new tests, but can you be more precise for the last test ("timing tests with real-world (ie "non-trivial") dependency graphs")?

Contributor

francoispluchino commented Jan 2, 2015

@bishopb Ok for the new tests, but can you be more precise for the last test ("timing tests with real-world (ie "non-trivial") dependency graphs")?

@bishopb

This comment has been minimized.

Show comment
Hide comment
@bishopb

bishopb Jan 2, 2015

@francoispluchino The test cases have a maximum of three dependencies. The packages and applications I regularly work on have hundreds of dependencies, specified in cross-cutting fashion: A depends upon B and C; B and C both depend on D; and B and D depend upon E. Network overhead aside, the composer solver takes some time to process that dependency graph (call that time T_i). I would like to have a performance test that illustrates how much average additional time, if any, the restarts and other changes add to T_i.

bishopb commented Jan 2, 2015

@francoispluchino The test cases have a maximum of three dependencies. The packages and applications I regularly work on have hundreds of dependencies, specified in cross-cutting fashion: A depends upon B and C; B and C both depend on D; and B and D depend upon E. Network overhead aside, the composer solver takes some time to process that dependency graph (call that time T_i). I would like to have a performance test that illustrates how much average additional time, if any, the restarts and other changes add to T_i.

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jan 2, 2015

Contributor

@bishopb For the cost of time, you can see this comment.

Contributor

francoispluchino commented Jan 2, 2015

@bishopb For the cost of time, you can see this comment.

@bishopb

This comment has been minimized.

Show comment
Hide comment

bishopb commented Jan 2, 2015

@schmunk42

This comment has been minimized.

Show comment
Hide comment
@schmunk42

schmunk42 Jan 9, 2015

Contributor

@francoispluchino Could your plugin detect if it was installed globally and if not, fail with a human friendly error mesage?

Contributor

schmunk42 commented Jan 9, 2015

@francoispluchino Could your plugin detect if it was installed globally and if not, fail with a human friendly error mesage?

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jan 10, 2015

Contributor

@schmunk42 I think this is feasible with the property HOME directory. But for example, you added the plugin such as dependency of your project, and you ask your users to install the plugin in global mode.

At the first installation, the plugin in global mode is used, on the other hand, once the plugin is installed in project mode, it is the project plugin which is used (project plugins has a priority to global plugin).

So if I add a exception, I will break compatibility.

Contributor

francoispluchino commented Jan 10, 2015

@schmunk42 I think this is feasible with the property HOME directory. But for example, you added the plugin such as dependency of your project, and you ask your users to install the plugin in global mode.

At the first installation, the plugin in global mode is used, on the other hand, once the plugin is installed in project mode, it is the project plugin which is used (project plugins has a priority to global plugin).

So if I add a exception, I will break compatibility.

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe Jan 10, 2015

Contributor

So basically what this fixes is the first install, right?

Contributor

cebe commented Jan 10, 2015

So basically what this fixes is the first install, right?

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jan 10, 2015

Contributor

@cebe Right.

Contributor

francoispluchino commented Jan 10, 2015

@cebe Right.

@bishopb

This comment has been minimized.

Show comment
Hide comment
@bishopb

bishopb Jan 10, 2015

@cebe @francoispluchino Reaffirming that "first" appears to be the root issue, as mentioned in the require-global conversation. And also reaffirming what @schmunk42 said in that same conversation, paraphrased: require-global will not prevent fixing 3082, but will prevent first-time related errors`.

bishopb commented Jan 10, 2015

@cebe @francoispluchino Reaffirming that "first" appears to be the root issue, as mentioned in the require-global conversation. And also reaffirming what @schmunk42 said in that same conversation, paraphrased: require-global will not prevent fixing 3082, but will prevent first-time related errors`.

@francoispluchino

This comment has been minimized.

Show comment
Hide comment
@francoispluchino

francoispluchino Jan 10, 2015

Contributor

@bishopb My proposal of require-global (see this comment) does not prevent an error if a global plugin is not installed. This could be possible only for the root Composer package (read the proposal).

Indeed, the sub dependencies indicating that they need a global plugin (via require-global) cannot be analyzed, because the problem is the same that this PR: that is, that Composer cannot know the require-global of the dependence, because, in this dependency, there will be a sub-dependency that Composer will not know, and therefore an error will thrown.

To correct the problem, we should in this case, performing a first analysis dedicated to require-global, and launch a second analysis to resolve dependencies.

For reminder, it was the first version of this PR, and performance is poor compared to the last version proposed (see this comment vs this comment).

It is true, that the require-global will allow for a complete project to avoid unnecessary error, but this PR is much more relevant, especially that this latest version does not change the performance for all current and future projects, but only for projects that require a plugin must be installed prior to analysis (and only for the first installation, not updates, il the plugin is not installed in global mode).

Contributor

francoispluchino commented Jan 10, 2015

@bishopb My proposal of require-global (see this comment) does not prevent an error if a global plugin is not installed. This could be possible only for the root Composer package (read the proposal).

Indeed, the sub dependencies indicating that they need a global plugin (via require-global) cannot be analyzed, because the problem is the same that this PR: that is, that Composer cannot know the require-global of the dependence, because, in this dependency, there will be a sub-dependency that Composer will not know, and therefore an error will thrown.

To correct the problem, we should in this case, performing a first analysis dedicated to require-global, and launch a second analysis to resolve dependencies.

For reminder, it was the first version of this PR, and performance is poor compared to the last version proposed (see this comment vs this comment).

It is true, that the require-global will allow for a complete project to avoid unnecessary error, but this PR is much more relevant, especially that this latest version does not change the performance for all current and future projects, but only for projects that require a plugin must be installed prior to analysis (and only for the first installation, not updates, il the plugin is not installed in global mode).

@Marlinc

This comment has been minimized.

Show comment
Hide comment
@Marlinc

Marlinc May 25, 2015

Any news on this?

Marlinc commented May 25, 2015

Any news on this?

kschu91 added a commit to AOEpeople/CmsComposerInstallers that referenced this pull request Jul 10, 2015

@Seldaek Seldaek closed this Feb 21, 2016

@schmunk42

This comment has been minimized.

Show comment
Hide comment
@schmunk42

schmunk42 Feb 21, 2016

Contributor

@Seldaek Do you have more feedback why this is closed now? Is it just closed or solved by a commit?

Contributor

schmunk42 commented Feb 21, 2016

@Seldaek Do you have more feedback why this is closed now? Is it just closed or solved by a commit?

@Seldaek

This comment has been minimized.

Show comment
Hide comment
@Seldaek

Seldaek Feb 22, 2016

Member

It's just not mergeable so closing to clean up. It's not solved and I doubt it is really solveable without relying on global pre-installed plugins as people have been doing until now.

Member

Seldaek commented Feb 22, 2016

It's just not mergeable so closing to clean up. It's not solved and I doubt it is really solveable without relying on global pre-installed plugins as people have been doing until now.

@schmunk42

This comment has been minimized.

Show comment
Hide comment
@schmunk42

schmunk42 Feb 23, 2016

Contributor

Thank you for the info. Let me reference #3607 one more time.

It would be really great if we could get something like that, because I've seen so many bug reports where people were missing a global installation of the mentioned plugin.

Contributor

schmunk42 commented Feb 23, 2016

Thank you for the info. Let me reference #3607 one more time.

It would be really great if we could get something like that, because I've seen so many bug reports where people were missing a global installation of the mentioned plugin.

@kirkmadera

This comment has been minimized.

Show comment
Hide comment
@kirkmadera

kirkmadera Jul 13, 2017

Just a note, we have been using this at the project level for a long while now without issues. It's only the initial installation that is an issue. We run composer require fxp/composer-asset-plugin, then require our npm/bower assets. Future composer install runs work because of the composer.lock file. Then, composer update and composer require commands work at that point because the plugin is installed.

kirkmadera commented Jul 13, 2017

Just a note, we have been using this at the project level for a long while now without issues. It's only the initial installation that is an issue. We run composer require fxp/composer-asset-plugin, then require our npm/bower assets. Future composer install runs work because of the composer.lock file. Then, composer update and composer require commands work at that point because the plugin is installed.

@schmunk42

This comment has been minimized.

Show comment
Hide comment
@schmunk42

schmunk42 Jul 14, 2017

Contributor

@kirkmadera Sorry, but I must state the exact opposite, i.e. recently we had an issue with a global installation of the plugin in version 1.2.1, while one package in the project was requiring the plugin - which lead to installation of 1.3.0 (which introduces changes in config options).

It's only the initial installation that is an issue.

Regarding the above statement, the outcome is that on the first installation/update composer uses 1.2.1 with no warnings. On the second update 1.3.0 is used, which issues warnings about deprecated config options. If you change composer.json settings to the new config options to get rid of the warnings, your third installation runs fine ... but the next first installation on another machine (with global 1.2.1 installation) will fail (or run without the correct config), since 1.2.1 does not understand the new options and it also does not issue any warnings about that.

CC: @marc7000 @handcode

Contributor

schmunk42 commented Jul 14, 2017

@kirkmadera Sorry, but I must state the exact opposite, i.e. recently we had an issue with a global installation of the plugin in version 1.2.1, while one package in the project was requiring the plugin - which lead to installation of 1.3.0 (which introduces changes in config options).

It's only the initial installation that is an issue.

Regarding the above statement, the outcome is that on the first installation/update composer uses 1.2.1 with no warnings. On the second update 1.3.0 is used, which issues warnings about deprecated config options. If you change composer.json settings to the new config options to get rid of the warnings, your third installation runs fine ... but the next first installation on another machine (with global 1.2.1 installation) will fail (or run without the correct config), since 1.2.1 does not understand the new options and it also does not issue any warnings about that.

CC: @marc7000 @handcode

@kirkmadera

This comment has been minimized.

Show comment
Hide comment
@kirkmadera

kirkmadera Jul 14, 2017

In our setup, we do not use a global installation. Here is a test.

Remove the global plugin if you have it:

composer global remove fxp/composer-asset-plugin;

Prep a test dir with jquery installed.

mkdir test;
cd test;
composer require fxp/composer-asset-plugin;
composer require npm-asset/jquery;

Remove the vendor dir so that you are in a state like you just cloned the repo.

Attempt to use the asset plugin here and it will fail because the plugin is not installed into vendor:

composer require npm-asset/bootstrap

Run composer install to get the plugin installed again.

Now, the bootstrap include will work:

composer require npm-asset/bootstrap

To make this work you have to:

  1. Commit your lock file
  2. Run composer install after a git clone before running composer require or composer update
  3. Avoid manually editting your composer.json file with additional npm assets or bower assets before installing fxp/composer-asset-plugin

Committing the lock file is good practice anyways to make sure everyone is working with the same versions of everything. Running composer install on a checkout should be pretty standard. The third one is the gotcha when you start a project and are trying to build a reusable composer.json file to start with.

kirkmadera commented Jul 14, 2017

In our setup, we do not use a global installation. Here is a test.

Remove the global plugin if you have it:

composer global remove fxp/composer-asset-plugin;

Prep a test dir with jquery installed.

mkdir test;
cd test;
composer require fxp/composer-asset-plugin;
composer require npm-asset/jquery;

Remove the vendor dir so that you are in a state like you just cloned the repo.

Attempt to use the asset plugin here and it will fail because the plugin is not installed into vendor:

composer require npm-asset/bootstrap

Run composer install to get the plugin installed again.

Now, the bootstrap include will work:

composer require npm-asset/bootstrap

To make this work you have to:

  1. Commit your lock file
  2. Run composer install after a git clone before running composer require or composer update
  3. Avoid manually editting your composer.json file with additional npm assets or bower assets before installing fxp/composer-asset-plugin

Committing the lock file is good practice anyways to make sure everyone is working with the same versions of everything. Running composer install on a checkout should be pretty standard. The third one is the gotcha when you start a project and are trying to build a reusable composer.json file to start with.

@schmunk42

This comment has been minimized.

Show comment
Hide comment
@schmunk42

schmunk42 Jul 14, 2017

Contributor

Just for reference...

Project scope installation
The installation in the project scope is not supported (see the issue fxpio/composer-asset-plugin#7).

https://github.com/fxpio/composer-asset-plugin/blob/master/Resources/doc/index.md#project-scope-installation

Also related: wikimedia/composer-merge-plugin#138 but the other way around

Contributor

schmunk42 commented Jul 14, 2017

Just for reference...

Project scope installation
The installation in the project scope is not supported (see the issue fxpio/composer-asset-plugin#7).

https://github.com/fxpio/composer-asset-plugin/blob/master/Resources/doc/index.md#project-scope-installation

Also related: wikimedia/composer-merge-plugin#138 but the other way around

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment