Drush and Drupal always downloaded (and the latter out of date)

[jpstacey]

Steps to repeat

• run composer update against the following:

{
  "repositories": {
    "drupal": {
      "type": "composer",
      "url": "http://static.drupal-packagist.org/v0.1.0/"
    }
  },
  "require": {
    "composer/installers": "~1.0",
    "drupal/advanced_help": "7.*"
  }
}

What should happen

• advanced_help has no dependencies in its .info and should therefore be downloaded on its own.

What actually happens

• advanced_help is downloaded
• Drupal 7.28 (superseded on July 16) is downloaded with it, but into the weird folder vendor/drupal/drupal

More info

Downloading Drupal isn't of much use if (a) it's in an odd location and (b) it's out of date (and is a version that now has multiple security advisories against it.) I know practically all modules "depend" on Drupal, but doing this in the above example has turned a 75kB download into a 3+MB one, without the user being warned or having a choice to prevent it.

The best thing would be to require explicit download of Drupal in the JSON file, so that the developer can choose a version (or indeed choose not to download it at all, if they already have a local copy, or maybe choose to run off their own fork e.g. Backdrop.)

Depending on the module, Drush is sometimes randomly downloaded without explicit request either. Very few modules "depend" on Drush, although many of them can make use of it. Drush is also quite a large download and codebase, compared to many modules: 600+kB zipped.

[winmillwill]

The fact that a drupal module can not be used without drupal means that it's a dependency, and composer should definitely download it. Similarly, different modules and profiles require specific versions of drupal, and if composer can handle that for you, then hoping a human gets it right seems like a less good workflow. Composer caches its downloads, so if your new project settles on using a version of drupal/drupal that you already use, then it will copy out of that cache and it won't re-download drupal/drupal.

The 'static' part of that url is meant to indicate that the metadata it makes available was published once and will not be updated. That's why the latest version of drupal/drupal that it's aware of is 7.28.0. I'm glad for the recent interest in this project, and I have just cut v0.2.0 today: http://static.drupal-packagist.org/v0.2.0

EDIT: My bad, I built that against an old cut of the 'everything' xml from here: http://updates.drupal.org/release-history/project-list/all (<--very large file, do not click)
I have yet to fold the logic for consuming this info into this repo, and I instead just processed this into a simple list of project names and fed it into a command I added to my fork of packagist for use with drupal: https://github.com/winmillwill/packagist/blob/drupal/src/Packagist/WebBundle/Command/BulkAddPackagesCommand.php

I agree that leaving it in the vendor directory is not useful. That's why I recommend using drupal/tangler or a simple bash install script to render a drupal project that is ready to bootstrap. See #2 . I apologize that this is nowhere documented, as it is obviously confusing. I plan to fix that shortly.

To use a fork, the recommended strategy for composer is to use branch aliases. This documentation assumes that the project you are forking already has a working composer.json, which is not likely to be the case we are dealing with, so there is an additional step where you would have to provide such a composer.json in your fork. Another way of doing this would be to fully specify a package repository. In either case, you would either set the name to drupal/drupal, or use replace links to indicate that downloading drupal/drupal in the normal way is not necessary, and then require that package. The replace link strategy would require a change like the one proposed in #2 if we want other tools to identify the installed package as 'Drupal'. Looking forward, if this idea catches on, I would love to see a self-sustaining drupal packagist ecosystem apart from the legacy drupal.org solution, and package maintainers would just provide their own (actually useful) composer.json.

I would like to support all Drupal forks, since support for forks is the easiest way to support improvement of any project. I welcome any additional information on what I can do to make this happen.

[winmillwill]

Additionally, please provide an example composer.json that downloads drush in the way you describe and confirm whether it persists in v0.2.0.

Thanks!

EDIT: if you meant the issue with the themes, that's because I made an error in inferring the type of project. I used a heuristic that checked for the lack of module files and the presence of drush files. This is close to what the code behind drush dl does, except that it does this in a case where it has already qualified the project as a module when checking the metadata. I have updated my code to match in 8a02eeb and that fix was included in the code I used to build v0.2.0.

[jpstacey]

As with #4, I think the best solution is all to do with the behaviour people will expect. The predominant codebase-building tool in Drupal is Drush make, and it requires an explicit statement of the core package, otherwise it won't build core.

There's a bigger problem too. On the one hand we have the existing composers/installers initiative saying "we expect people to run a composer.json from within sites/all", yet here we end up with Drupal core into vendors/drupal/drupal. If you combine the two, then every site will be built with a second copy of Drupal, inside sites/all/vendor/drupal/drupal !

So I think the simplest solution for now, and the one in line with people's expectations and existing tools, is to require people to put a single line into "require" for the relevant Drupal/Pressflow/Backdrop core, and to otherwise not download Drupal. Certainly downloading it into vendor/drupal/drupal doesn't help, as the codebase as it stands requires manual intervention to fix.

[winmillwill]

As with #4, I think the best solution is all to do with the behaviour people will expect. The predominant codebase-building tool in Drupal is Drush make...

Sorry, that's exactly the problem. You could just as easily argue that it would be best to be unfriendly to people that have no friends. Drush make is bad and wastes time and composer is good and saves time -- that's why they behave differently.

The expectation of composer/installers running from inside a drupal root, inside sites/all, is an expectation formed by like 5 guys afaik. I wouldn't buck that expectation except for the fact that it's a terrible idea if you want to manage the version of drupal that you install, i.e., if you ever want to upgrade core, or try out an alternative core like you suggest. I'm aware that my proposed approach is incompatible, but that's because it is designed to address a fundamental problem with the expectation you mentioned.

Certainly downloading it into vendor/drupal/drupal doesn't help, as the codebase as it stands requires manual intervention to fix.

Again, please look at drupal/tangler for automating exactly this without the problem of composer/installers, which supposes that you are going to version all of drupal core in your project repository, which is bad. Also, I have a fix to report the package type of drupal as 'drupal-core', so if you can convince the composer/installers guys to support that, I think you will be happy. Alternatively, write your own installer, which the project page for composer/installers more or less indicates you should do:

installers isn't intended on replacing all custom installers. If your package requires special installation handling then by all means, create a custom installer to handle it.

Again, be advised that this will quite literally install your contrib inside your core, and a core update will delete your contrib and any custom code inside it.

I elected to write and use drupal/tangler because it supports a development workflow much better because it installs the code you are developing inside the drupal root so you can test it and allows you to recreate the drupal application on demand.

Finally, consider that you can't specify which version of Drupal you need if you don't make the packages depend on Drupal. You should note that there are in fact a couple modules and many profiles that do have requirements for the core version beyond the major version. That means that multiple modules can't form a consensus on what version of Drupal to use if you leave out the Drupal dependency. That means that you have another manual, error prone step in setting up your project, or at least a whole pile of stuff to version besides the code you are actually creating. Consider that my approach yields a fully executable project and others do not; they 'expect' you to do at least one more manual step.

I believe I have answered your original issue and then some. Please open a separate issue for other problems.