Support Development with local Dependencies

[gossi]

Hey guys,

let's face it, development with local dependencies is ... a nightmare. To clearify what I'm talking about. You write package A which depends on package B which you are parallely developing. There are some workarounds although they also have pitfalls. I summarized all this in a blog post: http://gos.si/blog/composer-development-with-local-dependencies

I also have an idea what needs to be done to solve this issue (it's at the end of the article yet I repeat it here) which I put here for discussion:

Simply spoken: “Hey composer, here are my local packages, whenever I try to install one of them (in a development version), point them there”. Your local development belongs to the developer and is very much different across developers. Every developer needs to describe its environment to its own composer installation. Luckily there is a global ~/composer.json that already manages global package installs. This can be used to tell composer about local packages. Something like this:

// ~/composer.json
{
  ..
  "extra": {
    "packages": {
      "": "path/to/packages", // global location
      "<vendor>": "path/to/<vendor>",
      "<vendor/pkg>": "this/is/somewhere/else"
    }
  }
  ..
}

Whenever composer runs an update or install command, the first source to check for packages are those defined above. If they can’t be find locally, proceed with the current search progress. composer.json from local dependencies can be fetched from local sources and changed dependencies can be applied on either install or update. Locally defined packages are always symlinked to their desired location (that given, custom installers still work as expected), almost eliminating the need for composer update on code changes from local dependencies.

Refs: #1299 #1017 #3658 #601 #2171 #3254

[aderuwe]

AFAIK this is already possible by remapping particular namespaces to folders other than those in vendor in the "psr-0" or "psr-4" keys of the "autoload" section in composer.json. Don't have an example handy right now.

[gossi]

Nope, you missunderstood this. You only declare a dependency, for now it may be local, later it will be just a normal package consumed by somebody else through packagist.

[aderuwe]

Yup. So you remove the custom namespace mapping at that point - I don't see what's different between that and removing your special key in "extra" ...

[gossi]

I have a sense you do not even understand the problem - I'm waiting for those instead.

[sbuzonas]

This is already possible without modification to composer.

For loosely coupled dependencies where a commit is acceptable you can list all of your VCS repos in your global composer.json to allow an update to get your local changes without pushing.

More tightly coupled dependencies that may need you to be able to see the impact of a change to a file without a commit can be done with three approaches that I'm aware of:

• Making the change in the VCS of the package within the vendor directory and committing from there. Not very elegant, but effective.
• Symlinking, this is essentially the same as the above but allows you to have the package located elsewhere and incorporated in multiple projects if needed. This approach is a bit messy.
• Composer configuration. This is my preferred approach over the previous two and I'll explain further below.

You can use an alternate composer file, I use local.json, for your development cycle. I add the autoload configuration for all of the dependencies I want finer control over to the autoload block leveraging relative paths. Additionally, I duplicate the dependencies as needed within the require sections of this local.json. This configures composer to work on this defined set of dependencies as a sort of metapackage.

In any case it's a manual process and somewhat of a hassle. I agree there should be a way to manage it, but I don't know if it really belongs in the core. When you get to this level of configuration it is extremely opinionated, and you will never find a one size fits all solution. I believe this is a perfect use case for a global composer plugin. Then each developer can choose their own implementation that suits them even across the same project.

As an aside, your example configuration utilizes the extra block, this is generally intended for configuration of items outside of composer itself such as installers, plugins, and scripts. If it were to be in core, it would probably be more intuitive and better suited for the configuration to be spread across the config and repositories block depending on the scope of the configuration directive.

[aderuwe]

@gossi Sure, good luck.

Perhaps consider that the feature you request can be achieved by mapping a namespace to a custom directory (your local package's directory). Then read https://getcomposer.org/doc/04-schema.md#autoload

No pushes required, just a simple dump-autoload.

(Admittedly this doesn't allow remapping an entire vendor namespace, but that seems far-fetched in any case.)

[gossi]

@aderuwe The only thing necessary for declaring a (local) dependency is putting them in require section, if you are forced to use autoload you are doing something wrong. Remember the dependency for now lives local (for me), for you it is just another dependency from packagist. What you are describing is: You clone each dependency you want to have to your hdd and then put them into autoload? Sure not.

@slbmeh Good catch but these are all workarounds, which all have glitches, I summarized them here: http://gos.si/blog/composer-development-with-local-dependencies I used the extra, because it is what I have at the moment but you are right, for the global composer.json there can for sure be a better place. I would also consider writing a plugin first (which is what I've done, through a custom installer). I'm afraid that the required code is out of the scope for a plugin - do you know more about this?

[aderuwe]

@gossi you put your dependency in require, and override the autoload while working on them locally, so that you do not need to push every time and not run composer update for it. That's exactly the problem described in your blog post. The only drawback is making sure you don't commit those overrides, as they're just meant for you while working on the library. When you're done, remove the override and dump the autoloader.

[sbuzonas]

@gossi afaik everything you need is accessible from a plugin scope, it would need to be a global plugin otherwise it might not be available early enough in my experience. This may have changed.

I believe the solver will pick the package the solves the dependencies from the first repository it discovers them in, so adding a repository in the plugin's activate method should suffice for what you need.

I have a plugin that injects packages from a custom repository that is SVN backed, the concept is pretty much the same, but you're discovering the package metadata differently and you may have your repositories explicitly defined in this particular case.

https://github.com/fancyguy/composer-wordpress-plugin/blob/master/src/WordPressPlugin.php

[gossi]

@slbmeh That's already pretty close and your plugin is very impressive. Although it does not work entirely for what I described. So, you can install a plugin globally, which would work for global installations and the same for local installations. What I am looking for is a global plugin that would work for local installations and unfortunately this is out of scope for a plugin. This needs be handled via composer itself.

[sbuzonas]

@gossi I was approaching a very different topic when I wrote that plugin. I used it as an example because it does a lot of the things you would need to do in order to make it happen. Installing the plugin globally will also include it in local scope when installing packages.

I showed that as an example because it does introduce packages from an external source. In that particular case it would be SVN repositories with manifested metadata. The repository you create would be responsible for retrieving it from whatever source you define. This would probably be some sort of configuration in the extra block if it were to be a plugin. You can add to the configuration block and repositories block with a global plugin, but I ran in to some issues when you want to update your plugin such as running without plugins.

Depending on the approach you want to tackle, you can do a number of things with just a repository. You can have configuration for a directory to scan for VCS repositories within it. You can explicitly declare paths to repositories, etc... All of this would then be adding to the packages that the repository contains. This would work with VCS repositories without any additional effort.

If you want to have arbitrary directories that aren't in source control, you you need to handle that in the repository. You would probably also need to implement a downloader in order to have composer install it into vendor properly.

And finally, if you want to share the working directories between these dependencies you would need to implement an installer as well to basically be a null installer doing absolutely nothing except for providing the path information needed for the autoloader and such.

What do you need to achieve that you see as not possible to do from a plugin perspective?

[lorcom]

@slbmeh Hi, I've read your solution that use an alternative of composer.json and I like it, I think it's cleany and simple to mantaine. I tried to use local.json instead of composer.json with option command without success.
Can you explain me how you configure it?

Thank you in advance

[lorcom]

@slbmeh I found the solution here: https://getcomposer.org/doc/03-cli.md#environment-variables.

[gossi]

I tried to put this into a plugin today. There is one catch and that is custom installers. There can only be one installer for one package (as far as I understand the composer code atm):

1. the first installer that supports() wins the race, so there is no priority.
2. the installers aren't chained, for example: I do write a custom installer, which has getInstallPath() and in install() does also other required stuff.

So the install() method can do one or all of the following things:

• Requests the install path via getInstallPath()
• Handles the code install (or passes this one on to LibraryInstaller::install())
• Does additional stuff

What is possible at the moment: Write an installer, that preludes all others, when the install() method is called, check all the other available installers for the correct getInstallPath() and symlink to that folder. Yet all the additional stuff that such an installer would handle is ignored. That's only two out of three, how to get that last one. Any idea?

[sbuzonas]

@gossi that is a tricky topic with the installers. I've run into the issue with installers myself. What I've done in the past is used the decorator pattern to wrap the installers, iterating over everything in the installation manager. You can block on the supports call to force it the way you need it.

[gossi]

That's a similar approach to what I'd do at the end the day we have the same limitation. Would like to know what @Seldaek has to say about this and what is possible, what would make most sense in this scenario.

[gossi]

Some time ago I made some experiments. Yet I ran into some problems and shared them on composer-dev mailing list. The list seems to be dead, so I share the link with you, maybe you have an idea: https://groups.google.com/forum/#!topic/composer-dev/u-jKVnuxg2M

[Seldaek]

Have you seen https://getcomposer.org/doc/05-repositories.md#path ? That might help you achieve this.

[gossi]

I continued experimenting around today. Here are my findings:

1. Adding a local repo to the repository manager has no effect for dependency resolving
2. Using the path repo has no effect on dependency resolving (neither for root package nor the operating package - both added programmatically)

However, I managed to achieve what I wanted:
I scanned through the locally required packages and added their requires to the root package as well, at least one item is off my list ✊
Hooking the install manager is almost impossible from plugin itself as it is sandboxed within composer itself. In that rare cases, people must run the update/install command twice (sounds like latex, right?). An idea might be to run update/install again, after a composer-plugin package type is detected during installation, however that seems doing it wrong and stupid.
There is one remaining issue which I don't remember how to reproduce. Will see throughout my development.

Suggestions are still welcome.

[gossi]

Oh I forget to mention. Here is the plugin-repo: https://github.com/gossi/composer-localdev-plugin I just wrote a readme to get you going in case you wanna try.

[mreschke]

The https://getcomposer.org/doc/05-repositories.md#path does work great, almost 100% except for the version issues...see #4635

[gossi]

Your question is answered in gossi/composer-localdev-plugin#4

[ahuszko]

This may help you to improve local package development: http://ahuszko.ghost.io/compact/

[staabm]

@Seldaek I think we can close this one, because we have PathRepository now?
https://getcomposer.org/doc/05-repositories.md#path

[Seldaek]

Closing as duplicate of #601