Read a package from a relative local path. #2171

Closed
Vittel opened this Issue Aug 13, 2013 · 37 comments

Comments

Projects
None yet

Vittel commented Aug 13, 2013

Add a repository type: dir/file/local that accepts a local relative path as source

Let me point out the requirement context a little bit:
We have a working deployment system up and running, deploying needed code to the exact right versions and positions in file system.
Now we try to enable composer on certain parts of the code that we identified as packages and to resolve further dependencies and autoload the contained classes.
Its not an option for us to break our deployment system, but it shouldn't be necessary when we could just read the prepared package from a local path (where it was deployed).
This is currently not possible. All existing repository configurations expect the repo to be remote or be accessible with a remote URL.
This is only the first step of our composer migration, and we are already stumbling.

Owner

Seldaek commented Aug 13, 2013

If it's a local vcs repo, you can access it via file:// protocol. Same for composer repositories I think.

Vittel commented Aug 13, 2013

Composer repositories do not support file://. I tried that. This would be the most desirable solution.
Do you have an example for the vcs solution?

Vittel commented Aug 13, 2013

What i forgot: there is no need to move the sources in any way. They are fine just where they are, so all references and classmap-entries should point to the given local relative path. Therefore it might be clever to implement this repo-style separately.

Vittel commented Aug 13, 2013

I tried the VCS proposal and failed:

    "repositories":
    [
        {
            "type": "vcs",
            "url": "file://../../xyz/current/"
        }
    ]

results in a
[RuntimeException]
Directory does not exist: file://../../xyz/current/

and yes, the directory exists :)

Owner

Seldaek commented Aug 13, 2013

Have you tried with jsut the relative path no file://?

Vittel commented Aug 14, 2013

    "repositories":
    [
        {
            "type": "vcs",
            "url": "../../xyz/current/",
            "branches-path": false,
            "tags-path": false
        }
    ],

results in a
[Composer\Repository\InvalidRepositoryException]
No valid composer.json was found in any branch or tag of ../../xyz/current/, could
not load a package from it.

Owner

Seldaek commented Aug 14, 2013

Oh no this only works with git (and maybe hg), not svn..

Vittel commented Aug 14, 2013

Still this is not a desirable solution since it would just try to copy the content of the packages into the vendor directory.
It seems that there is no way but adding (the already existent) Filesystem repository to the available repository types?

Owner

Seldaek commented Aug 14, 2013

The filesystem repo doesn't do what you want though. The closest is the
Artifact repo except that it works on zips, and it still copies files.
Not copying files would make things more complicated for almost no
benefit so I am not too keen on doing that.

Vittel commented Aug 15, 2013

Well, the benefit on our side is quite high. We are running a system where each branch is created on several repositories and deployed to a working dev-system. On each of that deployments we would have to patch the composer-file to use the correct branch for the specific library. So it is kind of a hard way to go for us.

This feature would simplify the migration of legacy projects towards composer enormously. Also it enables "custom" setups to use composer to resolve their library-dependencies. Both might increase the number of customers you have.

When i think about it, most of the functionality should be already available. It should be no more than copying some code from other existing classes into a "CustomPackageRepository" or whatever. I'm not so deep into your code, but it looks manageable to do it with less effort. (I'd do it, but it was not approved)

Owner

Seldaek commented Aug 15, 2013

I didn't quite understand your explanation of why you need it, but what I know is nobody else needed it so far, and we do have quite a lot of users by now (not customers, this is open source.). So I am not really enclined to add something that does not make sense to me and is only needed by one person. You could still code is and use your fork of composer to work with it, and send a pull request. Maybe when seeing the code I would understand better, but in the worst case using your fork isn't the end of the world, especially if it's just temporarily to facilitate migrating legacy projects.

Vittel commented Aug 15, 2013

Ok, i would appreciate it, but makes the migration just a bit more complicated.
I accept when you don't have resources for it.
Just let me explain a but further just to make you understand our setup/requirements.

We have an multi-repository project where the code of each repository is either placed in parallel or inside the directory-structure of the other repositories. All together, in a fixed directory-structure, is our framework, consisting of multiple application-stacks.

The structure is maintained by a custom deployment system, and is cloned, each time a new development-branch is created. (Every repo gets an independent development state).
Now we identified one main package, the library, that we would see as a composer package in future. It has its own dependencies/requires and is required itself by all other packages. In the first step we added the composer.json to the library and build up composer there.
The second step that was planned, is to add a composer.json to each of the application-folders, that requires the library. But since the current deployment system is hard to change, we would like to reference the package right from where it is, parallel in the tree, because it already got the right version checked out and is ready for use.

An alternative was to directly get the library from the SVN. But here i ran into problems due to the different deployment-branches. Each new branch would require to patch the composer-file to the new branch, and the file has to be un-patched when merging back the changes.

Btw.: it would be great if one could read parts of the composer-file from referenced json-files. this would keep the main file free from patches and the other files could be managed by an automated system (branch-names, repository-urls etc.)

glen-84 commented Sep 28, 2013

@Seldaek

I'm also looking for a local repository type.

Use case: I'm developing a new Zend Framework application, and separately one or more modules that will be used inside the application. During the early stages of development of the modules, I want to be able to include them in the application without them having to be under version control (checking in 3 partially-written half-working files seems "dirty" to me, I would prefer my initial commit to actually be somewhat functional).

@igorw suggested committing a simple composer.json to a local Git repo and then adding it to the project (which clones it). After this you can work on the module from within the application, but there are problems:

  • You can't push to the origin as it's on the same branch and non-bare. (and I don't want to add a local bare clone as well ... that'd be 3 copies of the repo in total, just for 1 module)
  • My IDE (Eclipse PDT) does not provide support for Git (via EGit) to cloned repos within another repo, so I would have to use Windows Explorer Git integration to push the changes.

I thought of maybe adding a "package" type repo, but I'm not sure if there is a "folder/dir/fs/local" source type.

For example, in this config:

{
    "type": "package",
    "package": {
        "name": "smarty/smarty",
        "version": "3.1.7",
        "dist": {
            "url": "http://www.smarty.net/files/Smarty-3.1.7.zip",
            "type": "zip"
        },
        "source": {
            "url": "http://smarty-php.googlecode.com/svn/",
            "type": "svn",
            "reference": "tags/Smarty_3_1_7/distribution/"
        }
    }
}

Could the source URL point to a regular folder containing a composer.json? What "type" would this be?

Thanks.

Contributor

stof commented Sep 30, 2013

@glen-84 Your case is what has been started in #1017 and never completed

glen-84 commented Sep 30, 2013

@stof Thanks.

aoberoi commented Mar 14, 2014

@Seldaek

I'm trying to do as you said for a relative local path that is a git repo:

{
  ...
  "repositories": [
    {
      "type" : "vcs",
      "url": "../../"
    }
  ]
}

I'm getting the following error:

$ composer -vvv update
Reading ./composer.json
Executing command (CWD): git describe --exact-match --tags
Executing command (CWD): git branch --no-color --no-abbrev -v
Loading composer repositories with package information
Executing command (CWD): p4 -p ../../ info -s
  [InvalidArgumentException]
  No driver found to handle VCS repository ../../
Exception trace:
 () at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Repository/VcsRepository.php:115
 Composer\Repository\VcsRepository->initialize() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Repository/ArrayRepository.php:170
 Composer\Repository\ArrayRepository->getPackages() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/Pool.php:163
 Composer\DependencyResolver\Pool->addRepository() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer.php:337
 Composer\Installer->doInstall() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer.php:210
 Composer\Installer->run() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Command/UpdateCommand.php:118
 Composer\Command\UpdateCommand->execute() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:241
 Symfony\Component\Console\Command\Command->run() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:881
 Symfony\Component\Console\Application->doRunCommand() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:191
 Symfony\Component\Console\Application->doRun() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Console/Application.php:117
 Composer\Console\Application->doRun() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:121
 Symfony\Component\Console\Application->run() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Console/Application.php:83
 Composer\Console\Application->run() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/bin/composer:43
 require() at /usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar:14

aoberoi commented Mar 14, 2014

i replaced the value of the "type" key with "git" and i seem to have gotten a little further before failing. (note: the docs didn't mention this value was possible, but i found it looking at the source. future enhancement?) here is the exception i am getting now (with some paths replaced with generic names):

[RuntimeException]
  Failed to execute git clone '../../' '/Users/ankur/Developer/aaaaaa/a-git-repo-package/sample/my-current-project/vendor/some-vendor/some-dep' && cd '/Users/ankur/Developer/aaaaaa/a-git-repo-package/sample/my-current-project/vendor/some-vendor/some-dep' && git remote add composer '../../' && git fetch composer
  Cloning into '/Users/ankur/Developer/aaaaaa/a-git-repo-package/sample/my-current-project/vendor/some-vendor/some-dep'...
  done.
  fatal: '../../' does not appear to be a git repository
  fatal: Could not read from remote repository.
  Please make sure you have the correct access rights
  and the repository exists.
Exception trace:
 () at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Downloader/GitDownloader.php:402
 Composer\Downloader\GitDownloader->throwException() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Downloader/GitDownloader.php:387
 Composer\Downloader\GitDownloader->runCommand() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Downloader/GitDownloader.php:43
 Composer\Downloader\GitDownloader->doDownload() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Downloader/VcsDownloader.php:59
 Composer\Downloader\VcsDownloader->download() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Downloader/DownloadManager.php:182
 Composer\Downloader\DownloadManager->download() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer/LibraryInstaller.php:155
 Composer\Installer\LibraryInstaller->installCode() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer/LibraryInstaller.php:86
 Composer\Installer\LibraryInstaller->install() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer/InstallationManager.php:153
 Composer\Installer\InstallationManager->install() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer/InstallationManager.php:140
 Composer\Installer\InstallationManager->execute() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer.php:510
 Composer\Installer->doInstall() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer.php:210
 Composer\Installer->run() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Command/UpdateCommand.php:118
 Composer\Command\UpdateCommand->execute() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:241
 Symfony\Component\Console\Command\Command->run() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:881
 Symfony\Component\Console\Application->doRunCommand() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:191
 Symfony\Component\Console\Application->doRun() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Console/Application.php:117
 Composer\Console\Application->doRun() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:121
 Symfony\Component\Console\Application->run() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Console/Application.php:83
 Composer\Console\Application->run() at phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/bin/composer:43
 require() at /usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar:14

it looks like the url '../../' should be realpath expanded at some point if it really is a local relative filesystem url. if that were the case i believe this command would succeed.

Contributor

stof commented Mar 14, 2014

if the repo is not a bare repository (and if the path is ../.., it is not likely to be a bare repo), your path should be ../../.git to point to the git repo itself, not to the working directory

aoberoi commented Mar 14, 2014

thanks @stof, but i'm still getting the same RuntimeException with that URL. (tried with "type":"vcs" and "type":"git")

laurentj commented Apr 5, 2014

Hi everybody,

I have the same issue here. let's say I have a project, using a framework, with modules (that are similar to Symfony bundles for example), and external libraries managed by Composer. In this (old) framework, there is an autoload system like in Composer, and a dependencies system like in Composer. In a near future, I would like to use Composer features with my framework instead of providing same components and reinventing the wheel.

So, my idea is to put some composer.json inside all modules so these modules could be considered as packages by Composer. Note that there are no reason to publish these packages in a repository (public or private repository), there could be very specific to a project. And it could be a waste of resources to have a private repository, and finally useless : everything is already in sources of the project.

Unfortunately, It seems there is no way to indicate to Composer: "include in my project this local package, which is in this directory (without copying it somewhere)". So we cannot benefit of the autoloader and the dependency system of Composer for the wall project : we have to provide a second autoloader, a second dependency system etc.. That's a shame...

Zenexer commented Jul 25, 2014

@Seldaek Most of Composer's users probably aren't deploying large, enterprise-grade applications. When using something like continuous integration with many application servers, Composer falls short. There doesn't seem to be a way to say, "Look in this (relative) folder for packages. I've already copied everything there for you."

artifact is currently the most acceptable repository type for that purpose, but it falls short when several packages are in development simultaneously. What if I want to test an update that spans several packages? I'd need separate artifacts. As the number of developers, issues, and branches grows, this quickly becomes unmanageable. A more reasonable solution would be to put all of the packages in a single repository, but Composer doesn't seem to support loading packages from relative paths.

Recently Composer added/fixed support for loading packages from relative subdirectories of git repositories (assuming the repository is already available locally). This solves the problem functionally, but it has one major caveat: if the project is divided into many small, modular, independent packages (each providing a single feature), the repository can easily be cloned 30+ times!

Edit: Never mind; subdirectories of location git repositories still don't work. They're closer to working, though.

boussou commented Aug 31, 2014

Hi, I got the same need, no solution?

jreinke commented Apr 2, 2015

Just gave a solution for this on another related issue: #1299 (comment)

Vittel commented Apr 6, 2015

@jreinke even though your solution might work for most people, symlinking files instead of just reading them from their original location is not a solution for this problem.

Contributor

sbuzonas commented Apr 7, 2015

I work with a handful of 100+ dependency projects so I'll give my two cents.

I'm not quite sure how well a relative path repository declaration would work in the context of a project. Suppose someone checks out your project and doesn't have the same path structure. It generally works better for large teams and CI to have your repositories declared elsewhere. The file protocol works in some circumstances, but if you're working with a VCS repo you need to commit your work for it to be updated anyhow, why not just push it to a central location on some experimental branch.

With development and testing uncommitted changes we use an alternative composer file that excludes the codependencies from the require section and places their relative paths in autoload-dev. This also needs to have the child dependencies to be placed as a direct dependency of the root package.

@laurentj Unfortunately, It seems there is no way to indicate to Composer: "include in my project this local package, which is in this directory (without copying it somewhere)".: the approach in the second paragraph does exactly that.

Vittel commented Apr 7, 2015

The main advantage here is not the "composition" part (since everything is in the place to be),
but letting composer generate one autoload-file for all projects that are required, without duplicating or symlinking any of the code into each other project.

just like: forget about the packages, i managed that. but please take care of the autoloading for me

i am playing with a local repository type.
companies like google and facebook are working this way, so it should scale =)

https://github.com/timglabisch/composer_local_downloader_demo

Something like this along with the ability to merge in a 'composer.local.json' config would be a huge boon to me and my department. We develop a number of applications and the packages they share simultaneously and composer simply doesn't support such a basic workflow. At the moment we have to resort to manual symlinks. We mainly use SVN so local git repos aren't really an option and even when we can do this, having to commit every change in a package and then compose update really isn't a great workflow.

Putting myself on the list of "people who needed this" :) I'd love to point a repository to a relative path.

Is this a solution to this problem?
Composer's repository 'path' type

For those of you who need this because you want to have projects and libraries in a monolithic repository, you can use Conductor, it seamlessly hooks into composer processes
https://github.com/mybuilder/conductor

Contributor

staabm commented Sep 23, 2015

@Seldaek as noted above, I guess this one can be closed.

Seldaek closed this Sep 23, 2015

Contributor

TomasVotruba commented Oct 23, 2016 edited

@claudio-silva It is! New feature in composer (in that time of your comment).

For others:

{
    "repositories": [
        {
            "type": "path",
            "url": "../../packages/my-package"
        }
    ],
    "require": {
        "my/package": "*"
    }
}

@TomasVotruba I have tried using your solution but I still get:

Cloning 7a1b88ba32 from cache

Shouldn't this actually be symlinking it like it says here:

https://getcomposer.org/doc/05-repositories.md#path

Even trying to force it, there seems to be no change:

"options": {
    "symlink": false
}
Contributor

TomasVotruba commented Mar 4, 2017

@stephan-v I'm not sure how "symlink" options works. I never used it and everything works fine for me.

Might be OS. I use Linux Ubuntu 16.10.

What about you?

@TomasVotruba nevermind I had to throw away my composer.lock file for this to work. Thanks for replying anyways 👍

Contributor

TomasVotruba commented Mar 4, 2017

@stephan-v Good to know. I will share this information next time I will see similar issue.

Thanks and have fun :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment