Allow to run a satis build for one repo/package #125

Merged
merged 8 commits into from May 1, 2014

Projects

None yet
@shochdoerfer
Contributor

Added an additional flag to generate the satis build for a single package. This helps to speed up the satis run if you have to deal with a lot of repos. In our case Jenkins will trigger the satis build after a new release was created. Since we have a lot packages (with a lot of versions) in our local satis repo, I was looking for a way to speed up the satis run.

@shochdoerfer shochdoerfer Added additional flag to generate the satis build for a single packag…
…e. This helps to speed up the satis run if you have a lot of repos.
498df5a
@stof stof and 1 other commented on an outdated diff Feb 25, 2014
src/Composer/Satis/Command/BuildCommand.php
@@ -50,6 +50,7 @@ protected function configure()
->setDefinition(array(
new InputArgument('file', InputArgument::OPTIONAL, 'Json file to use', './satis.json'),
new InputArgument('output-dir', InputArgument::OPTIONAL, 'Location where to output built files', null),
+ new InputArgument('filter-package', InputArgument::OPTIONAL, 'Run the build just for the given package', null),
@stof
stof Feb 25, 2014 Contributor

shouldn't this be an array argument to allow specifying several packages, like for composer update ?

@shochdoerfer
shochdoerfer Feb 25, 2014 Contributor

Valid point. In our case I only needed one package but obviously this could be extended to allow multiple packages as filters.

@stof
Contributor
stof commented Feb 25, 2014

your filtered package seems to be missing the exclusion of AliasPackage and stability

@shochdoerfer
Contributor

@stof thanks for pointing out. Will add that as well.

@shochdoerfer
Contributor

When can I expect the PR to be merged in? Or is some more work needed on my side?

@adamlc
adamlc commented Mar 24, 2014

👍 +1 on this.

We only really want Satis to update the individual projects that have passed the tests on our CI server.

@akimsko
akimsko commented Mar 28, 2014

This would be a much welcomed feature. Anything in the way of merging this?

@auipga
Contributor
auipga commented Apr 16, 2014

👍 +2 on this.

@jakeasmith

👍 this would be fantastic

@ralouphie

+1!

@Burgov
Burgov commented Apr 18, 2014

Big 👍!

@kmelia
kmelia commented Apr 22, 2014

I actually deal with more than 200 repositories and this will be helpfull.

But I see that the most time past during the build is for building packages.json (if no newer package are found) Did the pull request fix that ?

@shochdoerfer
Contributor

@kmelia That is a good question. Honestly I do not know. We only deal with 10ish repos. Would you mind testing it in your setup?

@kmelia
kmelia commented Apr 22, 2014

I patch my file vendor/composer/satis/src/Composer/Satis/Command/BuildCommand.php and I have run with just one package : but my packages.json did not keep the others packages. Is that normal ?

@naderman
Member

Indeed just looking at the code of this it would generate a packages.json only for the filtered packages, not update an existing one with the filtered package info?

@shochdoerfer
Contributor

@kmelia @naderman Thanks for pointing out. Seems I misinterpreted the results. Will work on the update part in the next couple of days.

@akimsko
akimsko commented Apr 24, 2014

@shochdoerfer I added a PR to your fork/branch that should solve this, and some other issues.
bitExpert#1

@shochdoerfer
Contributor

Thanks to @akimsko the packages.json file gets now generated correctly as well as the html file. @kmelia Would you mind testing the patch in your setup?

@naderman
Member

Unfortunately I recently merged a change to use includes in packages.json so I think this doesn't quite work correctly with that. Can you rebase onto latest master and adapt this branch?

@shochdoerfer
Contributor

@naderman I merged in the changes from master this morning and changed the code to make it work with it.

@naderman
Member

Oh sorry, I missed that :)

@naderman naderman merged commit 11c9b28 into composer:master May 1, 2014
@akimsko

Hmm, is it possible to leave a version of this block in? Removing it removes the possibility to use the package filter, when having the "require-all" flag set to false.

Thx for pointing out. Seems during my rebase the code got removed somehow. Will send a PR to satis to fix this issue.

The change just got merged. Could you please check if it works for you? If not please come back to me.

Seems to work as intended.

Thanks for the work you put into this feature - much appreciated :)

@mpdude
Contributor
mpdude commented Jul 10, 2014

I've been playing with this and was hoping that a such a single-package build would only fetch and re-scan the vcs repository for the given package.

In fact, the console output shows that satis iterates over all repositories again.

Am I missing anything or did I get this feature wrong?

@mpdude
Contributor
mpdude commented Jul 10, 2014

BuildCommand::selectPackages() will as a first step do

         $repos = $composer->getRepositoryManager()->getRepositories();
          $pool = new Pool($minimumStability);
          foreach ($repos as $repo) {
              try {
                  $pool->addRepository($repo);
              } ...
          }

... which will pull and re-scan all defined VcsRepositories even before the filter is applied.

@akimsko
akimsko commented Jul 10, 2014

Well no, you didnt get it wrong. But it kinda only solved half the problem - the actual package fetching.

TMK you can't select a repo based on package. The information just isn't available. You can only ask if the repo contains a specific package, wich potentially scans all repos to find one specific package.

I guess you could store which repo the package came from (in packages.json or somewhere else), and read from that. But in theory a package can change repo (and maybe even be split across multiple repos in different versions?). So a full scan is the only way to be sure you get all versions of the package.

Of course it could fall back to a full scan, if it can no longer find the package in the last repo it got it from. That would be fairly safe, and very rarely result in a full scan. But it would possibly create some edge cases where new versions wouldn't be found, for instance if the package moved repo, but didnt get removed from the old one.

@mpdude
Contributor
mpdude commented Jul 10, 2014

Ok, trying to get this straight...

This PR allows to update selectively by providing a package name. It will then re-scan all configured repositories, figure out which versions of the package(s) are available in all of them and then dumpDownload() only versions of this package.

In contrary, PR #110 together with jkufner/satis#1 takes a repo URL as parameter and only re-scans this given repo for available packages. It then merges that information with the full set of packages which is kept serialized in a cache.

What I am aiming for are quick updates by mean of a webhook, as in #40. The URL-based "scan only one repo"-approach seems more fit here.

@mpdude
Contributor
mpdude commented Jul 10, 2014

Looking at the code more closely:

  • The new $packagesFilter argument in selectPackages serves to limit the list of packages available for dumping (line 154).
  • Immediately afterwards (line 159ff) we need to "heal" this filtered package list by re-loading dumped package information.
  • From that point on, nothing differs in behaviour.

So - wouldn't it be easier to apply the filtering in dumpDownloads only? That would make a much simpler implementation of the same behaviour IMO.

@mpdude
mpdude commented on c7f8d20 Oct 12, 2014

What exactly is the purpose of this special handling of AliasPackages? Where does it come into play and what does it change for the final result?

Not exactly sure because we do not use the alias functionality. I would assume that aliases are obviously no real packages so they should not show up in the list of packages available to install.

Maybe @stof or @seldaek can explain when such AliasPackages occur? Hope I can cobble together some tests for Satis...

stof replied Oct 13, 2014

The most common case where they appear is for branch aliases

That is, when I scan a package that uses a branch-alias one should show up?

In that case, will Satis show the aliased or original name as the available version, or does that not make a difference here at all?

stof replied Oct 13, 2014

@mpdude yeah. And the alias package is skipped here, because the target of the alias contains the info to recreate the alias when loading it

@mpdude
Contributor
mpdude commented Nov 21, 2014

@shochdoerfer could you please try to explain how this filter is supposed to work together with the require and require-all configuration settings, require-depenencies and require-dev-dependencies?

Thanks!

@shochdoerfer
Contributor

@mpdude is my comment in #184 enough? Should the discussion continue there?

@h4cc

This is a relative path that will break if you run your satis from a subdir like http://example.com/satis/. Because then composer will not try to fetch http://example.com/satis/includes/all$somehash.json but http://example.com/includes/all$somehash.json which will result in a 404.

@h4cc Good catch. Would you mind creating a PR against the official composer/satis repo for it?

h4cc replied Jan 28, 2015

I created a issue for this: composer#201

Will try to fix this then. 👍

@shochdoerfer shochdoerfer deleted the bitExpert:feature/single_package_build branch Jan 11, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment