Skip to content


Subversion checkout URL

You can clone with
Download ZIP


Allow to run a satis build for one repo/package #125

merged 8 commits into from

Added an additional flag to generate the satis build for a single package. This helps to speed up the satis run if you have to deal with a lot of repos. In our case Jenkins will trigger the satis build after a new release was created. Since we have a lot packages (with a lot of versions) in our local satis repo, I was looking for a way to speed up the satis run.

@shochdoerfer shochdoerfer Added additional flag to generate the satis build for a single packag…
…e. This helps to speed up the satis run if you have a lot of repos.
@stof stof commented on an outdated diff
@@ -50,6 +50,7 @@ protected function configure()
new InputArgument('file', InputArgument::OPTIONAL, 'Json file to use', './satis.json'),
new InputArgument('output-dir', InputArgument::OPTIONAL, 'Location where to output built files', null),
+ new InputArgument('filter-package', InputArgument::OPTIONAL, 'Run the build just for the given package', null),
stof added a note

shouldn't this be an array argument to allow specifying several packages, like for composer update ?

Valid point. In our case I only needed one package but obviously this could be extended to allow multiple packages as filters.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

your filtered package seems to be missing the exclusion of AliasPackage and stability


@stof thanks for pointing out. Will add that as well.


When can I expect the PR to be merged in? Or is some more work needed on my side?


:+1: +1 on this.

We only really want Satis to update the individual projects that have passed the tests on our CI server.


This would be a much welcomed feature. Anything in the way of merging this?


:+1: +2 on this.


:+1: this would be fantastic


Big :+1:!


I actually deal with more than 200 repositories and this will be helpfull.

But I see that the most time past during the build is for building packages.json (if no newer package are found) Did the pull request fix that ?


@kmelia That is a good question. Honestly I do not know. We only deal with 10ish repos. Would you mind testing it in your setup?


I patch my file vendor/composer/satis/src/Composer/Satis/Command/BuildCommand.php and I have run with just one package : but my packages.json did not keep the others packages. Is that normal ?


Indeed just looking at the code of this it would generate a packages.json only for the filtered packages, not update an existing one with the filtered package info?


@kmelia @naderman Thanks for pointing out. Seems I misinterpreted the results. Will work on the update part in the next couple of days.


@shochdoerfer I added a PR to your fork/branch that should solve this, and some other issues.


Thanks to @akimsko the packages.json file gets now generated correctly as well as the html file. @kmelia Would you mind testing the patch in your setup?


Unfortunately I recently merged a change to use includes in packages.json so I think this doesn't quite work correctly with that. Can you rebase onto latest master and adapt this branch?


@naderman I merged in the changes from master this morning and changed the code to make it work with it.


Oh sorry, I missed that :)

@naderman naderman merged commit 11c9b28 into composer:master

Hmm, is it possible to leave a version of this block in? Removing it removes the possibility to use the package filter, when having the "require-all" flag set to false.

Thx for pointing out. Seems during my rebase the code got removed somehow. Will send a PR to satis to fix this issue.

The change just got merged. Could you please check if it works for you? If not please come back to me.

Seems to work as intended.

Thanks for the work you put into this feature - much appreciated :)


I've been playing with this and was hoping that a such a single-package build would only fetch and re-scan the vcs repository for the given package.

In fact, the console output shows that satis iterates over all repositories again.

Am I missing anything or did I get this feature wrong?


BuildCommand::selectPackages() will as a first step do

         $repos = $composer->getRepositoryManager()->getRepositories();
          $pool = new Pool($minimumStability);
          foreach ($repos as $repo) {
              try {
              } ...

... which will pull and re-scan all defined VcsRepositories even before the filter is applied.


Well no, you didnt get it wrong. But it kinda only solved half the problem - the actual package fetching.

TMK you can't select a repo based on package. The information just isn't available. You can only ask if the repo contains a specific package, wich potentially scans all repos to find one specific package.

I guess you could store which repo the package came from (in packages.json or somewhere else), and read from that. But in theory a package can change repo (and maybe even be split across multiple repos in different versions?). So a full scan is the only way to be sure you get all versions of the package.

Of course it could fall back to a full scan, if it can no longer find the package in the last repo it got it from. That would be fairly safe, and very rarely result in a full scan. But it would possibly create some edge cases where new versions wouldn't be found, for instance if the package moved repo, but didnt get removed from the old one.


Ok, trying to get this straight...

This PR allows to update selectively by providing a package name. It will then re-scan all configured repositories, figure out which versions of the package(s) are available in all of them and then dumpDownload() only versions of this package.

In contrary, PR #110 together with jkufner/satis#1 takes a repo URL as parameter and only re-scans this given repo for available packages. It then merges that information with the full set of packages which is kept serialized in a cache.

What I am aiming for are quick updates by mean of a webhook, as in #40. The URL-based "scan only one repo"-approach seems more fit here.


Looking at the code more closely:

  • The new $packagesFilter argument in selectPackages serves to limit the list of packages available for dumping (line 154).
  • Immediately afterwards (line 159ff) we need to "heal" this filtered package list by re-loading dumped package information.
  • From that point on, nothing differs in behaviour.

So - wouldn't it be easier to apply the filtering in dumpDownloads only? That would make a much simpler implementation of the same behaviour IMO.


What exactly is the purpose of this special handling of AliasPackages? Where does it come into play and what does it change for the final result?

Not exactly sure because we do not use the alias functionality. I would assume that aliases are obviously no real packages so they should not show up in the list of packages available to install.

Maybe @stof or @seldaek can explain when such AliasPackages occur? Hope I can cobble together some tests for Satis...

The most common case where they appear is for branch aliases

That is, when I scan a package that uses a branch-alias one should show up?

In that case, will Satis show the aliased or original name as the available version, or does that not make a difference here at all?

@mpdude yeah. And the alias package is skipped here, because the target of the alias contains the info to recreate the alias when loading it


@shochdoerfer could you please try to explain how this filter is supposed to work together with the require and require-all configuration settings, require-depenencies and require-dev-dependencies?



@mpdude is my comment in #184 enough? Should the discussion continue there?


This is a relative path that will break if you run your satis from a subdir like Because then composer will not try to fetch$somehash.json but$somehash.json which will result in a 404.

@h4cc Good catch. Would you mind creating a PR against the official composer/satis repo for it?

I created a issue for this: composer#201

Will try to fix this then. :+1:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 25, 2014
  1. @shochdoerfer

    Added additional flag to generate the satis build for a single packag…

    shochdoerfer committed
    …e. This helps to speed up the satis run if you have a lot of repos.
  2. @shochdoerfer
Commits on Feb 26, 2014
  1. @shochdoerfer

    Added handling for multiple packages in the filtering process. You ca…

    shochdoerfer committed
    …n now pass a list of packages that satis should built the repos for.
Commits on Apr 24, 2014
  1. @akimsko
Commits on Apr 25, 2014
  1. @shochdoerfer

    Optimized the generation of the packages.json and index.html in case …

    shochdoerfer committed
    …satis is run for a single package.
Commits on Apr 26, 2014
  1. @shochdoerfer
  2. @shochdoerfer
  3. @shochdoerfer

    Merged in the latest changes from master and changed the logic to rel…

    shochdoerfer committed
    …y on the includes functionality introduced by #132.
This page is out of date. Refresh to see the latest.
Showing with 66 additions and 5 deletions.
  1. +66 −5 src/Composer/Satis/Command/BuildCommand.php
71 src/Composer/Satis/Command/BuildCommand.php
@@ -12,6 +12,7 @@
namespace Composer\Satis\Command;
+use Composer\Package\Loader\ArrayLoader;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Input\InputArgument;
@@ -50,6 +51,7 @@ protected function configure()
new InputArgument('file', InputArgument::OPTIONAL, 'Json file to use', './satis.json'),
new InputArgument('output-dir', InputArgument::OPTIONAL, 'Location where to output built files', null),
+ new InputArgument('packages', InputArgument::IS_ARRAY | InputArgument::OPTIONAL, 'Packages that should be built, if not provided all packages are.', null),
new InputOption('no-html-output', null, InputOption::VALUE_NONE, 'Turn off HTML view'),
new InputOption('skip-errors', null, InputOption::VALUE_NONE, 'Skip Download or Archive errors'),
@@ -101,6 +103,7 @@ protected function execute(InputInterface $input, OutputInterface $output)
$verbose = $input->getOption('verbose');
$configFile = $input->getArgument('file');
+ $packagesFilter = $input->getArgument('packages');
$skipErrors = (bool)$input->getOption('skip-errors');
if (preg_match('{^https?://}i', $configFile)) {
@@ -141,7 +144,7 @@ protected function execute(InputInterface $input, OutputInterface $output)
$composer = $this->getApplication()->getComposer(true, $config);
- $packages = $this->selectPackages($composer, $output, $verbose, $requireAll, $requireDependencies, $requireDevDependencies, $minimumStability, $skipErrors);
+ $packages = $this->selectPackages($composer, $output, $verbose, $requireAll, $requireDependencies, $requireDevDependencies, $minimumStability, $skipErrors, $packagesFilter);
if ($htmlView = !$input->getOption('no-html-output')) {
$htmlView = !isset($config['output-html']) || $config['output-html'];
@@ -152,6 +155,15 @@ protected function execute(InputInterface $input, OutputInterface $output)
$filenamePrefix = $outputDir.'/include/all';
+ $filename = $outputDir.'/packages.json';
+ if(!empty($packagesFilter)) {
+ // in case of an active package filter we need to load the dumped packages.json and merge the
+ // updated packages in
+ $oldPackages = $this->loadDumpedPackages($filename, $packagesFilter);
+ $packages += $oldPackages;
+ ksort($packages);
+ }
$packageFile = $this->dumpPackageIncludeJson($packages, $output, $filenamePrefix);
$packageFileHash = hash_file('sha1', $packageFile);
@@ -159,7 +171,6 @@ protected function execute(InputInterface $input, OutputInterface $output)
'include/all$'.$packageFileHash.'.json' => array( 'sha1'=>$packageFileHash ),
- $filename = $outputDir.'/packages.json';
$this->dumpPackagesJson($includes, $output, $filename);
if ($htmlView) {
@@ -176,7 +187,7 @@ protected function execute(InputInterface $input, OutputInterface $output)
- private function selectPackages(Composer $composer, OutputInterface $output, $verbose, $requireAll, $requireDependencies, $requireDevDependencies, $minimumStability, $skipErrors)
+ private function selectPackages(Composer $composer, OutputInterface $output, $verbose, $requireAll, $requireDependencies, $requireDevDependencies, $minimumStability, $skipErrors, array $packagesFilter = array())
$selected = array();
@@ -198,6 +209,7 @@ private function selectPackages(Composer $composer, OutputInterface $output, $ve
if ($requireAll) {
$links = array();
+ $filterForPackages = count($packagesFilter) > 0;
foreach ($repos as $repo) {
// collect links for composer repos with providers
@@ -206,8 +218,18 @@ private function selectPackages(Composer $composer, OutputInterface $output, $ve
$links[] = new Link('__root__', $name, new MultiConstraint(array()), 'requires', '*');
} else {
- // process other repos directly
- foreach ($repo->getPackages() as $package) {
+ $packages = array();
+ if($filterForPackages) {
+ // apply package filter if defined
+ foreach ($packagesFilter as $filter) {
+ $packages += $repo->findPackages($filter);
+ }
+ } else {
+ // process other repos directly
+ $packages = $repo->getPackages();
+ }
+ foreach ($packages as $package) {
// skip aliases
if ($package instanceof AliasPackage) {
@@ -425,6 +447,45 @@ private function dumpWeb(array $packages, OutputInterface $output, PackageInterf
file_put_contents($directory.'/index.html', $content);
+ private function loadDumpedPackages($filename, array $packagesFilter = array())
+ {
+ $packages = array();
+ $repoJson = new JsonFile($filename);
+ $dirName = dirname($filename);
+ if ($repoJson->exists()) {
+ $loader = new ArrayLoader();
+ $jsonIncludes = $repoJson->read();
+ $jsonIncludes = isset($jsonIncludes['includes']) && is_array($jsonIncludes['includes'])
+ ? $jsonIncludes['includes']
+ : array();
+ foreach ($jsonIncludes as $includeFile => $includeConfig) {
+ $includeJson = new JsonFile($dirName . '/' . $includeFile);
+ $jsonPackages = $includeJson->read();
+ $jsonPackages = isset($jsonPackages['packages']) && is_array($jsonPackages['packages'])
+ ? $jsonPackages['packages']
+ : array();
+ foreach ($jsonPackages as $jsonPackage) {
+ if (is_array($jsonPackage)) {
+ foreach ($jsonPackage as $jsonVersion) {
+ if (is_array($jsonVersion)) {
+ if(isset($jsonVersion['name']) && in_array($jsonVersion['name'], $packagesFilter)) {
+ continue;
+ }
+ $package = $loader->load($jsonVersion);
+ $packages[$package->getUniqueName()] = $package;
+ }
+ }
+ }
+ }
+ }
+ }
+ return $packages;
+ }
private function getMappedPackageList(array $packages)
$groupedPackages = $this->groupPackagesByName($packages);
Something went wrong with that request. Please try again.