Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Distributions/Archives #1567

Merged
merged 43 commits into from
@naderman
Owner

This pull requests adds an archive command to composer which allows generating zip/tar archives for a given package in a given version.

It's based on the work in #1024 which was based on #804.

@naderman naderman referenced this pull request
Closed

Feature dist (bis) #1024

@naderman
Owner

This version no longer uses git/hg to archive, but entirely relies on phar data to create a zip file or a tar archive. At the same time support for archive exclude patterns using the same syntax as gitignore has been added. They can be specified as a list of patterns in composer.json. For example as in the documentation included in the patch:

{
    "archive": {
        "exclude": ["/foo/bar", "baz", "/*.test", "!/foo/bar/baz"]
    }
}

The example will include /dir/foo/bar/file, /foo/bar/baz, /file.php,
/foo/my.test but it will exclude /foo/bar/any, /foo/baz, and /my.test.

All VCS files are also excluded when creating an archive as previously requested.

@naderman
Owner

This issue also fixes #57

@asm89

This creates an archive only for the current package. Maybe it would be interesting to be able to package the vendors too? I'm thinking about deployment distributions.

Edit: ah, I now read the issue this fixes and maybe it's not a great idea then.

src/Composer/Command/ArchiveCommand.php
((12 lines not shown))
+
+namespace Composer\Command;
+
+use Composer\Factory;
+use Composer\IO\IOInterface;
+use Composer\DependencyResolver\Pool;
+use Composer\Package\LinkConstraint\VersionConstraint;
+use Composer\Repository\CompositeRepository;
+
+use Symfony\Component\Console\Input\InputArgument;
+use Symfony\Component\Console\Input\InputInterface;
+use Symfony\Component\Console\Input\InputOption;
+use Symfony\Component\Console\Output\OutputInterface;
+
+/**
+ * Install a package as new project into new directory.
@asm89
asm89 added a note

;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/Composer/Package/Package.php
@@ -525,4 +526,22 @@ public function getNotificationUrl()
{
return $this->notificationUrl;
}
+
+ /**
+ * Sets a list of patterns to be excluded from archives
+ *
+ * @param array $excludes
+ */
+ public function setArchiveExcludes($excludes)
@asm89
asm89 added a note

array typehint here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@Seldaek
Owner

According to http://marc.info/?l=php-internals&m=135982159620513&w=2 it might be interesting to use phar files as archives instead of zips (or at least allow it so we can switch to that at some point), and then eventually optionally installing dependencies as phars could be quite neat, if zend optimizer really makes it into php core for 5.5.

res/composer-schema.json
@@ -157,6 +157,20 @@
}
}
},
+ "archive": {
+ "type": ["object"],
+ "description": "Options for creating package archives for distribution.",
+ "properties": {
+ "exclude": {
+ "type": "array",
+ "description": "A list of paths to exclude."
+ },
+ "include": {
@stof
stof added a note

This property is not supported by the code

@naderman Owner

Whoops indeed, forgot to change that after changing exclude to patterns that alow inclusion like gitignore files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@stof

This will force to duplicate the ignore rules between the .gitignore and the composer.json files. And it will make the zip archives inconsistent between the one provided by github (when using the GithubDriver for custom repositories) and the one provided by packagist or satis (if they start creating their own archives)

@naderman
Owner

Not sure what you are referring to with dev versions.

As for gitignore. The plan from the previous PR was to add support for .gitignore / .hgignore etc. - problem I have at the moment is that I'm not sure how to handle those. Either load them before the composer.json rules, or only load them if you don't have any composer.json rules. Also what to load if there are files for multiple vcs?

@naderman
Owner

The github driver can easily be fixed to remove excluded files after/during extraction.

@Seldaek
Owner

@naderman that'd have to be done in the ArchiveDownloader rather. But anyway overall it's probably not a huge deal if dev packages taken from github have extra files around. If they are really not supposed to be there you can add them to the .gitattributes as well.

@stof

The github driver is not an installer. And the installer should install the dist as is IMO.

@Seldaek this is also for tags, not only dev versions. And it can indeed be fixed through .gitattributes, but it still requires duplication.

@Seldaek
Owner
@naderman
Owner

Going to be tricky to support the glob/regex mode switches

@till

@naderman I am gonna start integrating your branch with satis. I guess I'll stay up to date. Have not lost hope that one day this will get into composer. In case you need help on this (aside from hg — cause I have no idea), please let me know. I'll make some time this weekend.

@till

Here's a small fix (let me know if you'd rather want a PR to your branch):

diff --git a/src/Composer/Package/Archiver/ArchiveManager.php b/src/Composer/Package/Archiver/ArchiveManager.php
index 66aefdb..52b5163 100644
--- a/src/Composer/Package/Archiver/ArchiveManager.php
+++ b/src/Composer/Package/Archiver/ArchiveManager.php
@@ -95,6 +95,6 @@ class ArchiveManager

         // Create the archive
         $sourceRef = $package->getSourceReference();
-        $usableArchiver->archive($sourcePath, $target, $format, $sourceRef, $package->getArchiveExcludes());
+        return $usableArchiver->archive($sourcePath, $target, $format, $sourceRef, $package->getArchiveExcludes());
     }
 }
diff --git a/src/Composer/Package/Archiver/PharArchiver.php b/src/Composer/Package/Archiver/PharArchiver.php
index 950832c..071424b 100644
--- a/src/Composer/Package/Archiver/PharArchiver.php
+++ b/src/Composer/Package/Archiver/PharArchiver.php
@@ -60,6 +60,7 @@ class PharArchiver implements ArchiverInterface
                 })
                 ->ignoreVCS(true);
             $phar->buildFromIterator($finder->getIterator(), $sources);
+            return $target;
         } catch (\UnexpectedValueException $e) {
             $message = sprintf("Could not create archive '%s' from '%s': %s",
                 $target,
@till

@naderman

Tested your latest changes and there is a small regression.

Before:

-rw-r--r-- 1 till staff 14K Feb 25 16:41 till-wtfdate-0.1.0.zip
-rw-r--r-- 1 till staff 16K Feb 25 16:41 till-wtfdate-0.1.1.zip
-rw-r--r-- 1 till staff 16K Feb 25 16:41 till-wtfdate-dev-master.zip

After:

-rw-r--r-- 1 till staff 14K Feb 25 16:45 till-wtfdate-0.1.0-0.1.0-zip.zip
-rw-r--r-- 1 till staff 16K Feb 25 16:45 till-wtfdate-0.1.1-0.1.1-zip.zip
-rw-r--r-- 1 till staff 16K Feb 25 16:45 till-wtfdate-dev-master-0.1.1-zip.zip

@Seldaek
Owner

@till is the regression that the stuff is duplicated in the name? That seems good to me, if it's version + reference. We need both for uniqueness.

@till

@Seldaek I didn't know. Is -zip.zip also wanted/needed?

@Seldaek
Owner

@till

@naderman @Seldaek

Do you guys have any thoughts if it would make sense to skip re-generating tags? Because right now it deletes the target if it exists.

@Seldaek
Owner

IMO if the file exists (and that's why we need the reference in there) already there is no point in regenerating it since git creates a new ref when anything changes even force pushes shouldn't be an issue.

@naderman
Owner

Well that unlink was unecessary to begin with however the command will now overwrite files if they already exist. I would say that's the expected behaviour as pretty much any tool writing to a file will overwrite it if it already exists. I'd rather add some flag one can use to skip files that already exist for usage in satis.You ok with that @till ?

@till

@naderman That sounds perfect.

@till

I am playing around with the replace and will hopefully share a patch shortly.

There's a small edge-case I ran into:

  • this creates packages of branches (aka dev-foo) as well
  • branches may or may not changes

Solutions:

  • do not create archives for branches
  • create archives but overwrite always
  • come up with another solution (e.g. including commit/rev in the name)?

Any ideas?

@Seldaek
Owner
@till

@Seldaek By reference you mean actual commit hash?

@till

Maybe getSourceReference()?

@Seldaek
Owner

Yes, like what the FileDownloader is using to get a cache key (same sort of problem, unique file name for zips): https://github.com/composer/composer/blob/master/src/Composer/Downloader/FileDownloader.php#L211-218

@till

Looks like the $sourceRef is already present but not used.

@till

Only problem I see is how to determine when a zip is stale. Like, we could just add this to the filename when it's a dev- or maybe even to each and every filename to accomodate people who like to delete and recreate tags. ;) But then we may end up with a bunch of old dist files.

@stof

@till You need these old dist files, as they could be referenced by lock files

@till

@stof I guess. Possibly. :)

diff --git a/src/Composer/Package/Archiver/ArchiveManager.php b/src/Composer/Package/Archiver/ArchiveManager.php
index ef10bdc..3028f14 100644
--- a/src/Composer/Package/Archiver/ArchiveManager.php
+++ b/src/Composer/Package/Archiver/ArchiveManager.php
@@ -30,6 +30,11 @@ class ArchiveManager
     protected $archivers = array();

     /**
+     * @var bool
+     */
+    protected $replace = true;
+
+    /**
      * @param DownloadManager $downloadManager A manager used to download package sources
      */
     public function __construct(DownloadManager $downloadManager)
@@ -46,6 +51,17 @@ class ArchiveManager
     }

     /**
+     * Disable overwriting archives which already exist.
+     *
+     * @return $this
+     */
+    public function disableReplace()
+    {
+        $this->replace = false;
+        return $this;
+    }
+
+    /**
      * Generate a distinct filename for a particular version of a package.
      *
      * @param PackageInterface $package The package to get a name for
@@ -62,10 +78,12 @@ class ArchiveManager
             $nameParts = array_merge($nameParts, array($package->getPrettyVersion(), $package->getDistReference()));
         }

-        return implode('-', array_filter($nameParts, function ($p) {
+        $name = implode('-', array_filter($nameParts, function ($p) {
             return !empty($p);
         }));

+        $name .= '-' . substr(sha1($package->getSourceReference()), 0, 6);
+        return $name;
     }

     /**
@@ -117,7 +135,6 @@ class ArchiveManager
         }

         // Create the archive
-        $sourceRef = $package->getSourceReference();
-        return $usableArchiver->archive($sourcePath, $target, $format, $sourceRef, $package->getArchiveExcludes());
+        return $usableArchiver->archive($sourcePath, $target, $format, $package->getArchiveExcludes(), $this->replace);
     }
 }
diff --git a/src/Composer/Package/Archiver/ArchiverInterface.php b/src/Composer/Package/Archiver/ArchiverInterface.php
index ce9f677..f12d68f 100644
--- a/src/Composer/Package/Archiver/ArchiverInterface.php
+++ b/src/Composer/Package/Archiver/ArchiverInterface.php
@@ -26,13 +26,12 @@ interface ArchiverInterface
      * @param string $sources   The sources directory
      * @param string $target    The target file
      * @param string $format    The format used for archive
-     * @param string $sourceRef The reference of the source to archive or null
-     *                          for the current reference
      * @param array  $excludes  A list of patterns for files to exclude
+     * @param bool   $replace   Replace the archive if it already exists.
      *
      * @return string The path to the written archive file
      */
-    public function archive($sources, $target, $format, $sourceRef = null, $excludes = array());
+    public function archive($sources, $target, $format, $excludes = array(), $replace = true);

     /**
      * Format supported by the archiver.
diff --git a/src/Composer/Package/Archiver/PharArchiver.php b/src/Composer/Package/Archiver/PharArchiver.php
index 4dfd6f0..10eead4 100644
--- a/src/Composer/Package/Archiver/PharArchiver.php
+++ b/src/Composer/Package/Archiver/PharArchiver.php
@@ -32,12 +32,19 @@ class PharArchiver implements ArchiverInterface
     /**
      * {@inheritdoc}
      */
-    public function archive($sources, $target, $format, $sourceRef = null, $excludes = array())
+    public function archive($sources, $target, $format, $excludes = array(), $replace = true)
     {
         $sources = realpath($sources);

         $excludePatterns = $this->generatePatterns($excludes);

+        if (true !== $replace) {
+            if (true === file_exists($target)) {
+                echo "Skip: {$target}" . PHP_EOL;
+                return $target;
+            }
+        }
+
         try {
             $phar = new \PharData($target, null, null, static::$formats[$format]);
             $finder = new Finder\Finder();
@till

@naderman This is against your branch.

@till
diff --git a/src/Composer/Package/Archiver/ArchiveManager.php b/src/Composer/Package/Archiver/ArchiveManager.php
index ef10bdc..b8d398b 100644
--- a/src/Composer/Package/Archiver/ArchiveManager.php
+++ b/src/Composer/Package/Archiver/ArchiveManager.php
@@ -30,6 +30,11 @@ class ArchiveManager
     protected $archivers = array();

     /**
+     * @var bool
+     */
+    protected $replace = true;
+
+    /**
      * @param DownloadManager $downloadManager A manager used to download package sources
      */
     public function __construct(DownloadManager $downloadManager)
@@ -46,6 +51,17 @@ class ArchiveManager
     }

     /**
+     * Disable overwriting archives which already exist.
+     *
+     * @return $this
+     */
+    public function disableReplace()
+    {
+        $this->replace = false;
+        return $this;
+    }
+
+    /**
      * Generate a distinct filename for a particular version of a package.
      *
      * @param PackageInterface $package The package to get a name for
@@ -62,10 +78,12 @@ class ArchiveManager
             $nameParts = array_merge($nameParts, array($package->getPrettyVersion(), $package->getDistReference()));
         }

-        return implode('-', array_filter($nameParts, function ($p) {
+        $name = implode('-', array_filter($nameParts, function ($p) {
             return !empty($p);
         }));

+        $name .= '-' . substr(sha1($package->getSourceReference()), 0, 6);
+        return $name;
     }

     /**
@@ -105,6 +123,12 @@ class ArchiveManager
         $target = realpath($targetDir).'/'.$packageName.'.'.$format;
         $filesystem->ensureDirectoryExists(dirname($target));

+        if (true !== $this->replace) {
+            if (file_exists($target)) {
+                return $target;
+            }
+        }
+
         if ($package instanceof RootPackage) {
             $sourcePath = realpath('.');
         } else {
@@ -117,7 +141,6 @@ class ArchiveManager
         }

         // Create the archive
-        $sourceRef = $package->getSourceReference();
-        return $usableArchiver->archive($sourcePath, $target, $format, $sourceRef, $package->getArchiveExcludes());
+        return $usableArchiver->archive($sourcePath, $target, $format, $package->getArchiveExcludes());
     }
 }
diff --git a/src/Composer/Package/Archiver/ArchiverInterface.php b/src/Composer/Package/Archiver/ArchiverInterface.php
index ce9f677..ffc93e4 100644
--- a/src/Composer/Package/Archiver/ArchiverInterface.php
+++ b/src/Composer/Package/Archiver/ArchiverInterface.php
@@ -26,13 +26,11 @@ interface ArchiverInterface
      * @param string $sources   The sources directory
      * @param string $target    The target file
      * @param string $format    The format used for archive
-     * @param string $sourceRef The reference of the source to archive or null
-     *                          for the current reference
      * @param array  $excludes  A list of patterns for files to exclude
      *
      * @return string The path to the written archive file
      */
-    public function archive($sources, $target, $format, $sourceRef = null, $excludes = array());
+    public function archive($sources, $target, $format, $excludes = array());

     /**
      * Format supported by the archiver.
diff --git a/src/Composer/Package/Archiver/PharArchiver.php b/src/Composer/Package/Archiver/PharArchiver.php
index 4dfd6f0..6ee1156 100644
--- a/src/Composer/Package/Archiver/PharArchiver.php
+++ b/src/Composer/Package/Archiver/PharArchiver.php
@@ -32,7 +32,7 @@ class PharArchiver implements ArchiverInterface
     /**
      * {@inheritdoc}
      */
-    public function archive($sources, $target, $format, $sourceRef = null, $excludes = array())
+    public function archive($sources, $target, $format, $excludes = array())
     {
         $sources = realpath($sources);
@till

So, to test this:

{
    "repositories": [{
        "type": "vcs", "url": "https://github.com/doctrine/common/"
    }]
}

After it generated all zips for it, I am left with this when I try to extract one of them:

Archive:  doctrine-common-2.2.0-2.2.0-595d55.zip
 extracting: build.properties        
 extracting: build.xml               
 extracting: composer.json           
 extracting: lib                     
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/Annotation.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/Annotation/Attribute.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/Annotation/Attributes.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/Annotation/IgnoreAnnotation.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/Annotation/Required.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/Annotation/Target.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/Annotation.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/AnnotationException.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/AnnotationReader.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/AnnotationRegistry.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/CachedReader.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/DocLexer.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/DocParser.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/FileCacheReader.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/IndexedReader.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/PhpParser.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/Reader.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Annotations/SimpleAnnotationReader.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Cache.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Cache/ApcCache.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Cache/ArrayCache.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Cache/Cache.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Cache/CacheProvider.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Cache/MemcacheCache.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Cache/MemcachedCache.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Cache/WinCacheCache.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Cache/XcacheCache.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Cache/ZendDataCache.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/ClassLoader.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Collections.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Collections/ArrayCollection.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Collections/Collection.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/CommonException.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Comparable.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/EventArgs.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/EventManager.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/EventSubscriber.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Lexer.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/NotifyPropertyChanged.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/AbstractManagerRegistry.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/ConnectionRegistry.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Event.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Event/LifecycleEventArgs.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Event/LoadClassMetadataEventArgs.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Event/ManagerEventArgs.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Event/OnClearEventArgs.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Event/PreUpdateEventArgs.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/ManagerRegistry.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/AbstractClassMetadataFactory.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/ClassMetadata.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/ClassMetadataFactory.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/Driver.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/Driver/AnnotationDriver.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/Driver/DefaultFileLocator.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/Driver/FileDriver.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/Driver/FileLocator.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/Driver/MappingDriver.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/Driver/MappingDriverChain.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/Driver/PHPDriver.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/Driver/StaticPHPDriver.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/Driver/SymfonyFileLocator.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/MappingException.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/ReflectionService.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/RuntimeReflectionService.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Mapping/StaticReflectionService.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/ObjectManager.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/ObjectManagerAware.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/ObjectRepository.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/PersistentObject.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Persistence/Proxy.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/PropertyChangedListener.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Util.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Util/ClassUtils.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Util/Debug.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Util/Inflector.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/Doctrine/Common/Version.php.
checkdir error:  lib exists but is not directory
                 unable to process lib/vendor.
checkdir error:  lib exists but is not directory
                 unable to process lib/vendor/doctrine-build-common.
 extracting: LICENSE                 
 extracting: phpunit.xml.dist        
 extracting: README.md               
 extracting: tests                   
checkdir error:  tests exists but is not directory
                 unable to process tests/Doctrine.
checkdir error:  tests exists but is not directory
                 unable to process tests/Doctrine/Tests.
checkdir error:  tests exists but is not directory
                 unable to process tests/Doctrine/Tests/Common.
checkdir error:  tests exists but is not directory
                 unable to process tests/Doctrine/Tests/Common/Annotations.
checkdir error:  tests exists but is not directory
                 unable to process tests/Doctrine/Tests/Common/Annotations/AbstractReaderTest.php.
checkdir error:  tests exists but is not directory
                 unable to process tests/Doctrine/Tests/Common/Annotations/AnnotationReaderTest.php.
checkdir error:  tests exists but is not directory
                 unable to process tests/Doctrine/Tests/Common/Annotations/CachedReaderTest.php.
checkdir error:  tests exists but is not directory
                 unable to process tests/Doctrine/Tests/Common/Annotations/DocLexerTest.php.
checkdir error:  tests exists but is not directory
                 unable to process tests/Doctrine/Tests/Common/Annotations/DocParserTest.php.
checkdir error:  tests exists but is not directory
@till

When I use tar instead, it works beautifully and I can untar it without errors.

Maybe my PHP is missing something we should test for? I see both zlib and zip in php -m though.

@naderman
Owner

Absolutely no idea, maybe search if there are any bugs against filed against PharData that could cause this?

@ralphschindler

The error means that inside the zip the entry for lib did not have a trailing slash. How was the zip originally created?

@till

@ralphschindler PharData — the code in this branch.

@till

I think PharData from a symfony2 finder something collection.

src/Composer/Package/Archiver/PharArchiver.php
((44 lines not shown))
+ $finder
+ ->in($sources)
+ ->filter(function (\SplFileInfo $file) use ($sources, $excludePatterns) {
+ $relativePath = preg_replace('#^'.preg_quote($sources, '#').'#', '', $file->getRealPath());
+
+ $include = true;
+ foreach ($excludePatterns as $patternData) {
+ list($pattern, $negate) = $patternData;
+ if (preg_match($pattern, $relativePath)) {
+ $include = $negate;
+ }
+ }
+ return $include;
+ })
+ ->ignoreVCS(true);
+ $phar->buildFromIterator($finder->getIterator(), $sources);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@till

@naderman I'll take a look at the finder thing, did you manage to push your exclude work for hg?

@till

I traced this back to Symfony\Component\Finder\SplFileInfo where $relativePath doesn't have a trailing slash. Probably for a good reason. But here I give up.

@ralphschindler

Tar directories really should have a trailing space if they are being added manually. Thats all I know :)

@till

@ralphschindler I think you mean .zip? Because .tar works beautifully.

But anyway, I have no idea in which of the 100s of classes I have to patch/add this. Maybe someone with more symfony2-fu can figure this out though.

@ralphschindler

Sorry, I did mean zip- got tar on the mind.

@nfx

This feature is practically useful in enterprise setups.

It could give input for PR #1728

@naderman
Owner

Alright this should be ready for a merge now

  • implemented support for reading .gitignore, .gitattributes and .hgignore
  • added api docs
  • fixed the zip bug by skipping directories altogether (same as git currently does, so behaviour is the same as github zips)
@till

:+1:

@Seldaek Any thoughts? ;)

src/Composer/Command/ArchiveCommand.php
((52 lines not shown))
+ )
+ ;
+ }
+
+ protected function execute(InputInterface $input, OutputInterface $output)
+ {
+ return $this->archive(
+ $this->getIO(),
+ $input->getArgument('package'),
+ $input->getArgument('version'),
+ $input->getOption('format'),
+ $input->getOption('dir')
+ );
+ }
+
+ public function archive(IOInterface $io, $packageName = false, $version = false, $format = 'tar', $dest = '.')
@stof
stof added a note

Wouldn't null be better than false for optional values ?

@stof
stof added a note

btw, why is it public ? If the goal is to make it reusable outside the command, it should be moved outside the command. Because currently, you would also have to build an Application to get the command from it as the code depends on it (because of the $this->getComposer() call) so it is not easily reusable.

@naderman Owner

Fixed both of these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/Composer/Command/ArchiveCommand.php
((100 lines not shown))
+ $repos = new CompositeRepository(array_merge(array($localRepo), $composer->getRepositoryManager()->getRepositories()));
+ } else {
+ $defaultRepos = Factory::createDefaultRepositories($this->getIO());
+ $output->writeln('No composer.json found in the current directory, searching packages from ' . implode(', ', array_keys($defaultRepos)));
+ $repos = new CompositeRepository($defaultRepos);
+ }
+
+ $pool = new Pool();
+ $pool->addRepository($repos);
+
+ $constraint = ($version) ? new VersionConstraint('>=', $version) : null;
+ $packages = $pool->whatProvides($packageName, $constraint);
+
+ if (count($packages) > 1) {
+ $package = $packages[0];
+ $io->write('<info>Found multiple matches, selected '.$package.'.</info>');
@stof
stof added a note

$package is an object here, not a string. You should use getPrettyString

@naderman Owner

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@stof stof commented on the diff
src/Composer/Package/Archiver/ArchiveManager.php
((99 lines not shown))
+ * @return string The path of the created archive
+ */
+ public function archive(PackageInterface $package, $format, $targetDir)
+ {
+ if (empty($format)) {
+ throw new \InvalidArgumentException('Format must be specified');
+ }
+
+ // Search for the most appropriate archiver
+ $usableArchiver = null;
+ foreach ($this->archivers as $archiver) {
+ if ($archiver->supports($format, $package->getSourceType())) {
+ $usableArchiver = $archiver;
+ break;
+ }
+ }
@stof
stof added a note

Do we still need this logic to choose the right archiver ? We don't have multiple archivers anymore

@naderman Owner

I would like to keep it so we can add alternatives easily

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/Composer/Package/Archiver/ExcludeFilterBase.php
((5 lines not shown))
+ *
+ * (c) Nils Adermann <naderman@naderman.de>
+ * Jordi Boggiano <j.boggiano@seld.be>
+ *
+ * For the full copyright and license information, please view the LICENSE
+ * file that was distributed with this source code.
+ */
+
+namespace Composer\Package\Archiver;
+
+use Symfony\Component\Finder;
+
+/**
+ * @author Nils Adermann <naderman@naderman.de>
+ */
+abstract class ExcludeFilterBase
@stof
stof added a note

It should be named BaseExcludeFilter to be consistent (we have BasePackage for instance)

@naderman Owner

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/Composer/Package/Archiver/PharArchiver.php
((10 lines not shown))
+ * file that was distributed with this source code.
+ */
+
+namespace Composer\Package\Archiver;
+
+use Composer\Package\BasePackage;
+use Composer\Package\PackageInterface;
+
+/**
+ * @author Till Klampaeckel <till@php.net>
+ * @author Nils Adermann <naderman@naderman.de>
+ * @author Matthieu Moquet <matthieu@moquet.net>
+ */
+class PharArchiver implements ArchiverInterface
+{
+ static protected $formats = array(
@stof
stof added a note

should be protected static to follow PSR-2

@naderman Owner

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@naderman
Owner

Fixed everything stof pointed out.

MattKetmo and others added some commits
@MattKetmo MattKetmo Refactored the archiver package 20e717f
@till till Initial feature-dist
 * extends BaseDumper, implements interface
 * put $keys into BaseDumper

 * WIP WIP WIP WIP
 * BaseDumper for utilities
 * interface to enforce 'dump()'
 * feature:
   * supports git
   * supports zip output
   * basic test to cover feature

 * add @todo for later
 * add vendor namespace to package name

 * add extension to getFilename() so we don't need to switch in there (HT, @naderman)

 * add extension (obviously 'zip' in ZipDumper)

 * create archive in destination dir (provided by __construct())

 * condensed ZipDumper
 * moved code to BaseDumper (hopefully easier re-use)

 * use ProcessExecutor from BaseDumper

 * fix assignments in __construct()
 * allow injection of ProcessExecutor

 * fix parameters

 * fix regex

 * write in 'system temp dir'
 * update test case (oh look, a duplicate regex)

 * move working directory related to BaseDumper

 * add quotes

 * place holder for these methods

 * use PharData to create zip/tar when necessary

 * add placeholder calls
 * add call to package() using PharData

 * finish downloadHg(), downloadSvn()

 * put to use

 * make BaseDumper abstract (to force extension)
 * make BaseDumper implement Interface (makes for less code in the implementation)

new functionality for dumping as .tar.gz

tar instead of tar.gz, new abstract dumpertest class

creates a local git repo instead of fetching a remote one

more oo-ish version of it

no constructor

 * refactor tests to be less linux-specific (used Composer\Util to wrap calls)

 * make filename only the version

 * various cs fixes (idention, tabs/spaces, doc blocks, etc.)
 * fixed a typo'd exception name

 * refactored downloading:
   * removed download*() methods
   * added dep on Composer\Factory to setup a DownloadManager instance

 * update CS with feedback from @stof

 * ArrayDumper doesn't extend BaseDumper anymore (hence no conflict on the interface)
 * move keys from BaseDumper back to ArrayDumper
 * interface now declares dump() to always return void

Apparently I had to update the lock.

CS fixes (tabs for spaces)
Bugfix: sprintf() was missing.

Fix docblock for @stof. ;)

Pull in lock from master.

Update lock one more time (hope it still merges).

whitespace

Revert ArrayDumper static keys
2acb033
@MattKetmo MattKetmo Moved archive Dumpers into its own Archiver package 3d0ce85
@MattKetmo MattKetmo Changed Package class due to upstream changes 2fd17ec
@MattKetmo MattKetmo Checks process execution 3b22791
@MattKetmo MattKetmo Update interface to merge vcs with basic archivers bfd2275
@MattKetmo MattKetmo Checks support before downloading the package b21bb1d
@MattKetmo MattKetmo Merged zip & tar archivers a733d76
@MattKetmo MattKetmo Fixed several typos
- break at first archiver supports
- use standard directory separator
- change exception message
- remove the BaseArchiver since tar & zip archivers have been merged
- plus coding style
d1d77dd
@MattKetmo MattKetmo Create ArchiveManager with the Factory 60b1cc7
@MattKetmo MattKetmo Fix how download manager is constructed
This fixes tests due to upstream changes.
The createDownloadManager in the Factory now takes the config as extra
parameter.
c248115
@MattKetmo MattKetmo Fix workflow & typos 9d24e17
@MattKetmo MattKetmo Cleaned archiver tests a2b404e
@naderman naderman Reorder ArchiveManager parameters to make the download manager optional ba51027
@naderman naderman Git can handle commit references in git archive just fine so use them bcbc50c
@naderman naderman Use a saner file name for package archives 33828b3
@naderman naderman Implement a basic archive command
It allows creating archives with the archive manager given a package/version
pair.
526f48e
@naderman naderman Remove Mercurial and Git Archivers as they cannot implement exclude r…
…ules
3e26502
@naderman naderman Define an option to exclude files in the archive command afcdad4
@naderman naderman Allow archiving the current project with composer archive ba375b6
@naderman naderman Skip the vendor dir when archiving the current project 735b59c
@naderman naderman Clarify composer archive argument optionality 5113546
@naderman naderman Add a missing array typehint 2856033
@naderman naderman Fix class description of archive command 838edd6
@naderman naderman Update json schema as archive include was removed 0b23643
@naderman naderman Generate a properly unique archive filename for dev revisions 48dd55b
@naderman naderman The ArchiveManager should return the written path for library usage 935f727
@naderman naderman Remove unnecessary dist type information from archive files 074af5d
@naderman naderman Remove unecessary unlink before writing archive 6ee08a2
@naderman naderman Make overwriting files an ArchiveManager option, use sourceRef in names 64941b0
@naderman naderman Respect gitignore, gitattributes and hgignore files in archiving deae503
@naderman naderman Skip directories in zip generation, empty dirs won't get archived
This seems ok as we currently rely on git generating archives which does
not archive empty directories either.
6066359
@naderman naderman Replace DIRECTORY_SEPARATOR in paths, not PATH_SEPARATOR 75d1759
@naderman naderman Use a FilterIterator rather than a modified IteratorIterator, simpler ecf4f42
@naderman naderman Rename ExcludeFilterBase to BaseExcludeFilter 1af2be9
@naderman naderman Follow PSR-2 for method modifier ordering 43be72a
@naderman naderman Output packages in archive command using getPrettyString 14ee67b
@naderman naderman Use null as default values rather than false
Also made archive() in the ArchiveCommand protected as it does not need
to be used from the outside. The ArchiveManager can be used instead.
870a87f
@naderman naderman Do not hardcode vendor dir exclusion on archive.
For one thing this wouldn't have worked for any custom installers anyway
which can write installed code to other places. This will now allow one
to use composer archive on a clean code checkout to build an archive as
we are used to. Or on one that had composer install run to build an
archive that can be used for deployment which includes the vendors.
cfd7a50
@naderman naderman Callbacks must be accessible publically on PHP 5.3 2204412
@naderman naderman Make sure git is setup to allow commits on travis 6f61d95
@naderman
Owner

Rebased to resolve .travis.yml conflict.

@naderman naderman merged commit 78c250d into composer:master

1 check passed

Details default The Travis build passed
@igorw

Docs for the archive command would be nice.

@chrisyue

why PharArchiver? and why not use di?
ZipArchive can have much better compression ratio than Phar, it can save a lot bandwidth. Make it configurable?

@chrisyue These classes are using DI. The Factory is the class instantiating them.

and anyway, commenting on a single commit which was in the middle of a 50-commits PR 7 months ago is not the good place to start a discussion. It would get lost very easily.

@stof Aha, sure, I've opened a issue: #2219, see you there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 28, 2013
  1. @MattKetmo @naderman

    Refactored the archiver package

    MattKetmo authored naderman committed
  2. @till @naderman

    Initial feature-dist

    till authored naderman committed
     * extends BaseDumper, implements interface
     * put $keys into BaseDumper
    
     * WIP WIP WIP WIP
     * BaseDumper for utilities
     * interface to enforce 'dump()'
     * feature:
       * supports git
       * supports zip output
       * basic test to cover feature
    
     * add @todo for later
     * add vendor namespace to package name
    
     * add extension to getFilename() so we don't need to switch in there (HT, @naderman)
    
     * add extension (obviously 'zip' in ZipDumper)
    
     * create archive in destination dir (provided by __construct())
    
     * condensed ZipDumper
     * moved code to BaseDumper (hopefully easier re-use)
    
     * use ProcessExecutor from BaseDumper
    
     * fix assignments in __construct()
     * allow injection of ProcessExecutor
    
     * fix parameters
    
     * fix regex
    
     * write in 'system temp dir'
     * update test case (oh look, a duplicate regex)
    
     * move working directory related to BaseDumper
    
     * add quotes
    
     * place holder for these methods
    
     * use PharData to create zip/tar when necessary
    
     * add placeholder calls
     * add call to package() using PharData
    
     * finish downloadHg(), downloadSvn()
    
     * put to use
    
     * make BaseDumper abstract (to force extension)
     * make BaseDumper implement Interface (makes for less code in the implementation)
    
    new functionality for dumping as .tar.gz
    
    tar instead of tar.gz, new abstract dumpertest class
    
    creates a local git repo instead of fetching a remote one
    
    more oo-ish version of it
    
    no constructor
    
     * refactor tests to be less linux-specific (used Composer\Util to wrap calls)
    
     * make filename only the version
    
     * various cs fixes (idention, tabs/spaces, doc blocks, etc.)
     * fixed a typo'd exception name
    
     * refactored downloading:
       * removed download*() methods
       * added dep on Composer\Factory to setup a DownloadManager instance
    
     * update CS with feedback from @stof
    
     * ArrayDumper doesn't extend BaseDumper anymore (hence no conflict on the interface)
     * move keys from BaseDumper back to ArrayDumper
     * interface now declares dump() to always return void
    
    Apparently I had to update the lock.
    
    CS fixes (tabs for spaces)
    Bugfix: sprintf() was missing.
    
    Fix docblock for @stof. ;)
    
    Pull in lock from master.
    
    Update lock one more time (hope it still merges).
    
    whitespace
    
    Revert ArrayDumper static keys
  3. @MattKetmo @naderman

    Moved archive Dumpers into its own Archiver package

    MattKetmo authored naderman committed
  4. @MattKetmo @naderman

    Changed Package class due to upstream changes

    MattKetmo authored naderman committed
  5. @MattKetmo @naderman

    Checks process execution

    MattKetmo authored naderman committed
  6. @MattKetmo @naderman

    Update interface to merge vcs with basic archivers

    MattKetmo authored naderman committed
  7. @MattKetmo @naderman

    Checks support before downloading the package

    MattKetmo authored naderman committed
  8. @MattKetmo @naderman

    Merged zip & tar archivers

    MattKetmo authored naderman committed
  9. @MattKetmo @naderman

    Fixed several typos

    MattKetmo authored naderman committed
    - break at first archiver supports
    - use standard directory separator
    - change exception message
    - remove the BaseArchiver since tar & zip archivers have been merged
    - plus coding style
  10. @MattKetmo @naderman

    Create ArchiveManager with the Factory

    MattKetmo authored naderman committed
  11. @MattKetmo @naderman

    Fix how download manager is constructed

    MattKetmo authored naderman committed
    This fixes tests due to upstream changes.
    The createDownloadManager in the Factory now takes the config as extra
    parameter.
  12. @MattKetmo @naderman

    Fix workflow & typos

    MattKetmo authored naderman committed
  13. @MattKetmo @naderman

    Cleaned archiver tests

    MattKetmo authored naderman committed
  14. @naderman
  15. @naderman
  16. @naderman
  17. @naderman

    Implement a basic archive command

    naderman authored
    It allows creating archives with the archive manager given a package/version
    pair.
  18. @naderman
  19. @naderman
  20. @naderman
  21. @naderman
  22. @naderman
  23. @naderman

    Add a missing array typehint

    naderman authored
  24. @naderman
  25. @naderman
  26. @naderman
  27. @naderman
  28. @naderman
  29. @naderman
  30. @naderman
  31. @naderman
  32. @naderman

    Skip directories in zip generation, empty dirs won't get archived

    naderman authored
    This seems ok as we currently rely on git generating archives which does
    not archive empty directories either.
  33. @naderman
  34. @naderman
  35. @naderman
  36. @naderman
  37. @naderman
  38. @naderman

    Use null as default values rather than false

    naderman authored
    Also made archive() in the ArchiveCommand protected as it does not need
    to be used from the outside. The ArchiveManager can be used instead.
  39. @naderman

    Do not hardcode vendor dir exclusion on archive.

    naderman authored
    For one thing this wouldn't have worked for any custom installers anyway
    which can write installed code to other places. This will now allow one
    to use composer archive on a clean code checkout to build an archive as
    we are used to. Or on one that had composer install run to build an
    archive that can be used for deployment which includes the vendors.
  40. @naderman
  41. @naderman
  42. @naderman
  43. @naderman
Something went wrong with that request. Please try again.