Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

use archiver for tar. use streams for gzip. #13

Closed
wants to merge 1 commit into from

1 participant

@ctalkington
Collaborator

this removes the need to manually copy files to a new dir and to use a special fstream. tar file sizes changed a bit since archiver uses the standard 512 bytes per block vs 256 bytes that node-tar was using. this may become configureable through archiver in the future.

@ctalkington ctalkington use archiver for tar. use streams for gzip.
this removes the need to manually copy files to a new dir and to use a
special fstream. tar file sizes changed a bit since archiver uses the
standard 512 bytes per block vs 256 bytes that node-tar was using. this
may become configureable through archiver in the future.
2afb780
@ctalkington
Collaborator

closing this. will get rc5 compat then deal with this. also archiver v0.3 will bring things that reduce this even more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jan 7, 2013
  1. @ctalkington

    use archiver for tar. use streams for gzip.

    ctalkington authored
    this removes the need to manually copy files to a new dir and to use a
    special fstream. tar file sizes changed a bit since archiver uses the
    standard 512 bytes per block vs 256 bytes that node-tar was using. this
    may become configureable through archiver in the future.
This page is out of date. Refresh to see the latest.
View
72 CONTRIBUTING.md
@@ -1,71 +1 @@
-# Contributing to grunt
-
-There are a number of grunt projects.
-
-* [grunt](https://github.com/gruntjs/grunt) - the main grunt project
-* [gruntjs.com](https://github.com/gruntjs/gruntjs.com) - the gruntjs.com website
-* [grunt-contrib collection](https://github.com/gruntjs/grunt-contrib) - a collection of all grunt "contrib" plugins
-
-In addition, each individual grunt-contrib plugin is a separate repository listed on the [gruntjs org homepage](https://github.com/gruntjs).
-
-## Filing issues
-If something isn't working like you think it should, please read the documentation first. If you'd like to chat with someone, [pop into IRC](#discussing-grunt) and ask your question there.
-
-The best way to ensure an issue gets addressed is to file it in the appropriate issues tracker.
-
-**If there's an issue with a specific grunt-contrib plugin:**
-Please file an issue on that plugin's issues tracker.
-
-**If you'd like to contribute a new plugin:**
-Please file an issue on the [grunt-contrib collection issues tracker](https://github.com/gruntjs/grunt-contrib/issues). We don't accept all plugins, but we'll certainly consider yours.
-
-**If there's an issue with the [website](http://gruntjs.com/):**
-Please file an issue on the [gruntjs.com website issues tracker](https://github.com/gruntjs/gruntjs.com/issues).
-
-**If there's an issue that isn't specific to any of the above:**
-Please file an issue on the [grunt issues tracker](https://github.com/gruntjs/grunt/issues).
-
-### Simplify the issue
-Try to [reduce your code](http://www.webkit.org/quality/reduction.html) to the bare minimum required to reproduce the issue. This makes it much easier (and much faster) to isolate and fix the issue.
-
-### Explain the issue
-If we can't reproduce the issue, we can't fix it. Please list the exact steps required to reproduce the issue. Include versions of your OS, Node.js, grunt, etc. Include relevant logs or sample code.
-
-## Discussing grunt
-Join the [freenode](http://freenode.net/) IRC #grunt channel. We've got a bot and everything.
-
-_No private messages, please._
-
-## Modifying grunt
-First, ensure that you have the latest [Node.js](http://nodejs.org/) and [npm](http://npmjs.org/) installed.
-
-1. Fork and clone the repo.
-1. Check out the correct branch. Currently, grunt development happens in the `devel` branch.
-1. Run `npm install` to install all grunt dependencies.
-1. Run `npm link` to put the dev version of grunt in the system path (this is only needed for developing grunt, not for plugins or the website).
-1. Run `grunt` to grunt grunt.
-
-Assuming that you don't see any red, you're ready to go. Just be sure to run `grunt` after making any changes, to ensure that nothing breaks.
-
-### Submitting pull requests
-
-1. Create a new branch, please don't work in your `master` or `devel` branch directly.
-1. Add failing tests for the change you want to make. Run `grunt` to see the tests fail.
-1. Fix stuff.
-1. Run `grunt` to see if the tests pass. Repeat steps 2-4 until done.
-1. Update the documentation to reflect any changes.
-1. Push to your fork and submit a pull request.
-
-### Syntax
-
-* Two space indents. Don't use tabs anywhere. Use `\t` if you need a tab character in a string.
-* No trailing whitespace, except in markdown files where a linebreak must be forced.
-* Don't go overboard with the whitespace.
-* No more than [one assignment](http://benalman.com/news/2012/05/multiple-var-statements-javascript/) per `var` statement.
-* Delimit strings with single-quotes `'`, not double-quotes `"`.
-* Prefer `if` and `else` to ["clever"](http://programmers.stackexchange.com/a/25281) uses of `? :` conditional or `||`, `&&` logical operators.
-* Comments are great. Just put them _before_ the line of code, _not_ at the _end_ of the line.
-* **When in doubt, follow the conventions you see used in the source already.**
-
-### Reverting back to the "official" grunt
-If you've used `npm link` to put a dev version of grunt in the system path and, for some reason, need to revert back to the current official grunt release, just reinstall grunt globally with `npm install -g grunt-cli`
+Please see the [Contributing to grunt](http://gruntjs.com/contributing) guide for information on contributing to this project.
View
4 Gruntfile.js
@@ -55,8 +55,8 @@ module.exports = function(grunt) {
'tmp/compress_test_v<%= test_vars.version %>.tgz': ['**'],
'tmp/compress_test_files_template.tgz': ['<%= files.compress_test %>/**'],
- 'tmp/compress_test_file.gz': ['test.js'],
- 'tmp/compress_test_file2.gz': ['folder_one/one.js']
+ 'tmp/compress_test_file.js.gz': ['test.js'],
+ 'tmp/compress_test_file2.js.gz': ['folder_one/one.js']
}
},
View
4 package.json
@@ -28,9 +28,7 @@
"test": "grunt test"
},
"dependencies": {
- "tar": "~0.1.13",
- "fstream": "~0.1.18",
- "archiver": "~0.1.0",
+ "archiver": "~0.2.2",
"rimraf": "~2.0.2",
"grunt-lib-contrib": "~0.3.0"
},
View
220 tasks/compress.js
@@ -12,6 +12,8 @@ module.exports = function(grunt) {
var fs = require('fs');
var path = require('path');
+ var zlib = require('zlib');
+ var archiver = require('archiver');
var rimraf = require('rimraf');
var helpers = require('grunt-lib-contrib').init(grunt);
@@ -40,10 +42,13 @@ module.exports = function(grunt) {
options.rootDir = false;
}
+ grunt.verbose.writeln('------');
grunt.verbose.writeflags(options, 'Options');
+ grunt.verbose.writeln();
+ var srcCwd = options.cwd;
var srcFiles = grunt.file.expandFiles(options.minimatch, this.file.srcRaw);
- var destFile = path.normalize(this.file.dest);
+ var destFile = unixifyPath(path.normalize(this.file.dest));
var destDir = path.dirname(destFile);
if (srcFiles.length === 0) {
@@ -68,186 +73,95 @@ module.exports = function(grunt) {
grunt.file.mkdir(destDir);
}
- methods[mode](srcFiles, destFile, options, function(err, written) {
- grunt.log.writeln('File ' + destFile + ' created (' + written + ' bytes written).');
- done();
- });
- });
-
- var getSize = function(filename) {
- try {
- return fs.statSync(filename).size;
- } catch (e) {
- return 0;
- }
- };
-
- var autoDetectMode = function(dest) {
- if (grunt.util._.endsWith(dest, '.tar.gz')) {
- return 'tgz';
- }
-
- var ext = path.extname(dest).replace('.', '');
-
- if (ext === 'gz') {
- return 'gzip';
- } else {
- return ext;
- }
- };
-
- var tarCopy = function(srcFiles, tempDir, options) {
- var newFiles = [];
- var newMeta = {};
-
+ var archive;
+ var destStream = fs.createWriteStream(destFile);
var srcFile;
- var destFile;
-
- var fileName;
- var filePath;
- srcFiles.forEach(function(file) {
- fileName = path.basename(file);
- filePath = path.dirname(file);
+ var gzipper = zlib.createGzip();
- srcFile = path.join(options.cwd, file);
+ if (mode === 'gzip') {
+ srcFile = path.join(srcCwd, srcFiles);
+ fs.createReadStream(srcFile).pipe(gzipper).pipe(destStream);
- if (options.flatten) {
- destFile = path.join(tempDir, fileName);
- } else {
- destFile = path.join(tempDir, filePath, fileName);
+ destStream.on('close', function() {
+ grunt.log.writeln('File ' + destFile + ' created (' + getSize(destFile) + ' bytes written).');
+ done();
+ });
+ } else {
+ if (mode === 'zip') {
+ archive = archiver.createZip(options);
+ } else if (mode === 'tar' || mode === 'tgz') {
+ archive = archiver.createTar(options);
}
- newFiles.push(destFile);
- newMeta[destFile] = {name: file};
-
- grunt.verbose.writeln('Adding ' + srcFile + ' to temporary structure.');
- grunt.file.copy(srcFile, destFile);
- });
-
- return [newFiles, newMeta];
- };
-
- var methods = {
- zip: function(srcFiles, dest, options, callback) {
- var zip = require('archiver').createZip(options);
-
- var destDir = path.dirname(dest);
-
- zip.pipe(fs.createWriteStream(dest));
+ if (mode === 'zip' || mode === 'tar') {
+ archive.pipe(destStream);
+ } else if (mode === 'tgz') {
+ archive.pipe(gzipper).pipe(destStream);
+ }
- zip.on('error', function(e) {
- grunt.log.error(e);
- grunt.fail.warn('zipHelper failed.');
+ archive.on('error', function(err) {
+ grunt.log.error(err);
+ grunt.fail.warn('archiver failed.');
});
- var destFile;
- var srcFile;
+ var internalFileName;
- var fileName;
- var filePath;
- var fileMeta;
-
- grunt.util.async.forEachSeries(srcFiles, function(file, next) {
- fileName = path.basename(file);
- filePath = path.dirname(file);
-
- srcFile = path.join(options.cwd, file);
+ grunt.util.async.forEachSeries(srcFiles, function(srcFile, next) {
+ internalFileName = srcFile;
+ srcFile = unixifyPath(path.join(srcCwd, srcFile));
if (options.flatten) {
- destFile = fileName;
- } else {
- destFile = path.join(filePath, fileName);
+ internalFileName = path.basename(internalFileName);
}
if (options.rootDir && options.rootDir.length >= 1) {
- destFile = path.join(options.rootDir, destFile);
+ internalFileName = path.join(options.rootDir, internalFileName);
}
- fileMeta = {
- name: destFile
- };
+ internalFileName = unixifyPath(internalFileName);
- zip.addFile(fs.createReadStream(srcFile), fileMeta, function() {
+ archive.addFile(fs.createReadStream(srcFile), { name: internalFileName }, function() {
+ grunt.verbose.writeln('Add ' + srcFile + ' -> ' + destFile + '/' + internalFileName);
next();
});
}, function(err) {
- zip.finalize(function(written) {
- callback(null, written);
+ archive.finalize(function(written) {
+ grunt.verbose.writeln();
+ grunt.log.writeln('File ' + destFile + ' created (' + written + ' bytes written).');
+ done();
});
});
- },
-
- tar: function(srcFiles, dest, options, callback, gzip) {
- var fstream = require('fstream');
- var tar = require('tar');
- var zlib = require('zlib');
-
- var destDir = path.dirname(dest);
- var destFile = path.basename(dest);
- var destFileExt = path.extname(destFile);
- var destFileName = path.basename(dest, destFileExt);
-
- var tempDir = path.join(destDir, 'tar_' + (new Date()).getTime());
- var tarDir;
-
- if (options.rootDir && options.rootDir.length >= 1) {
- tarDir = options.rootDir;
- options.rootDir = false;
- } else {
- tarDir = destFileName;
- }
-
- if (gzip === true) {
- tarDir = tarDir.replace('.tar', '');
- }
-
- var tarProcess;
-
- tarDir = path.join(tempDir, tarDir);
-
- tarCopy(srcFiles, tarDir, options);
-
- var reader = fstream.Reader({path: tarDir, type: 'Directory'});
- var packer = tar.Pack();
- var gzipper = zlib.createGzip();
- var writer = fstream.Writer(dest);
-
- if (gzip === true) {
- tarProcess = reader.pipe(packer).pipe(gzipper).pipe(writer);
- } else {
- tarProcess = reader.pipe(packer).pipe(writer);
- }
-
- tarProcess.on('error', function(e) {
- grunt.log.error(e);
- grunt.fail.warn('tarHelper failed.');
- });
+ }
+ });
- tarProcess.on('close', function() {
- rimraf.sync(tempDir);
- callback(null, getSize(dest));
- });
- },
+ var autoDetectMode = function(dest) {
+ if (grunt.util._.endsWith(dest, '.tar.gz')) {
+ return 'tgz';
+ }
- tgz: function(srcFiles, dest, options, callback) {
- methods.tar(srcFiles, dest, options, callback, true);
- },
+ var ext = path.extname(dest).replace('.', '');
- gzip: function(file, dest, options, callback) {
- var zlib = require('zlib');
+ if (ext === 'gz') {
+ return 'gzip';
+ } else {
+ return ext;
+ }
+ };
- var srcFile = path.join(options.cwd, file);
+ var unixifyPath = function(filepath) {
+ if (process.platform === 'win32') {
+ return filepath.replace(/\\/g, '/');
+ } else {
+ return filepath;
+ }
+ };
- zlib.gzip(grunt.file.read(srcFile), function(e, result) {
- if (!e) {
- grunt.file.write(dest, result);
- callback(null, result.length);
- } else {
- grunt.log.error(e);
- grunt.fail.warn('gzipHelper failed.');
- }
- });
+ var getSize = function(filename) {
+ try {
+ return fs.statSync(filename).size;
+ } catch (e) {
+ return 0;
}
};
};
View
16 test/compress_test.js
@@ -44,11 +44,11 @@ exports.compress = {
test.equal(expected, actual, 'should add files and folders into tar (grunt template in source)');
// Tar (gzip)
- actual = getSize('tmp/compress_test_files.tgz') >= 200;
+ actual = getSize('tmp/compress_test_files.tgz') >= 195;
expected = true;
- test.equal(expected, actual, 'should compress files into tar');
+ test.equal(expected, actual, 'should compress files into tgz');
- actual = getSize('tmp/compress_test_v0.1.0.tgz') >= 350;
+ actual = getSize('tmp/compress_test_v0.1.0.tgz') >= 315;
expected = true;
test.equal(expected, actual, 'should compress folders and their files into tgz (with template support)');
@@ -57,12 +57,12 @@ exports.compress = {
test.equal(expected, actual, 'should compress files and folders into tgz (grunt template in source)');
// gzip
- actual = getSize('tmp/compress_test_file.gz');
- expected = getSize('test/expected/compress_test_file.gz');
+ actual = getSize('tmp/compress_test_file.js.gz');
+ expected = getSize('test/expected/compress_test_file.js.gz');
test.equal(expected, actual, 'should gzip file');
- actual = getSize('tmp/compress_test_file2.gz');
- expected = getSize('test/expected/compress_test_file2.gz');
+ actual = getSize('tmp/compress_test_file2.js.gz');
+ expected = getSize('test/expected/compress_test_file2.js.gz');
test.equal(expected, actual, 'should gzip another file (multiple dest:source pairs)');
test.done();
@@ -80,7 +80,7 @@ exports.compress = {
expected = getSize('test/expected/compress_test_flatten.tar');
test.equal(expected, actual, 'should create a flat internal structure');
- actual = getSize('tmp/compress_test_flatten.tgz') >= 320;
+ actual = getSize('tmp/compress_test_flatten.tgz') >= 297;
expected = true;
test.equal(expected, actual, 'should create a flat internal structure');
View
0  test/expected/compress_test_file.gz → test/expected/compress_test_file.js.gz
File renamed without changes
View
0  test/expected/compress_test_file2.gz → test/expected/compress_test_file2.js.gz
File renamed without changes
View
BIN  test/expected/compress_test_files.tar
Binary file not shown
View
BIN  test/expected/compress_test_files_template.tar
Binary file not shown
View
BIN  test/expected/compress_test_flatten.tar
Binary file not shown
View
BIN  test/expected/compress_test_rootdir.tar
Binary file not shown
View
BIN  test/expected/compress_test_v0.1.0.tar
Binary file not shown
Something went wrong with that request. Please try again.