file.contents as a 'stream' instead of a 'buffer'

[zepedrorocha]

Hello.

When using "buffer: false" gulp option I will get a stream instead of a buffer in 'file.contents' but the copied file will have 0KB

example:

gulp.src(PATH/TO/FILE, {
      buffer: false
    })
    .pipe(gulp.dest(OUTPUT/PATH)) //500 kb size file

gulp.src(PATH/TO/FILE, {
      buffer: false
    })
    .pipe(hash())
    .pipe(gulp.dest(OUTPUT/PATH)) //0 kb size file

[joaocc]

Hi. Any news on this one? Thx

[Dragory]

Hey! I will look into this ASAP. I will be out of town for the weekend but I'll try to get this fixed next week.

[Dragory]

Pushed out a potential fix (cd6264d). Can you give it a try, @zepedrorocha or @joaocc?
It should be out on NPM soon as well, version 2.2.1.

[zepedrorocha]

Hey sorry for the slow reply.

Now it's not even completing the gulp task. Something breaks the pipeline but no error is given.

return gulp.src('./pack.unitypackage', { //100 MB
      buffer: false
    })
    .pipe(gulp.dest('./temp')); //100 MB file

return gulp.src('./pack.unitypackage', { //100 MB
      buffer: false
    })
   .pipe(hash())
   .pipe(gulp.dest('./temp')); // no file. The task that has this piece of code doesn't even finish.

Using gulp 3.9.0 and gulp-hash 2.2.1

[Dragory]

Hey,

I pushed out a new fix (version 2.2.2, commit 4694365). The old version didn't wait for the file contents to finish streaming to a temporary buffer (which is restreamed later) and pushed the file onward before being done with it (small files were fine).

Can you check if this one works?

[zepedrorocha]

Hey 2 problems. gulp task still not finish like it should. Having multiple src files will only exec the first pipeline file (I tested this with files <1GB).
And the other problem is that you are creating a new Buffer() which is basically invalidating the "buffer: false". Throws an error when a file is >1GB

"buffer.js:71
throw new RangeError('Attempt to allocate Buffer larger than maximum '
RangeError: Attempt to allocate Buffer larger than maximum size: 0x3fffffff bytes."

[Dragory]

Hmm, the crypto module definitely supports .update() to hash streamed contents without loading it all into a buffer, but if I change to that method, I need to figure out how to deal with custom hashing functions (or maybe I'll just phase those out altogether, not a very useful feature really). I'll look into this.

The problem with multiple src files I wasn't able to reproduce; I tried by having two ~120MB files in gulp.src and the command finished properly and the hashed files were output. Are you able to give more specific steps to reproducing this issue?

[joaocc]

Hi.
We are processing around 280 files, with total size of 13GB. Basically we calculate the hash of all files. luckily only one of those is larger than 1GB.
An alternative to sending you the file repo (which might be difficult) would be to use some kind of instrumented version of the plugin, for analysis after the crash.
Would you be able to do that?
Thx

[Dragory]

I rewrote most of the plugin and it should use streams properly now. Can you give it a try? The update is on its own branch at the moment and not on npm yet, so you'll need to do npm install git://github.com:Dragory/gulp-hash.git#3.0.0

Edit: 3.0.0 is now live. Doing npm install --save-dev gulp-hash@3.0.0 should do the trick.

[Dragory]

I'm closing this issue due to inactivity. Tests related to streams are currently passing. If you still encounter problems using streams, please comment here or open a new issue!