Plugins using through2.obj are limited to 16 files by default

[fabiendem]

Hello,

I am facing an issue with the plugin gulp-ftp, posted it in the issues of the repo and got redirected here. (See: https://github.com/sindresorhus/gulp-ftp/issues/25).

Here is a copy paste of the problem description:

I am trying to upload 25 files but only 16 files are uploaded via the plugin.
If I use the plugin gulp-using, I can see that the 25 files are passed to the ftp plugin :

gulp.src('dist/**/*.*')
        .pipe($.using())
        .pipe($.ftp(...})

But If in the index.js of the plugin I log the files going through through2:

...
return through.obj({highWaterMark: 32}, function (file, enc, cb) {
        ====> gutil.log(file.path);

        if (file.isNull()) {
            cb(null, file);
            return;
        }
...

I get only the first 16 files.

If I increase the highWaterMark option, all the files are uploaded:

....
return through.obj({highWaterMark: 32}, function (file, enc, cb) {
....

So basically the highWaterMark option is currently not dynamic, that's what I understand...
I am not sure it's the role of gulp to deal with this but I transfer the issue as advised by @sindresorhus

Thanks

[yocontra]

You need something consuming the stream. Make sure you are returning the stream for the task or consuming it yourself otherwise it will pause when the watermark is hit

[fabiendem]

Thanks !

If I consume it with a simple .size() from gulp-size it's working.

gulp.src('dist/**/*.*')
        .pipe($.ftp({
            host: ftpCredentials.host,
            user: ftpCredentials.user,
            pass: ftpCredentials.pass,
            remotePath : remotePath
        }))
        .pipe($.size());

[yocontra]

You should open an issue on gulp-ftp to fix this

[wbswjc]

I've spent a whole night trying to figure this out, many thanks @fabiendem

I made a sewer for gulp pipe like this..

.pipe(gulp.dest('./temp/sewer')

[01:48:51] CLI version 1.4.0
[01:48:51] Local version 4.0.0-alpha.2