Silent exit when too many files

[mlarcher]

When there is more than 200 files in the dependency tree, the 'done' event of parcelMap is never emitted. This is a big issue for us, as we are relying on it to build the project.

I tracked down the issue to the following line in index.js:

browserifyInstance.pipeline.get( 'label' ).unshift( through.obj( function( row, enc, next ) {

I can't go any much further though, as I lack experience with that kind of stream manipulations.

There is no error and the task exit code is 0, but the callback is never called in this scenario.
Could you please look into it and get back to me on the matter ?

[dgbeck]

Hi @mlarcher , we need to be able to reproduce the issue you are describing to help. Is it possible for you to provide the project files, or some mock project that reproduces this issue?

[mlarcher]

I can't provide the project files, but I'll do my best to provide a test case.
Anyhow from what I gather it is really a matter of the number of files in the dependency graph so it shouldn't be hard to reproduce on your side too.
I hope you can find out something in the source code in the meantime. For instance I was under the impression that there might be something to figure out with the unshift method and consuming the stream after looking at gulpjs/gulp#716 but again I'm not really one to say.

[dgbeck]

OK. But I don't think the number of files is at the root of the issue. I just checked a project that we are using parcel map for and it has 817 total files in the dependency graph but parcel map behaves as expected.

[ghost]

It might be a ulimit issue with the error getting eaten somewhere. Check ulimits on your system with ulimit -a. Anything under 5000 for open files (-n) is far too low.

[mlarcher]

I have the following in git bash (default setting):

open files                      (-n) 256

But I just increased it to 5000 and it didn't change the end result.

[mlarcher]

I just tried on a mac: ulimit for open files is 256 too, and the problem is also present but it happens at 204 files instead of 201.
FYI the reason I keep talking about number of file is that I first tried emptying the files to reduce the amount of in memory data and it didn't change anything, and then I changed the number of require() calls and always hit the wall when requiring the 201th file, regardless of if I was using empty files or not.

[mlarcher]

I've set up a testcase, you can get it on http://ringabell.org/temp/testcase.zip
You can try it out by first making a npm install (and npm install -g gulp if you don't have it), then using gulp check to test. You'll see a "CALLBACK WAS CALLED" when it is, nothing in the default state.

There is nothing more to the test case than a bunch of require calls and parcelMap following the browserify dependency graph. You'll see that commenting out the second block of require calls makes the callback appear when testing, while it is missing when those calls are made. I've seend the problem occur on two different windows machine and one OSX computer, so I think it's safe to assume it isn't due to the environment.

Hopefully this will help you figure out what's going wrong. Please let me know if there's anything more I can do to help fix the issue.

[dgbeck]

Hi @mlarcher ,

This issue here is that the browserify write stream is not being piped anywhere, and no callback is being supplied to bundle(), so the bundle command is not terminating. To fix this issue, you can change, for example,

  bundler.bundle();

To

  bundler.bundle( function( result ) {
    console.log( 'browserify done' );
  } );

In styles.js, and then you will get the output

➜  testcase  gulp check
[02:41:36] Using gulpfile ~/Downloads/testcase/gulpfile.js
[02:41:36] Starting 'modules-styles'...
CALLBACK CALLED
[02:41:37] Finished 'modules-styles' after 610 ms
[02:41:37] Starting 'check'...
[02:41:37] Finished 'check' after 7.55 μs
browserify done
➜  testcase

Closing for now, but let me know if that does not solve the problem on your end.

[mlarcher]

Looks like it fixes the issue :)
That point may need to appear somewhere in the documentation when you find time to update it, as we are here interested in the graph parcelMap produces and don't care about the output of the bundle() method...
Anyhow, thanks for your insight.
Just out of curiosity, do you have any idea why not piping the result is a deal breaker after a certain amount of files, and not before that limit is reached ?

[dgbeck]

Just out of curiosity, do you have any idea why not piping the result is a deal breaker after a certain amount of files, and not before that limit is reached ?

My guess is that some internal stream hit its high water mark, which stops the data flow.

https://nodejs.org/api/stream.html#stream_new_stream_readable_options

Just a guess, though.