Crash if a subfolder of the current folder has more than 50'000 files

[Carreau]

@Gansgar opened ipython/ipython#9165

iPython "crashes" or better gets stuck forever if at least one subdirectory of my current directory has more than 50'000 files.

I have this file structure:

notMNIST_large
A - 50'000 files
B - 50'000 files
...
J - 50'000 files

If I enter (or start in) the directory "notMNIST_large" iPython gets stuck and the web fronted shows a server failure. A new access to the web fronted gets stuck. All webbrowsers so far are trying to load it, but their wait till I need to kill ipython with a sigterm 3. The response to ^C is still there, but it never shuts down. (Probably yes after a long time that I'm not willing to wait)

[Carreau]

@takluyver commented

Are there any errors in the terminal where you started the notebook?

[Carreau]

@Gansgar commented

No, that would be a really bug. It's just that thr index script (I think) gets stuck during trying of indexing 550'000 mostly png files (I estimate that that would happen with every type of file). I would include a methode before indexing, that goves back an error of to much files if a folder has more than x files. Or something like this.

[Carreau]

@takluyver commented

I'm a bit puzzled that it falls over when a subdirectory has lots of files - I thought we only looked at the contents of the current directory. And I'm not sure what the indexing script you're talking about is. Maybe something got added to the project while I was away from it for a few months.

By the way, the notebook is now in a separate repo - https://github.com/jupyter/notebook . No need to reopen this issue, but for future issues that's the right place.

[Carreau]

@takluyver commented

So I can reproduce problems (and I would kind of expect them) if I open the directory containing 50k files in the notebook interface. But you say you see problems even opening the parent directory? Or have I misunderstood?

[Carreau]

@Gansgar commented

Yes. It gets stuck by opening the parent directory

[Carreau]

@takluyver commented

Maybe a browser is prefetching the link to the subdirectory. What browser are you using? Can you try running the notebook with the --debug flag, and see if there are any messages like:

200 GET /api/contents/A?type=directory...

[Carreau]

@takluyver commented

Or like

200 GET /tree/A ...

[Carreau]

@Gansgar commented

Used Safari, Chrome, Iceweasel. I can look more precise into the debug lines later.

[Carreau]

@takluyver commented

Shared folders in VMs are certainly slower, but I'm still mystified as to why it's happening when you're only browsing the parent directory of the ones with many files. Is it actually specific to when the subdirs have many files, or does it affect anything shared directory? If you start ipython in there and do os.listdir('.'), is it quick?

@minrk @Carreau am I missing something about how the notebook server scans files?

[Carreau]

@Gansgar commented

Works smooth in every other directory. Even reading all this data into a 1.7gb big array in a iPython Notebook works without problem (is just slow). The thing that is actually more interesting is: It only crashes in Iceweasle. Clearly. Every other access after that is not working anymore on any browser. But I can access all those directories without a problem, except a longer response time for once after that on Chrome or Safari before the crash with Iceweasle.

[Carreau]

@Gansgar commented

It looks like that a bug in the browser results into a crash of iPython

[Carreau]

@minrk commented

I looked through, and I think we list directories in is_hidden as a check for readability. I'm not sure that makes sense to do, though.

[Carreau]

@minrk commented

#998 switches to execute access, which should be equivalent and much less expensive.