wcpy dnn has a de facto limit to number of input files

[calcuttj]

Each input file is opened & indexed here. If there's a dataset matching the requested key, the file remains open indefinitely. This means that a de facto limit of ulimit -Sn files is imposed here.

[calcuttj]

If we close after indexing the insides of the files here and persist the fname

and then reopen/close here

and also remove the fp.filename ref here

this would work, though I'm not sure what performance degradation might arise.

Alternatively, though more complicatedly, in an attempt to reduce the number of open/close calls we could implement an internal counting mechanism per file. i.e. if we've accessed all relevant keys within a file, then close it and reopen on next request.

There still remains the potential scenario of shuffling the dataset and getting one entry from ~1024 files (or something less extreme/unlikely), thus hitting the limit again. We could then fallback to checking resource.getrlimit()

[calcuttj]

Testing with 300 files with 10 events each -- open/close vs leave open

Open/close

run: 0 epoch: 0 loss: 1.3480e-02 [b=6,n=2392] eval: 1.5978e-02 [b=1,n=598] dt=4.185e+02 s

Leave open:

run: 0 epoch: 0 loss: 8.5323e-03 [b=6,n=2392] eval: 1.4091e-02 [b=1,n=598] dt=4.174e+02 s

No meaningful difference between the 2 tests, so maybe we opt for open/close? @brettviren thoughs?

[brettviren]

Can one simply increase the user ulimit?

Otherwise, I think the direction you go seems good.

Perhaps the Dataset class can be changed to hold both in-file path and file system path and then do an on-demand h5py.File() object construction, get the data, close the file.

[calcuttj]

Can one simply increase the user ulimit?

Yes 😶