DMSDK Export Listener leaves too many files open during read/export

[chiamarc]

This happened in the past with the Java bulk API on Linux.

During dmsdk test runs where we read 100,000 documents from the database, larger batch sizes (100 and above) result in hundreds of thousands of temp files being opened and not closed until the test is shutdown. This results in a shortage of open file descriptors. In one case (batches of size 500), more than 1.2M descriptors were open simultaneously. Telltale signs are a plethora of messages like this:

INFO: File /tmp/MIME7250643930506885814.tmp was not deleted
Sep 27, 2016 2:44:38 PM org.jvnet.mimepull.WeakDataFile close

This is confirmed by counting the number of open files with lsof and grepping for "MIME".

Libraries:

client-api-java-4.0-20160916
data-movement-1.0-SNAPSHOT.jar (20160922)

[sammefford]

What is your hard and soft ulimit set to? You can find out with ulimit -Hn and ulimit -Sn.

How many processes do you have open max at once? ulimit is applied per-process

Can you say how many "was not deleted" messages you have in your output?

[chiamarc]

Originally, hard ulimit set to 8192, soft ulimit set to 4096.
I increased both to 100,000 but problem was still occurring:
Caused by: java.nio.file.FileSystemException: /tmp/MIME7848769172375907673.tmp: Too many open files
Ulimit is applied per-process as well as system-wide in /etc/security/limits.conf.
When I increased hard and soft to 1,000,000 in limits.conf, the problem disappeared, but I was observing 700K+ open /tmp/MIME* descriptors during any given run via lsof.

For a typical run (reading 100,000 documents from the database in batches of size 500), I see ~78,850 "was not deleted" messages and a peak of ~726K open file descriptors. It seems like for every temp file there is around 10x the number of file descriptors being opened.

[chiamarc]

I changed the title, because I was not actually reproducing this in the ML9 bulk api.

[sammefford]

Thanks Marc. Could you please attach the code, data, and instructions (mvce)? I'll need to duplicate this scenario.

[jmakeig]

@chiamarc Is perf harness part checked into this repo?