Too many open files

[wolkenschieber]

Analysing large projects fails as too many files are held open at the same time.

Scenario to reproduce:

• Clone java-bullshifier
• Generate sample project with 100 subprojects and 256 classes each
  • /tmp/java-bullshifier$ ./gradlew -Pclasses=256 -Psubprojects=100 clean run
  • /tmp/java-bullshifier/output$ ./gradlew fatjar
• Use jpeek jar from Readme
• Start analysis: /tmp/jpeek$ java -jar jpeek-0.5-jar-with-dependencies.jar /tmp/java-bullshifier/output /tmp/results

Run will fail following exception:

Exception in thread "main" java.io.IOException: java.io.UncheckedIOException: java.io.FileNotFoundException: /tmp/java-bullshifier/output/ProjAhgodwwubpj/build/tmp/fatjar-stage/ch/qos/logback/core/LogbackException.class (Too many open files)
        at org.cactoos.func.IoCheckedFunc.apply(IoCheckedFunc.java:74)
        at org.cactoos.scalar.IoCheckedScalar.value(IoCheckedScalar.java:60)
        at org.jpeek.App.analyze(App.java:115)
        at org.jpeek.Main.main(Main.java:61)
Caused by: java.io.UncheckedIOException: java.io.FileNotFoundException: /tmp/java-bullshifier/output/ProjAhgodwwubpj/build/tmp/fatjar-stage/ch/qos/logback/core/LogbackException.class (Too many open files)
        at org.cactoos.func.UncheckedFunc.apply(UncheckedFunc.java:61)
        at org.cactoos.iterator.Mapped.next(Mapped.java:70)
        at org.cactoos.iterator.Filtered.hasNext(Filtered.java:89)
        at org.cactoos.iterator.Mapped.hasNext(Mapped.java:65)
        at org.jpeek.metrics.JavassistClasses.metrics(JavassistClasses.java:139)
        at org.jpeek.metrics.JavassistClasses.xembly(JavassistClasses.java:103)
        at org.jpeek.metrics.cohesion.CAMC.xembly(CAMC.java:91)
        at org.jpeek.Report.xml(Report.java:124)
        at org.jpeek.Report.save(Report.java:88)
        at org.jpeek.App.lambda$analyze$0(App.java:112)
        at org.cactoos.func.FuncOf.lambda$new$3(FuncOf.java:88)
        at org.cactoos.func.FuncOf.apply(FuncOf.java:104)
        at org.cactoos.scalar.And.lambda$null$0(And.java:101)
        at org.cactoos.scalar.And.value(And.java:127)
        at org.cactoos.scalar.And.value(And.java:52)
        at org.cactoos.func.IoCheckedFunc.apply(IoCheckedFunc.java:66)
        ... 3 more
Caused by: java.io.FileNotFoundException: /tmp/java-bullshifier/output/ProjAhgodwwubpj/build/tmp/fatjar-stage/ch/qos/logback/core/LogbackException.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.jpeek.metrics.JavassistClasses.lambda$metrics$2(JavassistClasses.java:132)
        at org.cactoos.func.IoCheckedFunc.apply(IoCheckedFunc.java:66)
        at org.cactoos.func.UncheckedFunc.apply(UncheckedFunc.java:59)
        ... 18 more

[0crat]

Job gh:yegor256/jpeek#40 is in scope.

[yegor256]

@wolkenschieber I believe this problem is fixed already, could you please check?

[wolkenschieber]

I replayed the inital scenario. Good news is that no Too many open files exception occured. This closes this issue.

On the otherside jpeek didn't finish the analysis. After calculating some hours it ran out of memory:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
        at java.lang.StringBuffer.append(StringBuffer.java:367)
        at java.io.StringWriter.write(StringWriter.java:94)
        at net.sf.saxon.tree.tiny.CharSlice.write(CharSlice.java:254)
        at net.sf.saxon.serialize.XMLEmitter.writeCharSequence(XMLEmitter.java:720)
        at net.sf.saxon.serialize.XMLEmitter.characters(XMLEmitter.java:653)
        at net.sf.saxon.serialize.XMLIndenter.indent(XMLIndenter.java:318)
        at net.sf.saxon.serialize.XMLIndenter.endElement(XMLIndenter.java:211)
        at net.sf.saxon.event.NamespaceReducer.endElement(NamespaceReducer.java:228)
        at net.sf.saxon.event.ComplexContentOutputter.endElement(ComplexContentOutputter.java:576)
        at net.sf.saxon.event.ProxyReceiver.endElement(ProxyReceiver.java:182)
        at net.sf.saxon.event.NamespaceReducer.endElement(NamespaceReducer.java:228)
        at net.sf.saxon.dom.DOMSender.outputElement(DOMSender.java:337)
        at net.sf.saxon.dom.DOMSender.walkNode(DOMSender.java:247)
        at net.sf.saxon.dom.DOMSender.outputElement(DOMSender.java:335)
        at net.sf.saxon.dom.DOMSender.walkNode(DOMSender.java:247)
        at net.sf.saxon.dom.DOMSender.outputElement(DOMSender.java:335)
        at net.sf.saxon.dom.DOMSender.walkNode(DOMSender.java:247)
        at net.sf.saxon.dom.DOMSender.outputElement(DOMSender.java:335)
        at net.sf.saxon.dom.DOMSender.walkNode(DOMSender.java:247)
        at net.sf.saxon.dom.DOMSender.outputElement(DOMSender.java:335)
        at net.sf.saxon.dom.DOMSender.walkNode(DOMSender.java:247)
        at net.sf.saxon.dom.DOMSender.send(DOMSender.java:97)
        at net.sf.saxon.dom.DOMObjectModel.sendDOMSource(DOMObjectModel.java:280)
        at net.sf.saxon.dom.DOMObjectModel.sendSource(DOMObjectModel.java:264)
        at net.sf.saxon.event.Sender.send(Sender.java:204)
        at net.sf.saxon.jaxp.IdentityTransformer.transform(IdentityTransformer.java:374)
        at org.xembly.Xembler.xml(Xembler.java:256)
        at org.xembly.Xembler.xmlQuietly(Xembler.java:221)
        at org.jpeek.Report.xml(Report.java:167)

The generated project consisted of 29000 java files and approx 2000000 lines of text.

[0crat]

Oops! There is no order for job gh:yegor256/jpeek#40.

[0crat]

Job gh:yegor256/jpeek#40 is now out of scope.