Heap space error while processing multiple files

[bitsgalore]

Dev Effort

1D

Description

While trying to process a directory with 24 PDFs (total size 110 MB; size of largest PDF about 50 MB), veraPDF crashed with a Java heap space error. Here's the command-line I used:

~/verapdf/verapdf -x --policyfile /home/johan/pdfPolicyVeraPDF/schemas/demo-policy.sch ~/pdfAcrobatEngineering/multimedia/*.pdf > multimedia.xml

Errors/warnings sent to stderr:

WARNING: The JVM appears to have run out of memory
java.lang.OutOfMemoryError: Java heap space
	at com.adobe.xmp.impl.ByteBuffer.ensureCapacity(ByteBuffer.java:322)
	at com.adobe.xmp.impl.ByteBuffer.<init>(ByteBuffer.java:88)
	at com.adobe.xmp.impl.XMPMetaParser.parseXmlFromInputStream(XMPMetaParser.java:162)
	at com.adobe.xmp.impl.XMPMetaParser.parseXml(XMPMetaParser.java:128)
	at com.adobe.xmp.impl.XMPMetaParser.parse(XMPMetaParser.java:77)
	at com.adobe.xmp.XMPMetaFactory.parse(XMPMetaFactory.java:100)
	at com.adobe.xmp.impl.VeraPDFMeta.parse(VeraPDFMeta.java:75)
	at org.verapdf.gf.model.GFModelParser.obtainFlavour(GFModelParser.java:128)
	at org.verapdf.gf.model.GFModelParser.<init>(GFModelParser.java:82)
	at org.verapdf.gf.model.GFModelParser.createModelWithFlavour(GFModelParser.java:104)
	at org.verapdf.pdfa.VeraFoundry.createParser(VeraFoundry.java:80)
	at org.verapdf.pdfa.VeraFoundry.createParser(VeraFoundry.java:86)
	at org.verapdf.processor.ProcessorImpl.process(ProcessorImpl.java:113)
	at org.verapdf.processor.BatchFileProcessor.processItem(BatchFileProcessor.java:98)
	at org.verapdf.processor.BatchFileProcessor.processList(BatchFileProcessor.java:74)
	at org.verapdf.processor.AbstractBatchProcessor.process(AbstractBatchProcessor.java:102)
	at org.verapdf.cli.VeraPdfCliProcessor.processFiles(VeraPdfCliProcessor.java:157)
	at org.verapdf.cli.VeraPdfCliProcessor.processPaths(VeraPdfCliProcessor.java:131)
	at org.verapdf.cli.VeraPdfCli.main(VeraPdfCli.java:99)

All output of the PDFs that were processed before the crash were lost in the process. Stdout:

The JVM appears to have run out of memory
Memory Use: 529M/1975M
To increase the memory available to the JVM please assign the JAVA_OPTS environment variable.
The examples below increase the maximum heap available to the JVM to 2GB:
 - Mac or Linux users: 
   export JAVA_OPTS="-Xmx2048m"
 - Windows users: 
   SET JAVA_OPTS="-Xmx2048m"

It's not clear to me if the error arises from output that is kept in the buffer or something else. Using VeraPDF 1.4.7.

[bitsgalore]

Update: after some additional testing I'm now pretty sure the error is not related to the output buffer, but occurs for one fairly large (49 MB) PDF. It's the VolvoS40V50-Full.pdf file which used to be available from the Adobe Acrobat Engineering website (before that site was taken down):

http://acroeng.adobe.com/Test_Files/classic_multimedia//VolvoS40V50-Full.pdf

I have a local copy of the file; happy to send it over if that helps (if so just drop me a line at twitter@bitsgalore.org or my KB email address).

[carlwilson]

Hi @bitsgalore I've tested this on the latest development install against the adobe test corpus including the Volvo file. It now takes approx 2 seconds to process straightforwardly from a standard installation with JAVA_OPTS unassigned. I'm happy for somebody else to test it if you want to, I'll give it a go on the Mac at least just to see. If that gives a similar result I'm ready to close this.

[carlwilson]

Update is that the Mac managed this in 5.86 seconds with no memory issues from a standard install. Ready to close.