Large Data File Uncaught Exception Error

[jordanpadams]

From the end user:

Hi,
I am finishing up an old data set for submission to PDS for Galileo. I have one last error from Validate to try to fix.

I am getting the following error on a bundle with data files and only one data file gets this error. It is the largest data file by far, 556 MB with 809344 records. The data files have two headers followed by the data all in Table_Delimited.

FAIL: file:/Users/kustemb1/project/GalileoEPD/galileo_epd_corrected_calibrated_higherorder/data/high_res/high_res_1997.xml
ERROR [error.vadidation.internal_error] Uncaught exception while validating: null

I wasn’t sure what the error meant and wondered if there was a limit on data file size or the number of records? I am running it on a Mac but I got the same error in linux.

Thanks,
Martha

[jordanpadams]

FROM MARTHA:

Hi,
You can get the set of files from http://sd-www.jhuapl.edu/Galileo_EPD/web_available_data/galileo_epd_corrected_calibrated_higherorder.tar.gz

Sorry to make you get the entire tarset but I couldn’t get the dropoff at APL to work with my password. So the XML and data files with the problem are galileo_epd_corrected_calibrated_higherorder/data/high_res/1997

Martha

[jordanpadams]

Capturing my e-mail reply to this issue:

Hey Jordan/Galen/Dan,

I looked into this a bit this morning and it does look like there is a heap space issue with what Martha is reporting. I ran it in Eclipse and that is when I saw the error. When I upped the memory  to 4GB-6GB like so in the Validation startup shell script:

-Xms4096m -Xmx6144m
It runs to completion. I think in order for OutOfMemoryError issues to appear in the report, it can be caught by placing a catch block at [2].

-Mike

 
[1] https://github.jpl.nasa.gov/PDSEN/pds4-java/blob/e4a87b22d57ccb4c3a53a6752ab077c16eb66598/preparation/validate/src/main/resources/bin/validate#L50

[2] https://github.jpl.nasa.gov/PDSEN/pds4-java/blob/e4a87b22d57ccb4c3a53a6752ab077c16eb66598/preparation/core/src/main/java/gov/nasa/pds/tools/validate/rule/pds4/TableDataContentValidationRule.java#L194

[hhlee445]

Jordan/Mike,

By any chance, do you have a test data for this issue?
The URL listed above (http://sd-www.jhuapl.edu/Galileo_EPD/web_available_data/galileo_epd_corrected_calibrated_higherorder.tar.gz) is not available.

[jordanpadams]

@hhlee445 I think I found it:

http://sd-www.jhuapl.edu/Galileo_EPD/web_available_data/galileo_epd_corrected_calibrated_higherorder_pds.tar.gz

[hhlee445]

Added a problem definition to the problemListener in ValidationTask class to report an unexpected exception.
issue_8_report.txt

However, the uncaught error disappears with maven 3.