Uncaught exception thrown when only 1 of schemas/schematrons are provided via command-line

[rsjoyner]

Checked for duplicates

No - I haven't checked

🐛 Describe the bug

I installed the Windows version of validate_3.3.1 today.

(1) It appears that when I try to validate a PDS4 product, validate throws errors. See below.

(2) when I run this command: validate -V or validate --help
It works. So, I am guessing no issues with the installation.

(3) when I run this command: validate -t 227RI_PROPRETRG001_IS____680_F1_039R_edited.xml >> log.txt

This is displayed:

D:\WINWORD\Data_Prep_HandBook\aaaVer_9_20130225\schemas\dictionary_CASSINI_mission\ATMOS_test_20231113> validate331 227RI_PROPRETRG001_IS____680_F1_039R.xml >> log.txt
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at gov.nasa.pds.tools.validate.rule.AbstractValidationRule.execute(AbstractValidationRule.java:64)
        at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:191)
        at gov.nasa.pds.tools.validate.task.ValidationTask.execute(ValidationTask.java:130)
        at gov.nasa.pds.tools.validate.task.BlockingTaskManager.submit(BlockingTaskManager.java:26)
        at gov.nasa.pds.tools.label.LocationValidator.validate(LocationValidator.java:285)
        at gov.nasa.pds.validate.ValidateLauncher.doValidation(ValidateLauncher.java:1454)
        at gov.nasa.pds.validate.ValidateLauncher.processMain(ValidateLauncher.java:1740)
        at gov.nasa.pds.validate.ValidateLauncher.main(ValidateLauncher.java:1784)
Caused by: java.lang.NullPointerException
        at gov.nasa.pds.tools.validate.TargetExaminer.isTargetBundleType(TargetExaminer.java:59)
        at gov.nasa.pds.tools.util.Utility.getTargetType(Utility.java:301)
        at gov.nasa.pds.tools.validate.ValidationTarget.<init>(ValidationTarget.java:47)
        at gov.nasa.pds.tools.validate.ValidationTarget.<init>(ValidationTarget.java:41)
        at gov.nasa.pds.tools.util.Utility.getValidationTarget(Utility.java:88)
        at gov.nasa.pds.tools.validate.ValidationProblem.<init>(ValidationProblem.java:69)
        at gov.nasa.pds.tools.label.CachedLSResourceResolver.resolveResource(CachedLSResourceResolver.java:207)
        at gov.nasa.pds.tools.label.XMLCatalogResolver.resolveResource(XMLCatalogResolver.java:381)
        at gov.nasa.pds.tools.validate.rule.pds4.LabelValidationRule.loadSingleSchemaIntoSources(LabelValidationRule.java:397)
        at gov.nasa.pds.tools.validate.rule.pds4.LabelValidationRule.validateLabelSchemas(LabelValidationRule.java:572)
        at gov.nasa.pds.tools.validate.rule.pds4.LabelValidationRule.validateLabel(LabelValidationRule.java:250)
        ... 12 more

🕵️ Expected behavior

I expected validate to validate the product using this command:

validate -t 227RI_PROPRETRG001_IS____680_F1_039R_edited.xml >> log.txt

📜 To Reproduce

1. unzip files
2. validate -t 227RI_PROPRETRG001_IS____680_F1_039R_edited.xml >> log.txt
3. review log.txt
   ...

🖥 Environment Info

• Version of this software: validate_3.3.1
• Operating System: Windows
  ...

📚 Version of Software Used

validate_3.3.1 for Windows. I installed using the zip.

🩺 Test Data / Additional context

ATMOS_test_20231113.zip

🦄 Related requirements

• 🦄 As a user, I want to be able to use both online and local schema/schematron files. #599

⚙️ Engineering Details

[al-niessner]

@jordanpadams

I do not have a windows machine.

[jordanpadams]

@al-niessner I see the same error on my Mac.

[jordanpadams]

@al-niessner After investigating, this may or may not be the same issue, but I am seeing an error because it can't find the schema and/or schematron online for PDS4_CASSINI_1J00_1700 (and maybe others).

If you provide those via command-line (and download the rest of the appropriate schemas/schematrons 20231115.zip, I get these results, which make more sense:

$ validate-3.3.1-bin/bin/validate -t ~/Downloads/ATMOS_test_20231113/227RI_PROPRETRG001_IS____680_F1_039R.xml  --schema ~/Downloads/20231115/PDS4_*.xsd --schematron ~/Downloads/20231115/PDS4_*.sch


PDS Validate Tool Report

Configuration:
   Version                       3.4.0-SNAPSHOT
   Date                          2023-11-16T05:13:22Z

Parameters:
   Targets                       [file:/Users/jpadams/Downloads/ATMOS_test_20231113/227RI_PROPRETRG001_IS____680_F1_039R.xml]
   User Specified Schemas        [file:/Users/jpadams/Downloads/20231115/PDS4_CASSINI_1J00_1700.xsd, file:/Users/jpadams/Downloads/20231115/PDS4_DISP_1I00_1510.xsd, file:/Users/jpadams/Downloads/20231115/PDS4_GEOM_1B10_1700.xsd, file:/Users/jpadams/Downloads/20231115/PDS4_SP_1G00_1310.xsd]
   User Specified Schematrons    [file:/Users/jpadams/Downloads/20231115/PDS4_CASSINI_1J00_1700.sch]
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      /Users/jpadams/proj/pds/pdsen/workspace/validate/validate-3.4.0-SNAPSHOT/resources/registered_context_products.json



Product Level Validation Results

  FAIL: file:/Users/jpadams/Downloads/ATMOS_test_20231113/227RI_PROPRETRG001_IS____680_F1_039R.xml
      ERROR  [error.label.schema]   line 103, 25: cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'cart:Cartography'.
      ERROR  [error.label.missing_file]   URI reference does not exist: file:/Users/jpadams/Downloads/ATMOS_test_20231113/227RI_PROPRETRG001_IS____680_F1_039R.LBL
    Begin Content Validation: file:/Users/jpadams/Downloads/ATMOS_test_20231113/227RI_PROPRETRG001_IS____680_F1_039R.DAT
      ERROR  [error.array.value_out_of_data_type_range]   array SPECTRAL_QUBE or index 1, location (1, 14, 562): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array SPECTRAL_QUBE or index 1, location (1, 15, 564): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array SPECTRAL_QUBE or index 1, location (1, 15, 565): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array SPECTRAL_QUBE or index 1, location (1, 16, 566): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array SPECTRAL_QUBE or index 1, location (1, 16, 567): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
...
      ERROR  [error.array.value_out_of_data_type_range]   array MEAN_RING_BORESIGHT_LOCAL_TIME or index 1, location (226, 199): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array MEAN_RING_BORESIGHT_LOCAL_TIME or index 1, location (226, 200): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array MEAN_RING_BORESIGHT_LOCAL_TIME or index 1, location (226, 201): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array MEAN_RING_BORESIGHT_LOCAL_TIME or index 1, location (226, 202): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array MEAN_RING_BORESIGHT_LOCAL_TIME or index 1, location (227, 195): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array MEAN_RING_BORESIGHT_LOCAL_TIME or index 1, location (227, 196): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array TOTAL_SPECTRAL_POWER or index 1, location (6, 556): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array TOTAL_SPECTRAL_POWER or index 1, location (208, 629): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array TOTAL_SPECTRAL_POWER or index 1, location (210, 585): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array TOTAL_SPECTRAL_POWER or index 1, location (215, 600): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
    End Content Validation: file:/Users/jpadams/Downloads/ATMOS_test_20231113/227RI_PROPRETRG001_IS____680_F1_039R.DAT
        1 product validation(s) completed

Summary:

  9499 error(s)
  0 warning(s)

  Product Validation Summary:
    0          product(s) passed
    1          product(s) failed
    0          product(s) skipped

  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped

  Message Types:
    9497         error.array.value_out_of_data_type_range
    1            error.label.missing_file
    1            error.label.schema

End of Report

[jordanpadams]

@al-niessner it looks like this may have not been implemented: #599

[al-niessner]

@jordanpadams @rsjoyner

I fixed the internal error and now have this summary (too much for full output):

Summary:

  9526 error(s)
  24 warning(s)

  Product Validation Summary:
    0          product(s) passed
    1          product(s) failed
    0          product(s) skipped

  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped

  Message Types:
    9497         error.array.value_out_of_data_type_range
    25           error.label.unresolvable_resource
    2            error.label.schematron
    1            error.label.missing_file
    1            error.label.schema
    24           warning.label.schema

End of Report
Completed execution in 22417 ms

The only error of import here is error.label.schematron. Given the content of the test file, I took it to mean that those are the schematron that are desired (the URL in the XML produce a 404). In order to use those two schematron files, changed how validate is run to:

$ validate -t github760/227RI_PROPRETRG001_IS____680_F1_039R.xml -S github760/PDS4_CASSINI_1D00_1500.sch -x  github760/PDS4_CASSINI_1J00_1700.xsd

Nov 17, 2023 12:39:10 PM gov.nasa.pds.tools.label.LocationValidator setLabelExtension
INFO: setLabelExtension: xml
Nov 17, 2023 12:39:10 PM gov.nasa.pds.tools.label.LocationValidator validate
INFO: location file:github760/227RI_PROPRETRG001_IS____680_F1_039R.xml
Nov 17, 2023 12:39:10 PM gov.nasa.pds.tools.label.LocationValidator validate
INFO: Using validation style 'PDS4 Label' for location file:github760/227RI_PROPRETRG001_IS____680_F1_039R.xml
Nov 17, 2023 12:39:10 PM gov.nasa.pds.tools.validate.task.ValidationTask execute
INFO: Starting validation task for location 'file:github760/227RI_PROPRETRG001_IS____680_F1_039R.xml'

PDS Validate Tool Report

Configuration:
   Version                       3.4.0-SNAPSHOT
   Date                          2023-11-17T20:39:08Z

Parameters:
   Targets                       [file:github760/227RI_PROPRETRG001_IS____680_F1_039R.xml]
   User Specified Schemas        [file:github760/PDS4_CASSINI_1J00_1700.xsd]
   User Specified Schematrons    [file:github760/PDS4_CASSINI_1D00_1500.sch]
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      registered_context_products.json



Product Level Validation Results

  FAIL: file:github760/227RI_PROPRETRG001_IS____680_F1_039R.xml
      ERROR  [error.label.schema]   line 103, 25: cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'cart:Cartography'.
      ERROR  [error.label.schema]   line 126, 22: cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'geom:Geometry'.
      ERROR  [error.label.schema]   line 160, 36: cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'sp:Spectral_Characteristics'.
      ERROR  [error.label.schema]   line 186, 30: cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'disp:Display_Settings'.
      ERROR  [error.label.missing_file]   URI reference does not exist: file:github760/227RI_PROPRETRG001_IS____680_F1_039R.LBL
    Begin Content Validation: file:github760/227RI_PROPRETRG001_IS____680_F1_039R.DAT
    End Content Validation: file:github760/227RI_PROPRETRG001_IS____680_F1_039R.DAT
        1 product validation(s) completed
Nov 17, 2023 12:39:14 PM gov.nasa.pds.tools.validate.task.ValidationTask execute
INFO: Validation complete for location 'file:github760/227RI_PROPRETRG001_IS____680_F1_039R.xml'

<... CUT OUT NAN Error that we already know about and are working ...>

Summary:

  9502 error(s)
  0 warning(s)

  Product Validation Summary:
    0          product(s) passed
    1          product(s) failed
    0          product(s) skipped

  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped

  Message Types:
    9497         error.array.value_out_of_data_type_range
    4            error.label.schema
    1            error.label.missing_file

End of Report
Completed execution in 7236 ms

You should be able to call validate as shown above without any of the current changes and get the same errors that are shown here. You can also update your XML to point to the files but then would have to modify it if the schema ever moved to the https you list. See #747 and #761 for progress on the 9497 NAN false positives that you have.

[jordanpadams]

@al-niessner 🎉

any idea what happened to the implementation for #599 ? That was supposed to provide support for grabbing the online schemas if they are not provided via command-line.

[al-niessner]

@al-niessner 🎉

any idea what happened to the implementation for #599 ? That was supposed to provide support for grabbing the online schemas if they are not provided via command-line.

@jordanpadams

Nothing happened to #599 implementation and it did do the job. The newest problem is that when it could not find something it reports a problem but the problem reporter thinks all target URLs it is given are file. So a glob of things would cascade to make it fail with a null pointer but it is a null pointer in creating the problem not detecting it.

When testing the addition of schema on and off line, never tried a file that did not exist online and not give it offline so never noticed error reporting was going to give us a null pointer exception.