Validate does not catch NaNs in Binary Tables #514

agicquelb · 2022-06-09T21:03:41Z

I found Nan in Binary_Table. The validation tool didn't give me an error. I was looking at the data and I thought the nan. Is the validation tool not supposed to return an error? Also I can't upload file so I will see how I can send the example file to you.

Tightly coupled with #529 and #434

jordanpadams · 2022-06-09T21:19:40Z

@agicquelb is this bug supposed to be for validate? if so, could you also add how you ran the tool and the version you are running?

also, for uploading data, if you zip/tar the data/label up you can attach it.

agicquelb · 2022-06-09T21:23:48Z

@jordanpadams yes it's for validate.

I'm doing validate */*xml and the version is Version 2.2.0.

attached are the zip files.

pep_0396683517_0x691_sci.zip

jordanpadams · 2022-06-28T20:29:23Z

@rchenatjpl can you help triage this for us? Is this an actual bug?

rchenatjpl · 2022-06-28T20:42:32Z

Hi, @agicquelb , is there a PDS4 label to go with this? The .zip contained a .fit and a .lbl, which is PDS3. I believe the validate tool just ignores all the files in the directory and reports 0 errors.
@jordanpadams , I haven't been able to run the PDS3 vtool in a long time. validate -h claims that 'validate -R pds3.volume' does something. Does it actually validate PDS3 labels? If you don't know and think this is worth investigating, please tell me and I will test stuff out.

agicquelb · 2022-06-28T21:22:03Z

@jordanpadams @rchenatjpl attached is a zip file with the PDS4 label that I created. I never tried to validate PDS3 labels
20180818_039685.zip

jordanpadams · 2022-06-28T21:26:04Z

thanks @agicquelb

@rchenatjpl it runs the PDS3 Volume Validator (it does not validate individual labels)

rchenatjpl · 2022-06-28T21:43:51Z

Thanks, @agicquelb , that's great. Can you tell me where any NaN is - which Array_2D_Image (or Table_Binary), and within that, which Line and Sample (or record and field)? Or I could hunt down the binary.

Or maybe that's where the work lies. I'll start hunting.

agicquelb · 2022-06-28T22:02:43Z

@rchenatjpl For example the column L15S00FLUX in the EXTENSION_FLUX_TABLE

rchenatjpl · 2022-06-28T22:21:20Z

I'll look at your last comment later tonight, @agicquelb
Before I got it, I hacked up the .fit and .xml to insert NaN as well a neg and a pos Infinity, and I'm happy to say that validate caught them. The two hacked files are in the attached, as well as a screen capture of their hex values. Validate says:

% validate  -t x.xml
PDS Validate Tool Report

Configuration:
   Version                       2.2.0
   Date                          2022-06-28T22:18:31Z

Parameters:
   Targets                       [file:/Users/rchen/Desktop/x.xml]
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      /Users/rchen/PDS4tools/validate/resources/registered_context_products.json



Product Level Validation Results

  FAIL: file:/Users/rchen/Desktop/x.xml
    Begin Content Validation: file:/Users/rchen/Desktop/x.fit
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (1, 5): Value is not within the valid range of the data type 'IEEE754MSBSingle': Infinity
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (1, 9): Value is not within the valid range of the data type 'IEEE754MSBSingle': -Infinity
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (1, 13): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (1, 15): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
    End Content Validation: file:/Users/rchen/Desktop/x.fit
        1 product validation(s) completed

Summary:

  4 error(s)
  0 warning(s)

  Product Validation Summary:
    0          product(s) passed
    1          product(s) failed
    0          product(s) skipped

  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped

  Message Types:
    4            error.array.value_out_of_data_type_range

End of Report
Completed execution in 1872 ms

Archive.zip

rchenatjpl · 2022-06-28T22:29:02Z

yes, I see it (NaN) called out in pds4_viewer but not validate. I'll check this tonight

rchenatjpl · 2022-06-29T08:20:35Z

@agicquelb You are right, validate has a bug. It validates Array correctly but Table incorrectly. In the attachment, 4array.xml reads 4.bin as an array; 4table.xml reads 4.bin as a table. 4.png shows what pds4_viewer (correctly) thinks for each. Validate's output:

% validate -t 4array.xml
PDS Validate Tool Report
Configuration:
   Version                       2.2.0
   Date                          2022-06-29T08:11:54Z
Parameters:
   Targets                       [file:/Users/rchen/Desktop/4array.xml]
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      /Users/rchen/PDS4tools/validate/resources/registered_context_products.json
Product Level Validation Results
  FAIL: file:/Users/rchen/Desktop/4array.xml
    Begin Content Validation: file:/Users/rchen/Desktop/4.bin
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (1, 1): Value is not within the valid range of the data type 'IEEE754MSBSingle': Infinity
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (1, 2): Value is not within the valid range of the data type 'IEEE754MSBSingle': Infinity
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (2, 1): Value is not within the valid range of the data type 'IEEE754MSBSingle': -Infinity
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (2, 2): Value is not within the valid range of the data type 'IEEE754MSBSingle': -Infinity
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (4, 1): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (4, 2): Value is not within the valid range of the data type 'IEEE754MSBSingle': NaN
    End Content Validation: file:/Users/rchen/Desktop/4.bin
        1 product validation(s) completed
Summary:
  6 error(s)
  0 warning(s)
  Product Validation Summary:
    0          product(s) passed
    1          product(s) failed
    0          product(s) skipped
  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped
  Message Types:
    6            error.array.value_out_of_data_type_range
End of Report
Completed execution in 2680 ms
% 
% 
% validate -t 4table.xml
PDS Validate Tool Report
Configuration:
   Version                       2.2.0
   Date                          2022-06-29T08:12:03Z
Parameters:
   Targets                       [file:/Users/rchen/Desktop/4table.xml]
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      /Users/rchen/PDS4tools/validate/resources/registered_context_products.json
Product Level Validation Results
  PASS: file:/Users/rchen/Desktop/4table.xml
        1 product validation(s) completed
Summary:
  0 error(s)
  0 warning(s)
  Product Validation Summary:
    1          product(s) passed
    0          product(s) failed
    0          product(s) skipped
  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped
End of Report
Completed execution in 2346 ms
%

4.zip

rchenatjpl · 2022-06-29T08:39:03Z

Similarly for 8-byte values

% validate -t 8array.xml
PDS Validate Tool Report
Configuration:
   Version                       2.2.0
   Date                          2022-06-29T08:35:41Z
Parameters:
   Targets                       [file:/Users/rchen/Desktop/8array.xml]
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      /Users/rchen/PDS4tools/validate/resources/registered_context_products.json
Product Level Validation Results
  FAIL: file:/Users/rchen/Desktop/8array.xml
    Begin Content Validation: file:/Users/rchen/Desktop/8.bin
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (1, 1): Value is not within the valid range of the data type 'IEEE754MSBDouble': Infinity
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (1, 2): Value is not within the valid range of the data type 'IEEE754MSBDouble': -Infinity
      ERROR  [error.array.value_out_of_data_type_range]   array 1, location (2, 2): Value is not within the valid range of the data type 'IEEE754MSBDouble': NaN
    End Content Validation: file:/Users/rchen/Desktop/8.bin
        1 product validation(s) completed
Summary:
  3 error(s)
  0 warning(s)
  Product Validation Summary:
    0          product(s) passed
    1          product(s) failed
    0          product(s) skipped
  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped
  Message Types:
    3            error.array.value_out_of_data_type_range
End of Report
Completed execution in 2225 ms
% 
% 
% validate -t 8table.xml 
PDS Validate Tool Report
Configuration:
   Version                       2.2.0
   Date                          2022-06-29T08:35:59Z
Parameters:
   Targets                       [file:/Users/rchen/Desktop/8table.xml]
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      /Users/rchen/PDS4tools/validate/resources/registered_context_products.json
Product Level Validation Results
  PASS: file:/Users/rchen/Desktop/8table.xml
        1 product validation(s) completed
Summary:
  0 error(s)
  0 warning(s)
  Product Validation Summary:
    1          product(s) passed
    0          product(s) failed
    0          product(s) skipped
  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped
End of Report
Completed execution in 2284 ms
%

8.zip
Ball is yours, @jordanpadams

agicquelb · 2022-09-16T17:18:05Z

@jordanpadams Any update on this issue about the fact the validate doesn't catch NAN in Binary Tables?

jordanpadams · 2022-09-21T20:08:15Z

@agicquelb unfortunately, we are significantly behind in our backlog of bugs. this is in our plans for next build (June 2023), but as soon as we get this fixed, we can send you over a beta version to test out. sorry for the inconvenience.

al-niessner · 2023-01-26T15:46:24Z

@jordanpadams

Reading through this ticket it seems that tables are not finding NaN and +/- Inf as arrays are. Am I reading this correctly?

jordanpadams · 2023-01-26T16:35:09Z

@al-niessner that sounds about right to me!

* unit test to show fix works * fix INF/NAN check Co-authored-by: Al Niessner <Al.Niessner@xxx.xxx>

rchenatjpl · 2023-05-17T19:49:51Z

@agicquelb @al-niessner @jordanpadams: Can we step back?

Are NaN and Inf illegal in binary data? I should have asked this a year ago. In the PDS4 Standards doc, Table 5A-3, the row for ASCII_Real says "PDS does not allow ... INF... -INF...NaN". So I think it only applies to the ASCII data types, not the binary ones. If this is debatable, we should call others in.
Assuming INF and NaN are legal for binary types, what should validate do? I'd say it should do nothing or throw a warning but definitely not throw an error.

jordanpadams · 2023-05-18T15:24:05Z

from Dick Simpson:

...NaN is embedded in the IEEE754 standard, which we have adopted; there's little to be gained by claiming it's not acceptable in PDS4, so we've not attempted to say so. Further, there are multiple bit patterns that designate specific flavors of 'NaN'; on the off-chance that some of our data providers want to use those, there may be some benefit in letting them take advantage of the capability.

As for what to do, I think the present practice is acceptable — that is, nothing. If we accept that 'NaN' is allowed under the IEEE754 specification for binary data, then there's no reason to issue either a warning or an error. We do not want people to use 'NaN' in ASCII_Real fields, but that's a separate issue.

I would use the same logic for accepting '±Inf'.

jordanpadams · 2023-05-18T15:24:51Z

@al-niessner per you comment here, hopefully the discussion above resolves which is the right answer?

rsjoyner · 2023-05-18T16:43:05Z

My recollection is that “NaN” and “Inf” and other such identifiers are NOT legal in either ascii or binary data.
The God of PDS3 and PDS4 created a class of Special Constants where NaN / Inf / + each have to be represented by a number (of the same data type).

For example, the number “666.66” in the string of bits file could be defined to be “error_constant”. So, each time “666.66” is located by s/w, the s/w knows that it represents “NaN” or ???.

When identifying a specific number to represent one of the <special_constants>, the data supplier obviously must choose a number that is guaranteed NOT to be present in the “normal” range of numbers.

Does this make sense ?

<xs:complexType name="Field_Binary">
xs:annotation
xs:documentationThe Field_Binary class defines a field of a
binary record or a field of a binary group.</xs:documentation>
</xs:annotation>
xs:complexContent
<xs:extension base="pds:Field">
xs:sequence
<xs:element name="field_location" type="pds:field_location" minOccurs="1" maxOccurs="1"> </xs:element>
<xs:element name="data_type" type="pds:data_type" minOccurs="1" maxOccurs="1"> </xs:element>
<xs:element name="field_length" type="pds:field_length" minOccurs="1" maxOccurs="1"> </xs:element>
<xs:element name="field_format" type="pds:field_format" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="unit" type="pds:unit" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="scaling_factor" type="pds:scaling_factor" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="value_offset" type="pds:value_offset" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="description" type="pds:description" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="Special_Constants" type="pds:Special_Constants" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="Field_Statistics" type="pds:Field_Statistics" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="Packed_Data_Fields" type="pds:Packed_Data_Fields" minOccurs="0" maxOccurs="1"> </xs:element>
</xs:sequence>
</xs:extension>
</xs:complexContent>
</xs:complexType>

<xs:complexType name="Special_Constants">
xs:annotation
xs:documentationThe Special Constants class provides a set of
values used to indicate special cases that occur in the
data.</xs:documentation>
</xs:annotation>
xs:sequence
<xs:element name="saturated_constant" type="pds:saturated_constant" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="missing_constant" type="pds:missing_constant" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="error_constant" type="pds:error_constant" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="invalid_constant" type="pds:invalid_constant" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="unknown_constant" type="pds:unknown_constant" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="not_applicable_constant" type="pds:not_applicable_constant" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="valid_maximum" type="pds:valid_maximum" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="high_instrument_saturation" type="pds:high_instrument_saturation" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="high_representation_saturation" type="pds:high_representation_saturation" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="valid_minimum" type="pds:valid_minimum" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="low_instrument_saturation" type="pds:low_instrument_saturation" minOccurs="0" maxOccurs="1"> </xs:element>
<xs:element name="low_representation_saturation" type="pds:low_representation_saturation" minOccurs="0" maxOccurs="1"> </xs:element>
</xs:sequence>
</xs:complexType>

agicquelb added bug Something isn't working needs:triage labels Jun 9, 2022

agicquelb assigned jordanpadams Jun 9, 2022

jordanpadams transferred this issue from NASA-PDS/mi-label Jun 9, 2022

jordanpadams added B13.0 s.medium and removed needs:triage labels Jul 22, 2022

jordanpadams removed their assignment Jul 22, 2022

jordanpadams mentioned this issue Jul 22, 2022

ERROR [error.array.value_out_of_min_max_range] evaluation is not correct #529

Closed

jordanpadams changed the title ~~nan in Binary Table~~ Validate does not catch NaNs in Binary Tables Aug 25, 2022

jordanpadams added the B13.1 label Jan 25, 2023

jordanpadams assigned al-niessner Jan 25, 2023

jordanpadams removed the B13.0 label Jan 25, 2023

al-niessner mentioned this issue Jan 26, 2023

issue #514: table not finding NaNs and Infs #588

Merged

jordanpadams closed this as completed in #588 Jan 26, 2023

jordanpadams pushed a commit that referenced this issue Jan 26, 2023

issue #514: table not finding NaNs and Infs (#588)

d9f6351

* unit test to show fix works * fix INF/NAN check Co-authored-by: Al Niessner <Al.Niessner@xxx.xxx>

jordanpadams mentioned this issue Apr 12, 2023

Scaled value min/max error does not display scaled value #434

Closed

miguelp1986 added the i&t.done label May 12, 2023

al-niessner mentioned this issue May 17, 2023

Update to allow NaNs for IEEE754 data types #646

Merged

jordanpadams mentioned this issue May 31, 2023

Validate gives errors for 'NaN' and 'Inf' values in IEEE754 data #644

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validate does not catch NaNs in Binary Tables #514

Validate does not catch NaNs in Binary Tables #514

agicquelb commented Jun 9, 2022 •

edited by jordanpadams

Loading

jordanpadams commented Jun 9, 2022

agicquelb commented Jun 9, 2022 •

edited

Loading

jordanpadams commented Jun 28, 2022

rchenatjpl commented Jun 28, 2022

agicquelb commented Jun 28, 2022

jordanpadams commented Jun 28, 2022

rchenatjpl commented Jun 28, 2022 •

edited

Loading

agicquelb commented Jun 28, 2022

rchenatjpl commented Jun 28, 2022 •

edited by jordanpadams

Loading

rchenatjpl commented Jun 28, 2022 •

edited

Loading

rchenatjpl commented Jun 29, 2022

rchenatjpl commented Jun 29, 2022

agicquelb commented Sep 16, 2022

jordanpadams commented Sep 21, 2022

al-niessner commented Jan 26, 2023

jordanpadams commented Jan 26, 2023

rchenatjpl commented May 17, 2023

jordanpadams commented May 18, 2023

jordanpadams commented May 18, 2023

rsjoyner commented May 18, 2023

Validate does not catch NaNs in Binary Tables #514

Validate does not catch NaNs in Binary Tables #514

Comments

agicquelb commented Jun 9, 2022 • edited by jordanpadams Loading

jordanpadams commented Jun 9, 2022

agicquelb commented Jun 9, 2022 • edited Loading

jordanpadams commented Jun 28, 2022

rchenatjpl commented Jun 28, 2022

agicquelb commented Jun 28, 2022

jordanpadams commented Jun 28, 2022

rchenatjpl commented Jun 28, 2022 • edited Loading

agicquelb commented Jun 28, 2022

rchenatjpl commented Jun 28, 2022 • edited by jordanpadams Loading

rchenatjpl commented Jun 28, 2022 • edited Loading

rchenatjpl commented Jun 29, 2022

rchenatjpl commented Jun 29, 2022

agicquelb commented Sep 16, 2022

jordanpadams commented Sep 21, 2022

al-niessner commented Jan 26, 2023

jordanpadams commented Jan 26, 2023

rchenatjpl commented May 17, 2023

jordanpadams commented May 18, 2023

jordanpadams commented May 18, 2023

rsjoyner commented May 18, 2023

agicquelb commented Jun 9, 2022 •

edited by jordanpadams

Loading

agicquelb commented Jun 9, 2022 •

edited

Loading

rchenatjpl commented Jun 28, 2022 •

edited

Loading

rchenatjpl commented Jun 28, 2022 •

edited by jordanpadams

Loading

rchenatjpl commented Jun 28, 2022 •

edited

Loading