Packed_Data_Fields and bit fields do not validate as expected

[jordanpadams]

Describe the bug
From Dick Simpson:

Upon encountering Field_Binary.Packed_Data_Fields, Validate becomes confused and returns error messages of the form

  ERROR  [error.table.field_value_overlap]   table 6, record 1, field 2: The field is overlapping with the next field. Current field ends at 8. Next field starts at 4

where no such overlap exists. From the typical error message, it appears that Validate is ignoring the content of the Field_Binary with packed data and is not adjusting the pointer to the correct beginning of the next field. It also appears that Validate is not distinguishing between fields and bit fields; and it is not able to parse at finer resolution than integer bytes.

Validate operation on the attached product odf07155.{dat,xml} generates 33,589 error messages of the type above (see odf07155-report.docx, also attached).

I recommend first studying records in Tables 8, 10, and 12 (which all have the same format); each is short and has only a pair of bit fields in Field_Binary #5. Records in Table 6 are more complex; they have 4 instances of packed data (out of 7 Field_Binary objects). Those packed data objects have between 2 and 9 bit fields, each of which can have a length between 1 and 24 bits. I have attached a spreadsheet outlining the structures in this product type (Record_Structure_with_Bit_Fields.xlsx).

This error makes validation of migrated radio science data from missions prior to 2019 impossible in most cases, since the ODF (Orbit Data File) was very common when there was a gravity investigation (most missions start with gravity, then add other radio science investigations). Prior to 2001, all radio science investigations had products with bit fields. Going forward, bit fields are likely to be uncommon; but they are an option for certain radio science investigations with high sampling rates. They might also be chosen for mapping radar instruments, and they may occur in spacecraft telemetry data.

Test Data
bit_field_issue.tar.gz

See #209 for additional details for a similar issue

[jordanpadams]

Additional comments from Dick:

There seem to be several threads coming together here ...

whether the ticket you mentioned is for the same problem they're having

The bug report from Federico Salvioli appears to be for the same problem we are having.

or that they will file a bug report if it is something different

I filed a bug report in JIRA on 22 March (see SW-48). There is plenty of documentation included; but I can provide more if needed. SW-48 was almost immediately assigned to Galen Hollins; I've not heard of Galen before, nor have I heard from him or anyone else about the report since.

On our target binary file of 89 kB, Validate 1.22.1 returns over 33k 'overlap' error messages. There were errors only in Field_Binary objects which contained Packed_Data_Fields.

L5.PRP.VA.22 - The tool shall verify the following when the data file as referenced from a product label contains a binary table: (L4.PRP.3, UC 3.1, UC 3.2)
...
Fields that are “packed” in the table matches the defined number of bit-fields.
Fields that are “packed” in the table have bit-field values that match the defined data type.

I'm not sure how to interpret these requirements. The fact that Validate reports errors and seems to be completely confused about where the fields start and end suggests that even a minimal interpretation of the requirements is not being met.

There was some commentary that Packed_Data_Fields was not an allowed structure in Product_Observational and that SBN viewed DSN and radio science exceptions as "temporary".

In general, the policy has prohibited Packed_Data_Fields within Product_Observational for the past 3 years; but the exceptions exist. The policy also allows Packed_Data_Fields in non-observational subclasses. ESA/PSA apparently has no restrictions. So Validate needs to work in a variety of situations both within the NASA community and for our international partners. SBN does not, by itself, set PDS policy; radio science needs the exceptions. When the policy was being discussed, Todd King thought PPI might need it also for some P&F data; but I don't know what became of that concern.

The complete policy reads as follows; it obviously only applies to PDS:

Effective immediately, no PDS4 Product_Observational may contain a Packed_Data_Fields object. Packed_Data_Fields is allowed in other Product subclasses, if an unpacked version of the same data is also provided. DSN data and raw radio science data are exempt from this restriction.

[jordanpadams]

See comments on #224