As a user, I want to describe a relationship between an attribute in the model and the column of a table

[jordanpadams]

💪 Motivation

...so that I can use this relationship to ingest metadata within a table product into the appropriate attribute in the registry.

Specific to Product_Metadata_Supplemental (for now)

📖 Additional Details

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

[jordanpadams]

from @jshughes via email:

Defining a new class for the column header names is consistent with our overall object modeling paradigm. So, it seems reasonable to keep this approach as general as possible and allow a set of one or more related classes. In other words, the column headers should be allowed to reference attributes within an object model and not just the flat model forced by a single class. Even though the Product_Metadata_Supplemental table is flat, the underlaying model of “reality” may not be.

Ingest_LDD currently provides the functionalities that we need, including terminological entries for capturing aliases. These “table header” classes would be developed in parallel with the other classes in the namespace. In fact, there might be some overlap.

Using the titles of real/conceptual things is preferred, for example Rings_Geometry in the following.

rings:Rings_Geometry/rings:ring_radius

Continuing the example, it may be that “Rings_Geometry” is too generic and that another class level would be useful, for example:

rings:Rings_Geometry/rings:<Ring_Type>/rings:ring_radius

[tloubrieu-jpl]

We need to organize a meeting with @jordanpadams @tloubrieu-jpl to discuss how we should do that from @jshughes analysis

[tloubrieu-jpl]

Archive.zip

This is the result of the analysis done by @jshughes on this ticket to be discussed.

[tloubrieu-jpl]

@jshughes is progressing on this ticket

[jshughes]

Use Case #2 – Use <DD_Attribute.Terminological_Entry> to capture alias names for an attribute defined in Ingest_LDD.

@jordanpadams @tloubrieu-jpl

The “Mission Information Commons” LDD was used as a test case. Two PDS3 aliases were found for the attribute <Mission_Information.spacecraft_clock_start> and were added as terminological entries for the attribute. The two aliases, “SPACECRAFT_CLOCK_START_COUNT” and “SCLK_START_VALUE” can be found in the Ingest_LDD code snippet below. The complete Ingest_LDD file is attached. The SKOS relation name “exactMatch” was used.

No change was needed to the existing PDS4 common model.

New code was added to LDDTool to write a JSON file during Ingest_LDD processing. The output was generated using the “json-simple” JSON library.

The initial results are shown below. Under the key "Attribute Aliases" can be found the class name, “Mission_Information”, the attribute name, “spacecraft_clock_start”, followed by the two aliases, "SPACECRAFT_CLOCK_START_COUNT" and "SCLK_START_VALUE". Nested arrays are used to allow one or more classes, their attributes, and the aliases.

There are several candidate updates to consider.

1. Use the more complete attribute identifiers or xpaths, e.g., 0001_NASA_PDS_1.msn.Mission_Information.msn.spacecraft_clock_start
2. Add the SKOS relations name.
3. …

-- output-- (edited up for readability)

{"infoModelVersionId":"1.18.0.0",
"datetime":"2022-06-30T12:56:42",

"Attribute Aliases":
    [{"Mission_Information":
        [{"spacecraft_clock_start":
            ["SPACECRAFT_CLOCK_START_COUNT","SCLK_START_VALUE"]
                }]}],
"title":"PDS4 Terminological Entries"}

-- Ingest_LDD Snippet--

  <!-- Start attribute definitions for Surface_Mission class -->
  <DD_Attribute>
    <name>spacecraft_clock_start</name>
    <version_id>1.0</version_id>
    <local_identifier>spacecraft_clock_start</local_identifier>
    <nillable_flag>true</nillable_flag>
    <submitter_name>Susan Slavney</submitter_name>
    <!-- TODO finesse definition -->
    <definition>The spacecraft_clock_start is the value of the spacecraft clock at the beginning of the observation.</definition>

    <!--  Begin Terminological_Entry - Attribute -->
    
    <Terminological_Entry>
      <name>SPACECRAFT_CLOCK_START_COUNT</name>
      <definition>
        The spacecraft clock start count element provides the value of the spacecraft clock at the beginning of a time period of interest. Note: In the PDS, sclk start counts have been represented in the following ways: Voyager - Flight Data Subsystem (FDS) clock count (floating point 7.2) Mariner 9 - Data Automation Subsystem, Mariner 10 - FDS - spacecraft clock Mars Pathfinder - spacecraft clock
      </definition>
      <language>English</language>
      <preferred_flag>false</preferred_flag>
      <skos_relation_name>exactMatch</skos_relation_name>
      <External_Reference_Extended>
        <reference_text>Planetary Science Data Dictionary Document, JPL D-7116, Rev. F, June 11, 2008</reference_text>
        <description>Planetary Data System Version 3</description>
        <name>Planetary Science Data Dictionary Document</name>
        <url>https://pds.jpl.nasa.gov/datastandards/pds3/</url>
      </External_Reference_Extended>
    </Terminological_Entry> 
    
    <!--  End Terminological_Entry - Attribute -->
    
    <!--  Begin Terminological_Entry - Attribute -->
    
    <Terminological_Entry>
      <name>SCLK_START_VALUE</name>
      <definition>
        The sclk start value element is an alias for spacecraft clock start count which is used only by AMMOS-Magellan mission operations data files.
      </definition>
      <language>English</language>
      <preferred_flag>false</preferred_flag>
      <skos_relation_name>exactMatch</skos_relation_name>
      <External_Reference_Extended>
        <reference_text>Planetary Science Data Dictionary Document, JPL D-7116, Rev. F, June 11, 2008</reference_text>
        <description>Planetary Data System Version 3</description>
        <name>Planetary Science Data Dictionary Document</name>
        <url>https://pds.jpl.nasa.gov/datastandards/pds3/</url>
      </External_Reference_Extended>
    </Terminological_Entry> 
    
    <!--  End Terminological_Entry - Attribute -->

PDS4_MSN_IngestLDD_plus_TermEntry_StrtTime.zip

[jordanpadams]

Brought to DDWG today for initial review

[jimmie]

FIrst phase should receive OK shortly (~days).

[jshughes]

Latest JSON output for Term Mapping; Next step is to test with EPN-TAP mappings

PDS4_PDS_CCSDS_1I00_termentry_221201.txt

[jordanpadams]

Closing this requirement right now since the initial design and implementation is complete. Test cases to follow.

[miguelp1986]

Closing this requirement right now since the initial design and implementation is complete. Test cases to follow.

Hi @jordanpadams. Could we please get the test cases for this issue?

[jshughes]

@miguelp1986

In the attached zip file please find:

a) the script for running LDDTool with the -T option set - runapp_TermMap_AllUseCases.bat

b) the input file with the test cases - PDS4_MSN_IngestLDD_TermMap_AllUseCases.xml

-starting at line 761

1. Use Case 1 – PDS4 Attribute to PDS4 Attribute
2. Use Case 2.1 – PDS4 Attribute to PDS3 keyword
3. Use Case 2.2 – PDS4 Attribute to PDS3 keyword
4. Use Case 3 – External term (e.g. search keyword) mapped to a PDS4 Attribute.
5. Use Case 4 – Table column mapped to Existing PDS4 Attribute

c) the JSON termmap output file - PDS4_PDS_1K00_TM.JSON

d) the generated .xsd, .sch, and xml files.

Issue473.zip

[jordanpadams]

thanks @jshughes

[tloubrieu-jpl]

I&T Test involves, for each use case:

1. coding the input XML (see EPN example)
2. validate that it is valid against the schematron and schema of the latest stable IM (1K)
3. run the LDD tool from this input to generate OWL/SKOSS format
4. read the output.

[jshughes]

@miguelp1986 @tloubrieu-jpl @jordanpadams attached is the EPN-TAP LDD and the JSON output file.

Issue473_EPN.zip