Add support for data types based on what can be supported in XML and JSON schema

[david-waltermire]

User Story:

As an OSCAL model developer, I need to be able to define strict data types for information elements within my models that translate to validation of these types in XML and JSON schema.

Goals:

There are three major goals for this issue:

• First, it would be good to make definition of datatypes more uniform across the Metaschema constructs. Data types are defined inconsistently across various OSCAL Metaschema constructs. In define-field and define-flag the @as attribute is used to define the data type. The value "mixed" is used to indicate that simple markup is allowed, which excludes the use of tables and lists. On define-assembly the <prose> element is used to indicate that complex markup can be included, that allows the use of tables and lists.

  The following changes will allow data typing to be made more consistent:

  • A new attribute @as-type will be allowed on define-flag, define-field, and flag (see Allow flags to be defined within a field or assembly #435), which will define the data type of the field or flag value. The attribute @as-type is preferred over the current @as since it is more human-readable.

  • The <prose> element in the Metaschema can be replaced with use of <field named="prose" as-type="markup-multiline" @xml-unwrap="yes">.

  • The new attribute @xml-unwrap allows for the content model to be "unwrapped" in XML, if desired, by omitting the outer element based on the field name. The default behavior is to wrap in XML if the @xml-unwrap attribute is ommitted or set to the value @xml-unwrap="no".

  • A side effect of this feature is that two adjacent Metaschema fields cannot have the @xml-unwrap="yes", since in XML a parser will not be able to differentiate where one field ends and the other starts. The metaschema-check.sch needs to be updated to raise an error when this check fails.

  Given the following Metaschema:

  <assembly name="assembly">
    <field name="prose" as-type="markup-multiline"/>
  </assembly>
  

  When @xml-unwrap="no" or no @xml-unwrap attribute is provided the resulting XML content will look like this:

  <assembly>
    <prose>
      <p>text</p>
      <p>text</p>
    </prose>
  </assembly>
  

  When @xml-unwrap="yes" is provided the resulting XML content will look like this:

  <assembly>
    <p>text</p>
    <p>text</p>
  </assembly>
  

• Second, the following new datatypes need to be supported:

  Type	XML Type	JSON Schema Format	Description
  markup-line	"mixed content with HTML" (i)	string with "Markdown contents" (i)	A single paragraph of markup
  markup-multiline	"mixed content with HTML" (ii)	string with "Markdown contents" (ii)	Multiple paragraphs of markup
  date		date	An RFC3339 full-date
  date-time	dateTime (iii)	date-time	An RFC3339 date-time including timestamp
  email	string (iv)	email	An email as required by RFC 5322, section 3.4.1
  hostname	string (iv)	idn-hostname	An internationalized Internet host name as defined by RFC5890, section 2.3.2.3.
  ipv4	string (iv)	ipv4	IPv4 address, according to dotted-quad ABNF syntax as defined in RFC 2673, section 3.2
  ipv6	string (iv)	ipv6	IPv6 address, as defined in RFC 2373, section 2.2
  uri	anyURI (v)	uri	A universal resource identifier (URI), according to RFC3986.
  uri-reference	anyURI (v)	uri-reference	A URI Reference (either a URI or a relative-reference), according to RFC3986, section 4.1
  base64	base64Binary	string (vi)	Base64 encoded content

  1. For HTML, this would be OSCAL XML mapped HTML minus the outer <p> tags, or OSCAL JSON mapped Markdown as single line (no newlines). Support for lists and tables is also excluded from both the HTML and Markdown forms. This is consistent with the current HTML/Markdown support in OSCAL in a <field/>.

  2. For HTML, this would be OSCAL XML mapped HTML with the outer <p> tags, or OSCAL JSON mapped Markdown as a multi-line string. This is consistent with the current HTML/Markdown support in OSCAL in a <prose/>.

  3. For XML, a pattern needs to be included that makes the timezone required. This will follow the format of:

     time-hour       = 2DIGIT  ; 00-23
     time-minute     = 2DIGIT  ; 00-59
     time-numoffset  = ("+" / "-") time-hour ":" time-minute
     time-offset     = "Z" / time-numoffset
     

     Where Z is equivalent to +00:00 or -00:00

  4. Suitable patterns need to be defined for email, hostname, ipv4, and ipv6

  5. Suitable patterns need to be defined for a full URI, vs a fragment

  6. A value that is base64 encoded in JSON can be specified as base64 as follows:

  {
    "type": "string",
    "contentEncoding": "base64",
  }
  

  In JSON, a given format can be defined as follows:

  "properties": {
    "propName": {
      "description": "A typed property",
      "type": "string",
      "pattern": "^[A-Z]+$",
      "format": "date-time"
    }
  },
  

• Finally, the markup-line and markup-multiline data types can only be used on a define-field. In XML the flag content model are not compatible with HTML contents. Where HTML content is needed a field needs to be used. For cased where <field unwrapped="yes"/> use of the markup-line data type needs to be disallowed, as the content model of markup-line lacks a wrapping <p> in XML. These constraints need to be implemented in the metaschema-check.sch.

  A way of looking at this is for a field:

  Data Type	Wrapped	Unwrapped
  markup-line	allowed	disallowed
  markup-multiline	allowed	allowed

Dependencies:

This issue will address issue #323. This issue will also partially address issue #390, by providing for more data type validation.

Acceptance Criteria

• The Metaschema schema and schematron are updated to reflect all of the data type changes.
• XML and JSON documentation generated by the CI/CD workflow correctly represent the data types chosen for a given flag or field.
• All metaschema defintiions are updated to use the new features.
• The Metaschema definitions are updated in a way that result in no changes to associated content.
• Documentation on the use of Metaschema is updated to fully document these features.
• A single PR is approved that contains the above changes.

[wendellpiez]

Managing "simple-markdown" and "complex-markdown" (we need better names since the XML side won't have Markdown) will allow us to document the features supporting prose in the context not of a modeling construct (which perhaps confusingly is shared by all the models), but of datatypes. This bears on #357.

[david-waltermire]

Changed "markdown" to "markup".

[brian-ruf]

Per today's meeting, consider adding numeric types to this list.

I am aware of the need for integers and percentages within FedRAMP content. To be complete, we may want also consider decimal as well.

[wendellpiez]

Noting some specifications still incomplete -- tbd define lexical constraints (validable in XSD) for:

• email
• hostname
• ipv4
• ipv6
• uri
• uri-reference
• numeric types? percentages? as noted above

[wendellpiez]

Support for datatypes has now been added to both XSD and JSON Schema, but some alignment still remains.

For example, the JSON Schema date type does not appear to permit a time zone, so requiring the time zone (as we have it in the current spec) won't work in JSON.

The best way to flush out and reconcile these discrepancies is with unit testing for each of the defined types in both XSD and JSON validation. Let's make a new Issue for this? The unit tests could demonstrate coverage as well as show go/no-go testing for values expected to conform.

[wendellpiez]

Pending:

• Proceed with more unit testing as just described
• Make Issue for more unit testing including known trouble spots
• Make Issue for documentation of Metaschema datatyping semantics including the alignment mechanisms btw the XSD and JSON Schema capabilities

Otherwise this Issue is ready for review. The feature set continues to be tested along with other features, and appears to be working as intended.

[david-waltermire]

This has been completed in PR #464.