Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure that all field and flag values require a non-empty value #805

Closed
david-waltermire opened this issue Dec 13, 2020 · 5 comments · Fixed by #936
Closed

Ensure that all field and flag values require a non-empty value #805

david-waltermire opened this issue Dec 13, 2020 · 5 comments · Fixed by #936
Assignees
@wendellpiez
Labels
bug
Projects
Sprint 39
Milestone
OSCAL 1.0.0 (Full...

Comments

@david-waltermire
Copy link
Contributor

Describe the bug

For some data types, the XML and JSON schema generated for the OSCAL models allow an empty value to specified. This occurs on XML elements and attributes and JSON properties that are derived from Metaschema fields and flags.

Who is the bug affecting?

Users of the XML and JSON schemas for the OSCAL project.

What is affected by this bug?

All generated XML and JSON schemas for the OSCAL project.

Expected behavior (i.e. solution)

Empty text should cause a schema validation error. To remove this error, the content creator should remove the outer attribute, element, or property.

If this attribute, element, or property is required then an empty value should not occur, since this doesn't satisfy the outer requirement.

Other Comments

{Add any other context about the problem here.}
@david-waltermire david-waltermire added this to To do in Sprint 39 via automation Feb 11, 2021
@wendellpiez
Copy link
Contributor

Relatively lightweight ways to deal with this in the interim:

  • Use matches constraints in a Metatron implementation for this
  • Introduce a new 'charstring' datatype (for example) and call it where wanted
  • Refine markup-line and markup-line with extra constraints (over what is currently enforced).

NB there are other validations one might want such as matches(.,'\w'): it is hard to know where to draw the line here.

@wendellpiez
Copy link
Contributor

Attached is an XSLT (zipped for handling) that produces a breakout listing of all fields and flags, by assigned type, or "[string]" (in brackets) if a string type by default. It polls all the metaschema documents in a collection (line 9). To run, place the XSLT next to a pile of metaschema documents and apply it to itself. The XSLT could be enhanced to give more info (now it only produces plain text listing definitions in their modules).

metaschema-poll.zip

We can also use paths to find fields and flags by type -- //define-field[@as-type='uri'] //define-field[empty(@as-type)] etc.

@wendellpiez
Copy link
Contributor

To save a step: here is the result of running against the metaschemas in OSCAL/src/metaschema.

metaschema-field-types.txt

@david-waltermire david-waltermire added this to To do in Sprint 40 via automation Mar 11, 2021
@david-waltermire david-waltermire moved this from To do to In progress in Sprint 40 Mar 11, 2021
@david-waltermire david-waltermire mentioned this issue Mar 15, 2021
Closed
@david-waltermire david-waltermire removed this from In progress in Sprint 40 Apr 1, 2021
@wendellpiez
Copy link
Contributor

Do this with #911. Remove DTD-based datatype definitions and provide for defense against empty strings.

wendellpiez added a commit to wendellpiez/metaschema that referenced this issue May 21, 2021
@wendellpiez
Addressing datatype validation issues: whitespace collapsing; non-emp…
ea371c6 
…ty values; ncname-workalike in JSON Schema - see usnistgov/OSCAL#911  usnistgov/OSCAL#805 also usnistgov#33 usnistgov#67 usnistgov#68
wendellpiez added a commit to wendellpiez/OSCAL that referenced this issue May 21, 2021
@wendellpiez
Changes to OSCAL metaschemas in view of enhancements addressing usnis…
b8b5d2f 
…tgov#805, usnistgov#911, #67, #68
@wendellpiez wendellpiez mentioned this issue May 21, 2021
Merged
8 tasks
wendellpiez added a commit to wendellpiez/metaschema that referenced this issue May 21, 2021
@wendellpiez
Addressing datatype validation issues: whitespace collapsing; non-emp…
b92de17 
…ty values; ncname-workalike in JSON Schema - see usnistgov/OSCAL#911  usnistgov/OSCAL#805 also usnistgov#33 usnistgov#67 usnistgov#68
david-waltermire pushed a commit to usnistgov/metaschema that referenced this issue May 21, 2021
@wendellpiez
Further improvements to schema generation (#142)
1b405bc 
* Addressing datatype validation issues: whitespace collapsing; non-empty values; ncname-workalike in JSON Schema - see usnistgov/OSCAL#911  usnistgov/OSCAL#805 also #33 #67 #68
* Improvements to XSD production; fully aligning 'token' datatype across XSD and JSON Schema implementations.
@david-waltermire david-waltermire linked a pull request May 21, 2021 that will close this issue
Metaschema enhancements to string-based datatypes #936
Merged
8 tasks
david-waltermire pushed a commit that referenced this issue May 21, 2021
@wendellpiez
Metaschema enhancements to string-based datatypes (#936)
82ec272 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing #805, #911, #67, #868.
david-waltermire pushed a commit that referenced this issue May 27, 2021
@wendellpiez
Metaschema updates for v1.0.0 (#942)
d88194c 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing #805, #911, #67, #68
david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue May 27, 2021
@wendellpiez @david-waltermire
Metaschema enhancements to string-based datatypes (usnistgov#936)
b736023 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing usnistgov#805, usnistgov#911, usnistgov#67, usnistgov#868.
david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue May 27, 2021
@wendellpiez @david-waltermire
Metaschema updates for v1.0.0 (usnistgov#942)
93a0ec2 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing usnistgov#805, usnistgov#911, usnistgov#67, usnistgov#68
david-waltermire pushed a commit that referenced this issue May 27, 2021
@wendellpiez @david-waltermire
Metaschema enhancements to string-based datatypes (#936)
99f6ed9 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing #805, #911, #67, #868.
david-waltermire pushed a commit that referenced this issue May 27, 2021
@wendellpiez @david-waltermire
Metaschema updates for v1.0.0 (#942)
3643196 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing #805, #911, #67, #68
david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue May 27, 2021
@wendellpiez @david-waltermire
Metaschema updates for v1.0.0 (usnistgov#942)
afebea5 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing usnistgov#805, usnistgov#911, usnistgov#67, usnistgov#68
david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue May 27, 2021
@wendellpiez @david-waltermire
Metaschema updates for v1.0.0 (usnistgov#942)
636ab72 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing usnistgov#805, usnistgov#911, usnistgov#67, usnistgov#68
david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue May 27, 2021
@wendellpiez @david-waltermire
Metaschema updates for v1.0.0 (usnistgov#942)
d912972 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing usnistgov#805, usnistgov#911, usnistgov#67, usnistgov#68
david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue May 27, 2021
@wendellpiez @david-waltermire
Metaschema updates for v1.0.0 (usnistgov#942)
86ee953 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing usnistgov#805, usnistgov#911, usnistgov#67, usnistgov#68
david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue May 27, 2021
@wendellpiez @david-waltermire
Metaschema updates for v1.0.0 (usnistgov#942)
33783f9 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing usnistgov#805, usnistgov#911, usnistgov#67, usnistgov#68
david-waltermire added a commit to usnistgov/metaschema that referenced this issue Jun 6, 2021
@david-waltermire @wendellpiez
Feature pipeline refactor 2021 05 (#152)
b911752 
* Rework of docs focusing on JSON docs and model pipeline
* Improvements to composition toolchain
* Fixed a few small bugs in the metaschema-check. Improved performance of the compose pruning using an accumulator.
* Moved edge-case samples into testing directory
* Made shadowing warning a warning
* Initial commit of an Oxygen Metaschema framework.
* Creation of new compose schematron unit tests.
* Cross-linking XML and JSON syntax pages and other improvements to links
* Now building XML and JSON indexes to reference pages, with links to steps
* Reconfigured docs pipeline (XSLT entry points); adding new files including pipeline steps
* Migrating schema generation tools to new/improved composition pipeline
* Addressing usnistgov/OSCAL#902 thanks for finding this bug
* Enhancements to JSON Schema definition (with better performance too)
* Adding support for json-base-uri as a metaschema property
* Updated JSON schema $id; factoring out common docs XSLT
* Fixing IDs in JSON schema per issue usnistgov/OSCAL#933.
* Addressing datatype validation issues: whitespace collapsing; non-empty values; ncname-workalike in JSON Schema - see usnistgov/OSCAL#911  usnistgov/OSCAL#805 also #33 #67 #68
* Improvements to XSD production; fully aligning 'token' datatype across XSD and JSON Schema implementations.
* Updating bidirectional XML/JSON converter generators (#143)
* Committing a version that handles test data correctly (so far) from rebuilt metaschema composition addressing #51 #53 #76
* Now displaying constraints in documentation at point of definition;
* Docs generation revamp Reworked reference and other pages to sketch - #128 and others

Co-authored-by: Wendell Piez <wendell.piez@nist.gov>
david-waltermire pushed a commit that referenced this issue Jun 7, 2021
@wendellpiez @david-waltermire
Metaschema enhancements to string-based datatypes (#936)
c98b2a8 
* Adjusted metaschemas: new 'version'; json-base-uri
* Added 'complete' metaschema
* Changes to OSCAL metaschemas in view of enhancements addressing #805, #911, #67, #868.
@david-waltermire
Copy link
Contributor Author

This was addressed by PR #936 (merged with PR #948), which refactored data typing to require non-empty values for appropriate data types..

Open Security Controls Assessment Language automation moved this from To do to Done Jun 7, 2021
@david-waltermire david-waltermire mentioned this issue Jun 7, 2021
Closed
6 tasks
nikitawootten-nist pushed a commit to nikitawootten-nist/metaschema-xslt that referenced this issue Jul 21, 2023
@david-waltermire @wendellpiez
Feature pipeline refactor 2021 05 (#152)
39f2702 
* Rework of docs focusing on JSON docs and model pipeline
* Improvements to composition toolchain
* Fixed a few small bugs in the metaschema-check. Improved performance of the compose pruning using an accumulator.
* Moved edge-case samples into testing directory
* Made shadowing warning a warning
* Initial commit of an Oxygen Metaschema framework.
* Creation of new compose schematron unit tests.
* Cross-linking XML and JSON syntax pages and other improvements to links
* Now building XML and JSON indexes to reference pages, with links to steps
* Reconfigured docs pipeline (XSLT entry points); adding new files including pipeline steps
* Migrating schema generation tools to new/improved composition pipeline
* Addressing usnistgov/OSCAL#902 thanks for finding this bug
* Enhancements to JSON Schema definition (with better performance too)
* Adding support for json-base-uri as a metaschema property
* Updated JSON schema $id; factoring out common docs XSLT
* Fixing IDs in JSON schema per issue usnistgov/OSCAL#933.
* Addressing datatype validation issues: whitespace collapsing; non-empty values; ncname-workalike in JSON Schema - see usnistgov/OSCAL#911  usnistgov/OSCAL#805 also usnistgov#33 usnistgov#67 usnistgov#68
* Improvements to XSD production; fully aligning 'token' datatype across XSD and JSON Schema implementations.
* Updating bidirectional XML/JSON converter generators (#143)
* Committing a version that handles test data correctly (so far) from rebuilt metaschema composition addressing usnistgov#51 usnistgov#53 usnistgov#76
* Now displaying constraints in documentation at point of definition;
* Docs generation revamp Reworked reference and other pages to sketch - #128 and others

Co-authored-by: Wendell Piez <wendell.piez@nist.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wendellpiez wendellpiez

Labels
bug
Projects
No open projects
Sprint 39
  
To do
Milestone
OSCAL 1.0.0 (Full Release)
Development

Successfully merging a pull request may close this issue.

Metaschema enhancements to string-based datatypes wendellpiez/OSCAL
2 participants
@wendellpiez @david-waltermire