Machine Readable OSCAL Resources by Release #692
Comments
|
@brian-ruf, this requirements in this request, given the repository reorganization and new release artifacts process, is the GitHub Releases API insufficient and not meet these requirements. Can we close this for now? I can see potential future work in adding model metadata in back-matter to point to those for a given OSCAL version of a document with props, but we will need more feedback. As it stands, it looks like the MVP requirements you were looking for that were missing prior to that change have been met. |
|
To track down a potential requirements gap and outstanding questioned posed here, I will mark this as Further Analysis Needed and assign myself to communicate with Brian or other community members. If no follow-up occurs after a time, I will close the issue to be reopened later. |
|
@aj-stein-nist I don't have enough familiarity with the new spec to say for certain. The two big items from a requirements stand point are:
|
|
OK, @brian-easyd, so GitHub as an official REST and GraphQL API, the former is simpler to understand. If this makes sense what I summarize below, we can refine this issue to document this as the preferred approach in a follow-on ADR. Documentation is in the official GitHub API website. A fun fringe benefit of the previous ADR and communication of not auto-committing finish artifacts like schemas and converters into the git repo but only on tagged releases makes this a bit easier and cleaner because we no longer publish everything exclusively in just a zip file or tarball, you can get assets directly. So let's look at your questions. To start for the sake of discussion and because the API responses can be quite large (ironically because of the release notes embedded in them, that is the only significant size of the response payload, I will use an example response downloaded to a file (with the curl -L https://api.github.com/repos/usnistgov/OSCAL/releases | jq | tee github_api_response_usnistgov_oscal.json # use jq without arguments to lazily format the JSON with common pretty-printed style.I saved it as a file here for you if you have trouble with curl or other alternatives.
That is the first step here. $ cat ~/code/github_api_response_usnistgov_oscal.json | jq .[].tag_name
"v1.1.1"
"v1.1.0"
"v1.0.6"
"v1.0.5"
"v1.0.4"
"v1.0.3"
"v1.0.2"
"v1.0.1"
"v1.0.0-rc2"
"v1.0.0"
"v1.0.0-rc1"
"v1.0.0-milestone3"
"v1.0.0-milestone2"
"v1.0.0-milestone1"
# This is equivalent to:
# curl -L https://api.github.com/repos/usnistgov/OSCAL/releases | jqWe only have tagged releases of "valid versions" and that you get from the GitHub Releases API for free.
You can inspect each release and pull down the relevant files by API, not just its metadata I can look in the array of releases for each release, look at each array of assets in a release, and pull out the name of the artifacts. We can look at the first ( $ cat ~/code/github_api_response_usnistgov_oscal.json | jq .[0].assets[].name
"oscal-1.1.1.tar.bz2"
"oscal-1.1.1.zip"
"oscal_assessment-common_metaschema_RESOLVED.xml"
"oscal_assessment-plan_json-to-xml-converter.xsl"
"oscal_assessment-plan_metaschema_RESOLVED.xml"
"oscal_assessment-plan_schema.json"
"oscal_assessment-plan_schema.xsd"
"oscal_assessment-plan_xml-to-json-converter.xsl"
"oscal_assessment-results_json-to-xml-converter.xsl"
"oscal_assessment-results_metaschema_RESOLVED.xml"
"oscal_assessment-results_schema.json"
"oscal_assessment-results_schema.xsd"
"oscal_assessment-results_xml-to-json-converter.xsl"
"oscal_catalog_json-to-xml-converter.xsl"
"oscal_catalog_metaschema_RESOLVED.xml"
"oscal_catalog_schema.json"
"oscal_catalog_schema.xsd"
"oscal_catalog_xml-to-json-converter.xsl"
"oscal_complete_json-to-xml-converter.xsl"
"oscal_complete_metaschema_RESOLVED.xml"
"oscal_complete_schema.json"
"oscal_complete_schema.xsd"
"oscal_complete_xml-to-json-converter.xsl"
"oscal_component_json-to-xml-converter.xsl"
"oscal_component_metaschema_RESOLVED.xml"
"oscal_component_schema.json"
"oscal_component_schema.xsd"
"oscal_component_xml-to-json-converter.xsl"
"oscal_control-common_metaschema_RESOLVED.xml"
"oscal_implementation-common_metaschema_RESOLVED.xml"
"oscal_metadata_metaschema_RESOLVED.xml"
"oscal_poam_json-to-xml-converter.xsl"
"oscal_poam_metaschema_RESOLVED.xml"
"oscal_poam_schema.json"
"oscal_poam_schema.xsd"
"oscal_poam_xml-to-json-converter.xsl"
"oscal_profile_json-to-xml-converter.xsl"
"oscal_profile_metaschema_RESOLVED.xml"
"oscal_profile_schema.json"
"oscal_profile_schema.xsd"
"oscal_profile_xml-to-json-converter.xsl"
"oscal_ssp_json-to-xml-converter.xsl"
"oscal_ssp_metaschema_RESOLVED.xml"
"oscal_ssp_schema.json"
"oscal_ssp_schema.xsd"
"oscal_ssp_xml-to-json-converter.xsl"
# curl -L https://api.github.com/repos/usnistgov/OSCAL/releases | jq .[0].assets[].name$ cat ~/code/github_api_response_usnistgov_oscal.json | jq .[].assets[].name
"oscal-1.1.1.tar.bz2"
"oscal-1.1.1.zip"
"oscal_assessment-common_metaschema_RESOLVED.xml"
# Theere is going to be a lot truncated
# curl -L https://api.github.com/repos/usnistgov/OSCAL/releases | jq .[].assets[].nameYou can adapt the Our names are computer generated and consistent, so you can write code to filter based on what you want. We can discuss in more in follow-on comments. You can write more complex code to pick a particular version on partial
On that note, let me now how you feel about this thus far. I will declare up-front: I am strongly inclined to rely on this and not invest limited staff resources in productionalizing a novel API. I guess there could be a way of rigging up something through our static website and storing structured JSON information to emulate a dynamic REST API, but that is going to cause quality and resiliency concerns from my end (pages.nist.gov is not implemented or designed with that in mind). |
|
@ohsh6o I'm realizing I hadn't responded to this here. I believe we chatted verbally since you posted this. I agree that provided NIST continues to use the GitHub capabilities for published versions, this satisfies the requirement for tools to "know" all valid versions of OSCAL by querying this official source of truth from NIST. Unfortunately, with the more recent restructuring of the OSCAL repository, it is no longer possible to use this information to get directly to a specific resource. The use case I was targeting is as follows:
With this approach, tools can automatically handle new versions of OSCAL with no human intervention. At least in terms of schema validation and conversion capabilities are concerned. With the revised repository structure, the above is still possible, only the tool is forced to download and uncompress the entire ZIP file for all models and formats. Often with information the tool doesn't need at all (or at least doesn't need right now). The prior structure allowed a tool to get exactly what it needed. No more and no less. I'll leave it to the NIST OSCAL Team to decide whether to revert to the old structure to better support this use case. I have to acknowledge that requirement is still satisfied, even if under less efficient circumstances. |
|
@brian-comply0 - If you are referring to the |
|
@iMichaela you are correct. I missed this when I commented. With that, I agree the issue is satisfied. |
|
@brian-comply0 - Thank you. Closing the issue as satisfied. |
User Story:
As an OSCAL tool developer, it would be helpful to have a machine-readable file published by NIST, which enumerates each release of OSCAL, the models available in each release, the resources available for each model (syntax validators, format converters, and release version translators), and the links to those resources.
With a file like this, an OSCAL tool can always know of the latest OSCAL releases, and the current location of each resource for any release. This also also enable NIST to relocate old resources at a later date and make the revised links available to OSCAL tools seamlessly.
Goals:
Dependencies:
None
Acceptance Criteria
All OSCAL website and readme documentation affected by the changes in this issue have been updated. Changes to the OSCAL website can be made in the docs/content directory of your branch.
A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.
Depending on decisions made while working this issue, there may be an additional model for this. Perhaps as part of the "Governance" Layer described in Issue [EPIC] Adjudication or Governance Model #640.
The text was updated successfully, but these errors were encountered: