NIST Big Data Reference Architecture Interfaces
pandoc-codeblock-include is broken, a fixed version can be installed from
pip install --upgrade --force git+https://github.com/laszewsk/pandoc-codeblock-include
The working draft of the specification is in
The snapshot of the volume is available in various formats that should not be modified but can be used for reviewes at
It is easy to contribute to the document with the help of pull requests that allow eddits that will be addressed by the editor of this Volume (Gregor von Laszewski). Pull request can even be created via the GUI features of Github. You will need a github.com account and login into github to make use of this.
With Pandoc a version in MS Word can be created and formating changes can be applied. However all editorial non look and feel changes must not be conducted in the docx version but in the markdown version so we have a record of the changes in the github history. The look and feel, can be managed by a professional editor.
Once finalized, the Spec Draft will be hosted here in word format
and uploaded to NIST.
The document workflow is as follows:
- contributor gets github account
- contributor logs into github
- contributor eidts the md document and creates a pull request
- pull request is reviewed by Gregor von Laszewski and others while discussing it in the working group meetings if needed
- pull request is either rejected, accepted, or accepted with changes
- Once a number of changes are accepted a new version is created, however the most up to date version is always avalaible as markdown document
- The version number of the document is increased.
- Outdated document: https://github.com/cloudmesh/nist-volume8
- On Gregors Computer ~/Desktop/Documents/OneDrive/nist-volume-8
New Python function to Microservice generator
Please note that we have developed a convenient python function to microservices toolkit at
This tool just takes a function or class defined in python and generates the OpenAPI as well as the server code automatically. It can also be used as a starting point for customizing the API specification.
We need more things which we have not yet written down here
$ npm install -g spectacle-docs $ brew install ghc cabal-install $ cabal update $ cabal install pandoc-include-code $ cabal install pandoc-include
Get the code
git clone https://github.com/cloudmesh-community/nist.git cd nist/cloudmesh/openapi/service
We assume all terminals are cd'ed to the directory
Running the test program
make service make start # in one window make test # in the other window
Creating the spec in HTML
viewing the doc
make view # or open public/index.html in your browser
Publish the spec
Create a new service
cm-openapi service init SERVICE
./bin/cm-openapi service init health services/example -> services/health
A Service Example
This example demonstrates two points. First, it adds security support of http basic auth and apikey/secret auth. Second it provides the concept of the virtualdirectory as a demonstration specification to showcase an example usage of a service with authentication, as well as showcasing the idea of virtualdirectory.
The source for this example is located in
$ cd mongo $ make start
To run the demo:
Make sure the mongo container is running as required by the previous examples;
In the top level of nist git repo say
$ git clone git clone https://github.com/cloudmesh-community/nist.git $ cd nist $ make all
$ cd service; make all
This will run our new combined service
In another terminal window, go also the the nist git repo directory. The auth credential is configured in the file:
However we have not yet integrated this into the code. Thus be careful that when creating pull requests not to check in your version of secconf.yaml.
$ cd services/virtualdirectory/test $ ./test_virtualdirectory.sh
This will add an ftp endpoint as an entry of the virtualdirectory. (currently ftp is the only supported protocol for the virtualdirectory).
Then run these from a terminal to test the two service endpoints.
$ curl -u admin:secret http://localhost:8080/cloudmesh/virtualdirectories $ curl -u admin:secret http://localhost:8080/cloudmesh/virtualdirectory/UbuntuFTP
One line Convenient Demo Script
We assume that you have docker installed on your machine. This demo works at this time only on macOS
To execute the demo do the following
$ git clone git clone https://github.com/cloudmesh-community/nist.git $ cd nist $ make demo
In case you do not have an macOS machine we have shot a convenient video that explains how simple it is to run our services.
Please note that the service here is build from an open API specification that is includes in
The code for the service backend is located at
The password configuration file is located at
As this is only a demonstration, we have used the most simple security possible. We assume the server runs only on the local host. Please make sure to properly secure it.