docs: Adds notes on Hermes packaging requirements (#71)

* chore(docs): stubs chapter for Hermes packaging documentation

* Add some notes for reference

* Add design notes

* Add some design notes

* feat: sketches for file and metadata in hdf5 terms

* chore: update docs

* chore: cleanup markdown lints

* chore: tidy up md lints

* docs: organize application.md and cleanup

* docs: add section and diagrams for metadata

* docs: update diagrams and cleanup

* docs: specify sections, add metadata information

* docs: add section for app data and signatures

move diagram around

* docs: reorganize overview.md subsections

* docs: add references to overview

* chore(docs): stubs chapter for Hermes packaging documentation

* Add some notes for reference

* Add design notes

* Add some design notes

* feat: sketches for file and metadata in hdf5 terms

* chore: update docs

* chore: cleanup markdown lints

* chore: tidy up md lints

* docs: organize application.md and cleanup

* docs: add section and diagrams for metadata

* docs: update diagrams and cleanup

* docs: specify sections, add metadata information

* docs: add section for app data and signatures

move diagram around

* docs: reorganize overview.md subsections

* docs: add references to overview

* docs: first draft of overview

Describes package file structure and key concepts with URLs

* docs: update diagrams and cleanup

* fix: typo

* docs(docs): Update docs to use latest cat-ci version

* docs(docs): WIP update of the packaging docs

* docs(docs): Define the hermes app metadata

* chore(vscode): Add d2 extensions to vscode recommendations

* docs(hermes): Add hermes application metadata schema initial revision

* docs(hermes): small improvements to the app metadata json schema.

* docs(docs): Include json schemas and examples in docs and fix up references to them.

* docs(hermes): Add Application Package Manifest File schema and example.

* docs(hermes): Add manifest details to the app packaging docs

* docs(hermes): Document how the http gateway works and how the files are served from the app package

* docs(hermes): Add WASM module json schemas and examples

* docs(hermes): Completed first draft of the Application packaging docs

* docs(hermes): Fix markdown and spelling checks

---------

Co-authored-by: Steven Johnson <stevenj@users.noreply.github.com>
Co-authored-by: Steven Johnson <sakurainds@gmail.com>

.config/dictionaries/project.dic

@@ -5,7 +5,9 @@ asyncio
 auditability
 backpressure
 bindgen
+blockdiag
 blockfetch
+bmac
 BROTLI
 cardano
 cbor
@@ -19,6 +21,7 @@ coti
 crontabs
 cryptoxide
 dbsync
+dcbor
 delegators
 dockerhub
 dotenv
@@ -35,6 +38,7 @@ fdatasync
 fdstat
 Filesize
 filestat
+filestorage
 filesystems
 fmtchk
 fmtfix
@@ -46,7 +50,10 @@ futimens
 genhtml
 GETFL
 gmtime
+happ
 hardano
+hasher
+hmod
 ideascale
 idents
 IFMT
@@ -81,6 +88,7 @@ openat
 opentelemetry
 outlen
 Outparam
+parameterises
 permissioned
 permissionless
 pg_isready
@@ -106,9 +114,11 @@ rustdoc
 rustdocflags
 rustflags
 saibatizoku
+sandboxed
 Sched
 seckey
 slotno
+smac
 stevenj
 symlinkat
 tacho
@@ -120,6 +130,7 @@ timelike
 toolsets
 Traceback
 txns
+typenum
 unlinkat
 utimensat
 vitss
@@ -132,7 +143,3 @@ webasm
 webassembly
 WORKDIR
 yoroi
-smac
-bmac
-typenum
-hasher

.markdownlint-cli2.jsonc

@@ -10,7 +10,8 @@
   "ignores": [
     ".config/dictionaries/**",
     "**/target/**",
-    "hermes/wasm/*/**"
+    "hermes/wasm/*/**",
+    "**/.dart_tool/**"
   ],
   // Set standard config options in `/.markdownlint.jsonc`
   "config": {

.vscode/extensions.json

@@ -23,5 +23,6 @@
         "bytecodealliance.wit-idl",
         "foxundermoon.shell-format",
         "dtsvet.vscode-wasm",
+        "terrastruct.d2",
     ]
 }

cspell.json

@@ -228,6 +228,7 @@
         "*.excalidraw",
         ".vscode/**",
         "**/.idea/**",
-        "hermes/crates/cardano-chain-follower/examples/snapshot_data"
+        "hermes/crates/cardano-chain-follower/examples/snapshot_data",
+        "**/.dart_tool/**"
     ]
 }

docs/Earthfile

@@ -6,25 +6,28 @@ VERSION 0.7
 # Copy all the source we need to build the docs
 src:
     # Common src setup
-    DO github.com/input-output-hk/catalyst-ci/earthly/docs:v2.7.0+SRC
+    DO github.com/input-output-hk/catalyst-ci/earthly/docs:v2.9.4+SRC
 
     # Now copy into that any artifacts we pull from the builds.
     COPY --dir ../+repo-docs/repo /docs/includes
     # copy Rust docs
     COPY ./../hermes+build/doc /docs/src/api/rust-docs
     # Copy the WASM Component model API Docs
     COPY --dir ./../wasm/wasi+build/wasi-hermes-docs /docs/src/api/wasi-hermes
+    # Copy the Hermes JSON Schemas and Examples
+    COPY --dir ./../hermes+json-schemas/schemas /docs/includes/
+
 
 # Build the docs here.
 docs:
     FROM +src
 
-    DO github.com/input-output-hk/catalyst-ci/earthly/docs:v2.7.0+BUILD
+    DO github.com/input-output-hk/catalyst-ci/earthly/docs:v2.9.4+BUILD
 
 # Make a locally runable container that can serve the docs.
 local:
     # Build a self contained service to show built docs locally.
-    DO github.com/input-output-hk/catalyst-ci/earthly/docs:v2.7.0+PACKAGE
+    DO github.com/input-output-hk/catalyst-ci/earthly/docs:v2.9.4+PACKAGE
 
     # Copy the static pages into the container
     COPY +docs/ /usr/share/nginx/html

docs/src/architecture/08_concepts/.pages

@@ -1 +1,4 @@
 title: Cross Cutting Concepts
+arrange:
+  - hermes_runtime_engine
+  - hermes_packaging_requirements

docs/src/architecture/08_concepts/hermes_packaging_requirements/.pages

@@ -0,0 +1,10 @@
+title: Hermes Application Packaging Requirements
+arrange:
+  - overview.md
+  - app_package.md
+  - app_metadata.md
+  - app_srv_www.md
+  - app_data.md
+  - wasm_modules.md
+  - app_signatures.md
+  - app_loading.md

docs/src/architecture/08_concepts/hermes_packaging_requirements/app_data.md

@@ -0,0 +1,9 @@
+# Hermes Application Data
+
+Applications can contain read-only shared embedded data.
+
+This data is shared between all WASM Component Modules within the application.
+
+The data can be used for any purpose defined by the Application and is stored in `/srv/share`.
+
+*Note: Even though the data is "shared" it is not available to any other application running inside a Hermes node.*

docs/src/architecture/08_concepts/hermes_packaging_requirements/app_loading.md

@@ -0,0 +1,92 @@
+# The Application Loading Process
+
+The high level steps of the app loading process are:
+
+![Diagram](images/app_loading_overview.d2)
+
+## Loading/Creating the App RW HDF5 Filesystem
+
+All Applications will need a RW Filesystem, even if its just for Hermes to store data about the application.
+The Process of loading the application is actually the process of creating or validating the RW Filesystem.
+
+Initially the Applications RW HDF5 Filesystem looks like this:
+
+<!-- markdownlint-disable max-one-sentence-per-line line-length no-inline-html -->
+| Name | Type | Description | Writable | Required |
+| --- | ----------- | ---- | -------- | --- |
+| `/`   | :octicons-file-directory-fill-16: | Root Directory | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: green;">:octicons-check-circle-fill-12:</span> |
+| `/tmp` | :octicons-file-directory-16: | Temporary Files stored in memory | <span style="color: green;">:octicons-check-circle-fill-12:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/etc` | :octicons-file-directory-fill-16: | Writable settings | <span style="color: green;">:octicons-check-circle-fill-12:</span> | <span style="color: green;">:octicons-check-circle-fill-12:</span> |
+<!-- markdownlint-enable max-one-sentence-per-line line-length no-inline-html -->
+
+The Application is at this stage un-configured.
+Once the user has configured the Application,
+the following files are created in the Application RW Storage and loading can continue.
+
+<!-- markdownlint-disable max-one-sentence-per-line line-length no-inline-html -->
+| Name | Type | Description | Writable | Required |
+| --- | ----------- | ---- | -------- | --- |
+| `/etc/settings.json` | :octicons-file-16: | Hermes Engine settings for this application. | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/etc/<module-name>/settings.json` | :octicons-file-16: | Module specific</br>Runtime Configurable Settings | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+<!-- markdownlint-enable max-one-sentence-per-line line-length no-inline-html -->
+
+If the Application has requested RW storage, then it is created (and sized accordingly) at:
+
+<!-- markdownlint-disable max-one-sentence-per-line line-length no-inline-html -->
+| Name | Type | Description | Writable | Required |
+| --- | ----------- | ---- | -------- | --- |
+| `/var/` | :octicons-file-directory-fill-16: |  Contains variable data files. (Persistent) | <span style="color: green;">:octicons-check-circle-fill-12:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+<!-- markdownlint-enable max-one-sentence-per-line line-length no-inline-html -->
+
+## Loading the Application itself
+
+At this stage, the RW Filesystem is now prepared.
+
+HDF5 allows us to create symbolic links between different HDF5 files.
+We use this capability to create RO symbolic links between the RW filesystem and the Application HDF5 package.
+
+During this process, any files which are defined in `/usr/lib` which would over-ride the
+contents of a module are linked inside the module, rather than the original module contents.
+
+This allows us the re-create the view the application sees of itself, without editing the actual application at-all.
+
+During this process symbolic RO links are created for the following files within the Application package:
+
+<!-- markdownlint-disable max-one-sentence-per-line line-length no-inline-html -->
+| Name | Type | Description | Writable | Required |
+| --- | ----------- | ---- | -------- | --- |
+| `/srv` | :octicons-file-directory-fill-16: | Data which is served by this system. | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/srv/www` | :octicons-file-directory-fill-16: | Files automatically served for this application on HTTP. | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/srv/share` | :octicons-file-directory-fill-16: | Data files which are not automatically served but can be shared by all Wasm Modules in the application. | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/usr` | :octicons-file-directory-fill-16: |  Shareable, read-only data. | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/usr/lib` | :octicons-file-directory-fill-16: |  Application over-rides for webasm library modules. | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/usr/lib/<module-name>` | :octicons-file-directory-fill-16: |  Application over-rides for named webasm library module. | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/usr/lib/<module-name>/config.json` | :octicons-file-16: |  Config to use for the module instead of its bundled config. | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/usr/lib/<module-name>/share` | :octicons-file-directory-fill-16: | Overrides for a modules shareable readonly data | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/lib` | :octicons-file-directory-fill-16: | Wasm Component Module Library Directory | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: green;">:octicons-check-circle-fill-12:</span> |
+| `/lib/<module-name>/metadata.json` | :octicons-file-16: | Modules Metadata | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: green;">:octicons-check-circle-fill-12:</span> |
+| `/lib/<module-name>/module.wasm` | :octicons-file-binary-16: | Actual WASM Module | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: green;">:octicons-check-circle-fill-12:</span> |
+| `/lib/<module-name>/config.schema.json` | :octicons-file-16: | Modules Fixed Configuration Schema | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/lib/<module-name>/config.json` | :octicons-file-16: | Modules Fixed Configuration (Must match the schema) | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/lib/<module-name>/settings.schema.json` | :octicons-file-16: | Modules User Settings Schema | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/lib/<module-name>/share` | :octicons-file-directory-fill-16: | Modules shareable readonly data | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+| `/lib/<module-name>/author.cose` | :octicons-file-badge-16: | Modules Author Signature | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: green;">:octicons-check-circle-fill-12:</span> |
+| `/metadata.json` | :octicons-file-16: | Applications Metadata | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: green;">:octicons-check-circle-fill-12:</span> |
+| `/author.cose` | :octicons-file-badge-16: | Application Author Signature | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: green;">:octicons-check-circle-fill-12:</span> |
+| `/publisher.cose` | :octicons-file-badge-16: | Application Publisher Signature | <span style="color: orange;">:octicons-circle-16:</span> | <span style="color: orange;">:octicons-circle-16:</span> |
+<!-- markdownlint-enable max-one-sentence-per-line line-length no-inline-html -->
+
+## Mounting the `/srv/www` filesystem in the HTTP gateway
+
+At this stage the Applications `/srv/www` from the RW Filesystem (as linked to the application package itself)
+is registered with the HTTP gateway inside the hermes node, and it can begin serving those files.
+
+## Loading and Initializing the WASM Modules
+
+The final step is to iterate all the WASM Modules in the RW Filesystem (as linked to the App) and load them
+in canonical order into the WASM Executor.
+
+This needs to be the final step because when a WASM Module is first loaded, it may be initialized.
+That process may require access to any of the data or configuration stored within the Application.
+It is only safe to access that data when the entire RW Filesystem and its cross linked
+application resources are ready to be accessed.

docs/src/architecture/08_concepts/hermes_packaging_requirements/app_metadata.md

@@ -0,0 +1,71 @@
+# Application Metadata
+
+Metadata for Hermes Applications is specified as a `metadata.json` file located in the root group
+in the HDF5 File.
+It is a `json` document and must conform to the Application metadata json schema to be valid.
+
+## Application Metadata Contents
+
+The Application Metadata is formally defined by [`hermes_app_metadata.schema.json`](#metadata-schema).
+
+All Application metadata files MUST conform to that schema, or the Application will be considered invalid.
+
+### Diagram: Hermes Application Metadata
+
+![Diagram](images/application_metadata.d2)
+
+*Note: Diagram is illustrative of metadata contents only.
+See [`hermes_app_metadata.schema.json`](#metadata-schema) for the formal definition.*
+
+### Application identifying Data
+
+* Application Name : Single Line name of the application.
+* Icon
+* Version
+* Description - Short Description of the Application
+* About - Long form Description of the Application
+* Copyright - Copyright Notices
+* License - SPDX Strings and/or a Link to a License held within the app image.
+
+### Application Developer/Author
+
+* Developers Name
+* Optional contact information
+* Optional payment address
+
+### Resources
+
+Each application will need a minimum set of resources from Hermes.
+The Metadata also lists the minimum viable resources required for the application.
+It should also list what are the resources the Application would like to have available.
+It can optionally list the maximum allocatable resources which could enable enhanced features or other functions in the application.
+
+If a resource minimum is set as 0, then it means the resource can be denied by the user but the application can still operate.
+
+### Permissions
+
+When Hermes has permissioned resources, the metadata will list the permissions being requested by the application.
+
+## Configuration
+
+Other than resourcing and permissions, the `metadata.json` file does not contain the configuration of the application.
+Application configuration is defined by the WASM Component modules.
+
+## Metadata Schema
+
+<!-- markdownlint-disable max-one-sentence-per-line -->
+
+??? note "Schema: `hermes_app_metadata.schema.json`"
+
+    ```json
+    {{ include_file('includes/schemas/hermes_app_metadata.schema.json', indent=4) }}
+    ```
+
+## Metadata Example
+
+??? note "Example: `hermes_app_metadata.json`"
+
+    ```json
+    {{ include_file('includes/schemas/example/hermes_app_metadata.json', indent=4) }}
+    ```
+<!-- markdownlint-enable max-one-sentence-per-line -->