docs(hdi): add/edit crate level docs for hdi and hdk (#1443)

* docs(hdk): reduce to only validate callback

* write create docs for holochain_deterministic_integrity

* flesh out doc comments

* add links to examples of integrity & ccordinator zomes

* fmt

* remove irrelevant phrase

* add dna manifest examples

* Update crates/holochain_deterministic_integrity/src/lib.rs

Co-authored-by: Paul d'Aoust <paul.daoust@holo.host>

* Update crates/holochain_deterministic_integrity/src/lib.rs

Co-authored-by: Paul d'Aoust <paul.daoust@holo.host>

* Update crates/holochain_deterministic_integrity/src/lib.rs

Co-authored-by: Paul d'Aoust <paul.daoust@holo.host>

* updates to hdi crate

* add integrity zome summary to hdk crate

* add icon for integrity zomes

* fmt

* fix hdk changelog

Co-authored-by: Paul d'Aoust <paul.daoust@holo.host>

crates/hdk/CHANGELOG.md

@@ -7,6 +7,7 @@ default_unreleasable: true
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## Unreleased
+- Docs: Add section on coordinator zomes and link to HDI crate.
 
 ## 0.0.140
 

crates/hdk/src/lib.rs

@@ -2,16 +2,20 @@
 //!
 //! Functions of a Holochain application (hApp) can be organized into reusable components. In Holochain terminology these components are called "zomes".
 //! One or multiple zomes are compiled into a WebAssembly (WASM) binary, referred to as a DNA. All of the DNAs of an application are bundled to a hApp.
+//! In short, that structure is __hApp -> DNA -> zome -> function__.
+//!
+//! hApps are required to produce and validate data deterministically. There's a data model and a domain logic part to each hApp. In Holochain, the
+//! data model is defined in integrity zomes and the domain logic is written in coordinator zomes. See Integrity zomes and Coordinator zomes further down and
+//! [Holochain Deterministic Integrity](holochain_deterministic_integrity) for more information.
 //!
-//! hApps are required to produce and validate data deterministically, which is stored in a content-addressable manner retrieved by hash value.
 //! Since hApps are run as a binary on the hosting system, they must be sandboxed to prevent execution of insecure commands.
 //! Instead of writing and maintaining a custom format and specification for these artifacts as well as a runtime environment to execute them,
 //! Holochain makes use of WASM as the format of its applications. WASM binaries meet the aforementioned requirements as per the
 //! [WebAssembly specification](https://webassembly.github.io/spec/core).
 //!
 //! hApps can be installed on a device that's running a so-called conductor, Holochain's runtime. Clients can then call each zome's functions via Remote Procedure Calls (RPC).
 //! Holochain employs websocket ports for these RPCs, served by the conductor. Calls are made either from a client on localhost or from other nodes on the network.
-//! The zome function to be executed must be specified in each call. Every zome function in turn defines the response it returns to the client as part of a zome's code.
+//! The zome function to be executed must be specified in each call. Every zome function defines the response it returns to the client.
 //! [More info on Holochain's architecture](https://developer.holochain.org/concepts/2_application_architecture)
 //!
 //! Low-level communication between the conductor and WASM binaries, like typing and serialization of data, is encapsulated by the HDK.
@@ -24,18 +28,35 @@
 //! The HDK is used in all the WASMs used to test Holochain itself.
 //! As they are used directly by tests in CI they are guaranteed to compile and work for at least the tests we define against them.
 //!
-//! At the time of writing there were about 40 example/test WASMs that can be browsed [on Github](https://github.com/holochain/holochain/tree/develop/crates/test_utils/wasm/wasm_workspace).
+//! At the time of writing there were about 40 example/test WASMs that can be browsed
+//! [on Github](https://github.com/holochain/holochain/tree/develop/crates/test_utils/wasm/wasm_workspace).
 //!
 //! Each example WASM is a minimal demonstration of specific HDK functionality, such as generating random data, creating entries or defining validation callbacks.
 //! Some of the examples are very contrived, none are intended as production grade hApp examples, but do highlight key functionality.
 //!
+//! # Integrity zomes 📐
+//!
+//! Integrity zomes describe a hApp's domain model by defining a set of entry and link types and providing a validation callback
+//! function that checks the integrity of any operations that manipulate data of those types.
+//!
+//! The wasm workspace contains examples of integrity zomes:
+//! <https://github.com/holochain/holochain/blob/develop/crates/test_utils/wasm/wasm_workspace/integrity_zome/src/lib.rs>
+//!
+//! # Coordinator zomes 🐜
+//!
+//! Coordinator zomes are the counterpart of integrity zomes in a DNA. They contain the domain logic of how data is read and written.
+//! Whereas data is defined and validated in integrity zomes, functions to manipulate data are implemented in coordinator zomes.
+//!
+//! An example coordinator zome can be found in the wasm workspace of the Holochain repository:
+//! <https://github.com/holochain/holochain/blob/develop/crates/test_utils/wasm/wasm_workspace/coordinator_zome/src/lib.rs>.
 //!
 //! # HDK structure 🧱
 //!
 //! HDK implements several key features:
 //!
 //! - Base HDKT trait for standardisation, mocking, unit testing support: [`hdk`] module
 //! - Capabilities and function level access control: [`capability`] module
+//! - [Holochain Deterministic Integrity (HDI)](holochain_deterministic_integrity)
 //! - Application data and entry definitions for the source chain and DHT: [`entry`] module and [`entry_defs`] callback
 //! - Referencing/linking entries on the DHT together into a graph structure: [`link`] module
 //! - Defining tree-like structures out of links and entries for discoverability and scalability: [`hash_path`] module
@@ -142,17 +163,11 @@
 //! - `fn validate_delete_link(delete_link_data: ValidateDeleteLinkData) -> ExternResult<ValidateLinkCallbackResult>`:
 //!   - Allows the guest to pass/fail/retry link deletion validation.
 //!   - Only the zome that deleted the link is called.
-//! - `fn validate_{{ create|update|delete }}_{{ agent|entry }}_{{ <entry_id> }}(validate_data: ValidateData) -> ExternResult<ValidateCallbackResult>`:
-//!   - Allows the guest to pass/fail/retry entry validation.
-//!   - <entry_id> is the entry id defined by entry defs e.g. "comment".
+//! - `fn validate(op: Op) -> ExternResult<ValidateCallbackResult>`:
+//!   - Allows the guest to pass/fail/retry any operation.
 //!   - Only the originating zome is called.
 //!   - Failure overrides retry.
-//! - `fn validation_package_{{ <entry_id> }}(entry_type: AppEntryType) -> ExternResult<ValidationPackageCallbackResult>`:
-//!   - Allows the guest to build a validation package for the given entry type.
-//!   - Can pass/retry/fail/not-implemented in reverse override order.
-//!   - <entry_id> is the entry id defined by entry defs e.g. "comment".
-//!   - Only the originating zome is called.
-//!
+//!   - See [`validate`](holochain_deterministic_integrity::prelude::validate) for more details.
 //!
 //! # HDK has layers 🧅
 //!
@@ -334,6 +349,7 @@ pub mod countersigning;
 /// For example, an agent could choose to 'block' another agent and ignore all their updates.
 pub mod entry;
 
+pub use holochain_deterministic_integrity;
 pub use holochain_deterministic_integrity::entry_defs;
 
 pub mod hash;

crates/holochain_deterministic_integrity/CHANGELOG.md

@@ -3,7 +3,9 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## Unreleased
+- Docs: crate level documentation for `holochain_deterministic_integrity`.
 
+### Added
 ## 0.0.12
 
 ## 0.0.11

crates/holochain_deterministic_integrity/src/lib.rs

@@ -1,4 +1,54 @@
-//! The Holochain Deterministic Integrity
+//! Holochain Deterministic Integrity (HDI) is Holochain's data model and integrity toolset for
+//! writing zomes.
+//!
+//! The logic of a Holochain DNA can be divided into two parts: integrity and coordination.
+//! Integrity is the part of the hApp that defines the data types and validates data
+//! manipulations. Coordination encompasses the domain logic and implements the functions
+//! that manipulate data.
+//!
+//! # Examples
+//!
+//! An example of an integrity zome with data definition and data validation can be found in the
+//! wasm workspace of the Holochain repository:
+//! <https://github.com/holochain/holochain/blob/develop/crates/test_utils/wasm/wasm_workspace/integrity_zome/src/lib.rs>.
+//!
+//! # Data definition
+//!
+//! The DNA's data model is defined in integrity zomes. They comprise all data type definitions
+//! as well as relationships between those types. Integrity zomes are purely definitions and do
+//! not contain functions to manipulate the data. Therefore a hApp's data model is encapsulated
+//! and completely independent of the domain logic, which is encoded in coordinator zomes.
+//!
+//!  The MVC (model, view, controller) design pattern can be used as an analogy. The
+//! application’s integrity zomes comprise its model layer — everything that defines the shape
+//! of the data. In practice, this means three things:
+//! - entry type definitions
+//! - link type definitions
+//! - a validation callback that constrains the kinds of data that can validly be called entries
+//! and links of those types (see also [`validate`](prelude::validate)).
+//!
+//! The coordination zomes comprise the application's controller layer — the code that actually
+//! writes and retrieves data, handles countersigning sessions and sends and receives messages
+//! between peers or between a cell and its UI. In other words, all the zome functions, `init`
+//! functions, remote signal receivers, and scheduler callbacks will all live in coordinator zomes.
+//!
+//! Advantages of this approach are:
+//! * The DNA hash is constant as long as the integrity zomes remain the same. The peer network of
+//! a DNA is tied to its hash. Changes to the DNA hash result in a new peer network. Changes to the
+//! domain logic enclosed in coordinator zomes, however, do not affect the DNA hash. Hence the DNAs
+//! and therefore hApps can be modified without creating a new peer network on every
+//! deployment.
+//! * Integrity zomes can be shared among DNAs. Any coordinator zome can import an integrity
+//! zome's data types and implement functions for data manipulation. This composability of
+//! integrity and coordinator zomes allows for a multitude of permutations with shared integrity
+//! zomes, i. e. a shared data model.
+//!
+//! # Data validation
+//!
+//! The second fundamental part of integrity zomes is data validation. For every [operation](holochain_integrity_types::Op)
+//! that can be performed on the data, a validation rule can be specified. Both data types and data
+//! values can be validated. All of these validation rules are written in a central callback
+//! which is called by the Holochain engine for each operation.
 
 pub use hdk_derive::hdk_entry_defs;
 pub use hdk_derive::hdk_entry_helper;

crates/holochain_integrity_types/src/lib.rs

@@ -5,7 +5,7 @@
 //! typically included as a dependency in Holochain Zomes, which are
 //! distributed as chunks of Wasm.
 //!
-//! This crate is also designed to be deterministic and more stable then
+//! This crate is also designed to be deterministic and more stable than
 //! the higher level crates.
 
 #![deny(missing_docs)]

crates/holochain_types/src/dna/dna_manifest/dna_manifest_v1.rs

@@ -3,8 +3,60 @@ use holo_hash::*;
 use holochain_zome_types::ZomeName;
 use serde_with::serde_as;
 
-/// The structure of data that goes in the DNA bundle manifest,
-/// i.e. "dna.yaml"
+/// The structure of data that goes in the DNA bundle manifest "dna.yaml".
+///
+/// Navigating through this structure reveals all configurable DNA properties.
+///
+/// # Examples
+///
+/// An example "dna.yaml" with 2 integrity and 2 coordinator zomes:
+///
+/// ```yaml
+/// manifest_version: "1"
+/// name: multi integrity dna
+/// integrity:
+///   uid: 00000000-0000-0000-0000-000000000000
+///   properties: ~
+///   origin_time: 2022-02-11T23:05:19.470323Z
+///   zomes:
+///     - name: zome1
+///       bundled: ../dna1/zomes/zome1.wasm
+///     - name: zome2
+///       bundled: ../dna2/zomes/zome1.wasm
+/// coordinator:
+///   zomes:
+///     - name: zome3
+///       bundled: ../dna1/zomes/zome2.wasm
+///       dependencies:
+///         - name: zome1
+///     - name: zome4
+///       bundled: ../dna2/zomes/zome2.wasm
+///       dependencies:
+///         - name: zome1
+///         - name: zome2
+/// ```
+///
+/// When there's only one integrity zome, it will automatically be a dependency
+/// of the coordinator zomes. It doesn't need to be specified explicitly.
+///
+/// ```yaml
+/// manifest_version: "1"
+/// name: single integrity dna
+/// integrity:
+///   uid: 00000000-0000-0000-0000-000000000000
+///   properties: ~
+///   origin_time: 2022-02-11T23:05:19.470323Z
+///   zomes:
+///     - name: zome1
+///       bundled: ../dna1/zomes/zome1.wasm
+/// coordinator:
+///   zomes:
+///     - name: zome3
+///       bundled: ../dna1/zomes/zome2.wasm
+///     - name: zome4
+///       bundled: ../dna2/zomes/zome2.wasm
+/// ```
+
 #[serde_as]
 #[derive(
     Serialize,
@@ -21,12 +73,12 @@ pub struct DnaManifestV1 {
     /// The friendly "name" of a Holochain DNA.
     pub name: String,
 
-    /// Only this affects the hash.
+    /// Only this affects the [`DnaHash`].
     pub integrity: IntegrityManifest,
 
     #[serde(default)]
-    /// Coordinator zomes to install with this dna.
-    /// Doesn't not affect the [`DnaHash`].
+    /// Coordinator zomes to install with this DNA.
+    /// Does not affect the [`DnaHash`].
     pub coordinator: CoordinatorManifest,
 }