Introduce the concept of a WASM manifest; use cacache for local storage #39

ceejbot · 2023-02-02T00:56:23Z

This PR reworks the agent API and the cli to support a new concept of a "manifest". This is a toml specification of a WASM job type. It includes a name, a namespace, a path to a binary, and some human-readable descriptions.

Here's an example manifest:

name = "loudify"
namespace = "sh.serval"
binary = "../../wasm-samples/build/loudify.wasm"
version = "1"
description = "SHOUT SHOUT LET IT ALL OUT"
required_extensions = ["birdfeeder"]

Once a WASM manifest + executable has been stored in the system, you can execute that wasm using the
name in the manifest: pounce run sh.serval.loudify inputfile.txt

The local blob storage has been reimplemented to use the cacache
crate. This is a content-adddressable store that also allows lookup of items by key. In our case
we store manifests with keys like name.manifest.toml and wasm executables with names like
name.wasm. Execution of jobs is now more indirect than it was, in that we look up the manifest by
name, and from that look up where the binary is by version. This indirection allows us to support
versioning more fully in the long term.

This PR also significantly reworks the agent API and changes the CLI to support this new API. The agent endpoints now look like this:

GET /monitor/ping: cheap health check
POST /v1/jobs/:name/run: body is interpreted as job input
GET /v1/storage/manifests: fetch a list of manifests
POST /v1/storage/manifests: store a new manifest
GET /v1/storage/manifests/:name: get a manifest by name
HEAD /v1/storage/manifests/:name: 404s if manifest not found
PUT /v1/storage/manifests/:name/executable/:version: store a new wasm executable associated with the named task manifest
GET /v1/storage/manifests/:name/executable/:version: get a wasm executable by manifest name & version
GET /v1/monitor/history: very cheap run history; to be deleted
GET /v1/jobs: list all running jobs; not implemented

Also, take a few minor updates to crates; revert change to the workspace syntax because it is making VSCode angry.

@shinypb

- Use the newly-published tokio version of cacache. - Rework the blob storage module to use cacache. - Extend its api a bunch because cacache lets us do more. - Rework endpoints in the mesh agent to support the new slightly-larger API. Also, take a few minor updates to crates; revert change to the workspace syntax because it is making VSCode angry. Dry-coded a new agent API on top of a new blob API There is a new concept called a "Manifest" that has some of the fields specified in @shinypb's job manifest spec. (I skipped a bunch just so I could get started with the implementation.) The blob storage now knows about wasm executables and manifests and can build type-aware keys for each kind of data. The agent endpoints now look like this: `GET /monitor/ping` cheap health check `POST /v1/jobs/:name/run` body is interpreted as job input `GET /v1/storage/manifests` fetch a list of manifests `POST /v1/storage/manifests` store a new manifest `GET /v1/storage/manifests/:name` get a manifest by name `HEAD /v1/storage/manifests/:name` 404s if manifest not found `PUT /v1/storage/manifests/:name/executable/:version` store a new wasm executable associated with the named task manifest `GET /v1/storage/manifests/:name/executable/:version` get a wasm executable by manifest name & version `GET /v1/monitor/history` very cheap run history; to be deleted `GET /v1/jobs` list all running jobs; not implemented The underlying blob store API supports this external API. Up next: making the CLI use the new API. IT'S ALIVE The CLI can now store wasm jobs in the mesh (okay, in one node) and then run those jobs only passing up input data.

This needs more than the seat-of-the-pants design I've given it just now, but this pass removes redundant information. Also, we now standardize on using the fully-qualified name everywhere. While I was there, introduced some `prettytable` to the CLI to start formatting some displays nicely. This is throwaway work in the long term, but it makes things a little nicer now.

oh oh oh oh

shinypb · 2023-02-02T22:54:05Z

agent/src/api/v1/jobs.rs

+) -> impl IntoResponse {
+    let mut lock = state.lock().await;
+    let storage = lock.storage.as_ref().unwrap();
+    let Ok(manifest) = storage.manifest(&name).await else {


Does this mean that only nodes that have storage locally available can run jobs?

At the moment, yes. I think the immediate next followup work is to make a node ask if if has the appropriate binary, and if not, find it.

Makes sense. Thoughts on swapping BlobStore out for an abstract trait that has CacacheBlobStore as one implementation and RemoteBlobStore (something that proxies to a serval_storage instance) as the other? If that sounds reasonable to me I am happy to start chipping away at it.

utils/src/errors.rs

shinypb · 2023-02-02T23:19:38Z

utils/src/structs.rs

+    executable: Vec<u8>,
+    /// Input data
+    input: Vec<u8>,
+    // TODO: might have version chosen to run here, plus run options; might also store the input


Right, something like an Option<u16> to reference a specific version number (rather "latest") would make sense.

shinypb · 2023-02-02T23:24:54Z

cli/src/main.rs

@@ -86,55 +87,56 @@ fn build_url(path: String, version: Option<&str>) -> String {
 }

 fn upload_manifest(manifest_path: PathBuf) -> Result<()> {
-    println!("Reading manifest at {}", manifest_path.display());
+    println!("Reading manifest: {}", manifest_path.display());


Is this is the equivalent of

println!("Reading manifest: {manifest_path:?}");

?

Not a debugging print, but converting a pointer to a pathbuf to its display version. I think it's cleaner; haven't looked at what the output is with the :? formatter.

shinypb · 2023-02-02T23:26:50Z

utils/src/structs.rs

-    pub fn name(&self) -> String {
-        self.fq_name.clone()
+    pub fn fq_name(&self) -> String {
+        let name = self.name.to_ascii_lowercase().replace('-', "_");


Is the - support just because of our wasm-samples naming convention?

This should get a TODO next to it, because IMO it needs to be cleaned more definitively to restrict it to allowed characters.

cli/src/main.rs

shinypb · 2023-02-02T23:47:26Z

cli/src/main.rs

+    table.add_row(row!["WASM task name:", manifest.fq_name()]);
+    table.add_row(row!["Version:", manifest.version()]);
+
+    let url = build_url("storage/manifests".to_string(), Some("1"));


We pass Some("1") around often enough that a build_url_v1 helper function probably wouldn't be a bad idea at some point.

Agreed! It annoyed me.

cli/src/main.rs

shinypb · 2023-02-02T23:51:27Z

This looks good to me, modulo the rustfmt error giving you grief. 😅 Calling out that landing this means that STORAGE_ROLE=never nodes can no longer run jobs, which (temporarily) means that we can't have nodes on more than one machine working together due to our lack of distributed storage.

As is right and proper.

ceejbot added 2 commits February 1, 2023 16:53

ceejbot marked this pull request as ready for review February 2, 2023 22:37

run run run run run run away

68ba96c

oh oh oh oh

shinypb reviewed Feb 2, 2023

View reviewed changes

ceejbot changed the title ~~Squash of eight commits; new writeup needed~~ Introduce the concept of a WASM manifest; use cacache for local storage Feb 2, 2023

shinypb reviewed Feb 2, 2023

View reviewed changes

utils/src/errors.rs Show resolved Hide resolved

shinypb reviewed Feb 2, 2023

View reviewed changes

cli/src/main.rs Outdated Show resolved Hide resolved

shinypb reviewed Feb 2, 2023

View reviewed changes

cli/src/main.rs Outdated Show resolved Hide resolved

ceejbot added 3 commits February 2, 2023 16:59

Clean up error -> status mapping.

45aa4f3

WASM paths are now relative to the manifest path.

56fdfbf

As is right and proper.

better cargo run hint!

6071f3a

ceejbot merged commit 8e4624d into main Feb 3, 2023

ceejbot deleted the ceej/cacache-in branch February 3, 2023 01:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce the concept of a WASM manifest; use cacache for local storage #39

Introduce the concept of a WASM manifest; use cacache for local storage #39

ceejbot commented Feb 2, 2023 •

edited

shinypb Feb 2, 2023

ceejbot Feb 2, 2023

shinypb Feb 2, 2023

shinypb Feb 2, 2023

shinypb Feb 2, 2023

ceejbot Feb 2, 2023

shinypb Feb 2, 2023

ceejbot Feb 2, 2023

shinypb Feb 2, 2023

ceejbot Feb 3, 2023

shinypb commented Feb 2, 2023

Introduce the concept of a WASM manifest; use cacache for local storage #39

Introduce the concept of a WASM manifest; use cacache for local storage #39

Conversation

ceejbot commented Feb 2, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shinypb commented Feb 2, 2023

ceejbot commented Feb 2, 2023 •

edited