Skip to content

Commit

Permalink
chore: comments, logs, & cleanup (#456)
Browse files Browse the repository at this point in the history
# Description

Remove many TODOs, add comments on *pub* (*pub* crate), and work through
logs.

- Includes a few other minor fixins and additions. 

## Type of change

- [X] Refactor (non-breaking change that updates existing functionality)

Closes #262.

---------

Signed-off-by: Zeeshan Lakhani <zeeshan.lakhani@gmail.com>
Co-authored-by: Brian Ginsburg <7957636+bgins@users.noreply.github.com>
  • Loading branch information
Zeeshan Lakhani and bgins authored Nov 29, 2023
1 parent 476dc11 commit 480311a
Show file tree
Hide file tree
Showing 71 changed files with 3,680 additions and 3,002 deletions.
21 changes: 21 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ lto = true
debug-assertions = false

[profile.release.package.homestar-runtime]
# Will slow-down compile, but improve perf on generated code.
codegen-units = 1
debug-assertions = false

Expand All @@ -87,6 +88,7 @@ debug-assertions = false
# Example: `cargo build -p homestar-functions-test --target wasm32-unknown-unknown --profile release-wasm-fn`
[profile.release-wasm-fn]
inherits = "release"
# Will slow-down compile, but improve perf on generated code.
codegen-units = 1
# Tell `rustc` to optimize for small code size.
opt-level = "z" # 'z' to optimize "aggressively" for size
Expand Down
41 changes: 38 additions & 3 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## Outline

- [Building and Running the Project](#building-and-running-the-project)
- [Testing the Project](#testing-the-project)
- [Running the Runtime on Docker](#running-the-runtime-on-docker)
- [Nix](#nix)
Expand All @@ -10,18 +11,51 @@
- [Recommended Development Flow](#recommended-development-flow)
- [Conventional Commits](#conventional-commits)

## Building and Running the Project

- Building `homestar`:

For the fastest compile-times and prettiest logs while developing `homestar`,
build with:

``` console
cargo build --no-default-features --features dev
```

This removes underlying `wasmtime` `zstd` compression while also turning on
ANSI color-coded logs. If you build with default features, `zstd` compression
and other `wasmtime` defaults will be included in the build.

- Running the `homestar` server/runtime:

``` console
cargo run --no-default-features --features dev -- start
```

- Running with [`tokio-console`][tokio-console] for diagnosis and debugging:

``` console
cargo run --no-default-features --features dev,console -- start
```

Then, in another window:

```console
tokio-console --retain-for 60sec
```

## Testing the Project

- Running the tests:

We recommend using [cargo nextest][cargo-nextest], which is installed by default
in our [Nix flake](#nix) or can be [installed separately][cargo-nextest-install].
We recommend using [cargo nextest][cargo-nextest], which is installed by default
in our [Nix flake](#nix) or can be [installed separately][cargo-nextest-install].

```console
cargo nextest run --all-features --no-capture
```

The above command translates to this using the default `cargo test`:
This command translates to the default `cargo test` command:

```console
cargo test --all-features -- --nocapture
Expand Down Expand Up @@ -139,4 +173,5 @@ a type of `fix`, `feat`, `docs`, `ci`, `refactor`, etc..., structured like so:
[nix]:https://nixos.org/download.html
[nix-flake]: https://nixos.wiki/wiki/Flakes
[pre-commit]: https://pre-commit.com/
[tokio-console]: https://github.com/tokio-rs/console
[wit-bindgen]: https://github.com/bytecodealliance/wit-bindgen
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ Our current list includes:

- [websocket relay](./examples/websocket-relay/README.md) - An example
(browser-based) application that connects to the `homestar-runtime` over a
websocket connection in order to run a couple static Wasm-based, image
WebSocket connection in order to run a couple static Wasm-based, image
processing workflows that chain inputs and outputs.

## Workspace
Expand Down Expand Up @@ -177,7 +177,6 @@ We would be happy to try to answer your question or try opening a new issue on G
- [IPVM - IPFS and WASM][ipfs-thing-ipvm] by Brooklyn Zelenka
- [Breaking Down the Interplanetary Virtual Machine][blog-1]
- [Ucan Invocation Spec][ucan-invocation]
- [Wasm/Wit Demo - Februrary 2023][demo-1] by Zeeshan Lakhani

## License

Expand Down
2 changes: 1 addition & 1 deletion examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ and the `homestar runtime`. Each example is set up as its own crate,
demonstrating the necessary dependencies and setup(s).

* [websocket relay](./websocket-relay) - An example (browser-based) application
that connects to the `homestar-runtime` over a websocket connection in order
that connects to the `homestar-runtime` over a WebSocket connection in order
to run a couple static Wasm-based, image processing workflows that chain
inputs and outputs using [inlined promises][pipelines].

Expand Down
Binary file modified examples/websocket-relay/example_test.wasm
Binary file not shown.
12 changes: 6 additions & 6 deletions examples/websocket-relay/relay-app/src/lib/workflow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ export const workflowOnePromised = WorkflowBuilder.workflow({
WorkflowBuilder.crop({
name: "crop",
resource:
"ipfs://bafybeiczefaiu7464ehupezpzulnti5jvcwnvdalqrdliugnnwcdz6ljia",
"ipfs://bafybeig6u35v6t3f4j3zgz2jvj4erd45fbkeolioaddu3lmu6uxm3ilb7a",
args: {
data: "{{ cid:bafybeiejevluvtoevgk66plh5t6xiy3ikyuuxg3vgofuvpeckb6eadresm }}",
x: 150,
Expand All @@ -268,15 +268,15 @@ export const workflowOnePromised = WorkflowBuilder.workflow({
WorkflowBuilder.rotate90({
name: "rotate90",
resource:
"ipfs://bafybeiczefaiu7464ehupezpzulnti5jvcwnvdalqrdliugnnwcdz6ljia",
"ipfs://bafybeig6u35v6t3f4j3zgz2jvj4erd45fbkeolioaddu3lmu6uxm3ilb7a",
args: {
data: "{{needs.crop.output}}",
},
}),
WorkflowBuilder.blur({
name: "blur",
resource:
"ipfs://bafybeiczefaiu7464ehupezpzulnti5jvcwnvdalqrdliugnnwcdz6ljia",
"ipfs://bafybeig6u35v6t3f4j3zgz2jvj4erd45fbkeolioaddu3lmu6uxm3ilb7a",
args: {
data: "{{needs.rotate90.output}}",
sigma: 20.2,
Expand All @@ -293,7 +293,7 @@ export const workflowTwoPromised = WorkflowBuilder.workflow({
WorkflowBuilder.crop({
name: "crop",
resource:
"ipfs://bafybeiczefaiu7464ehupezpzulnti5jvcwnvdalqrdliugnnwcdz6ljia",
"ipfs://bafybeig6u35v6t3f4j3zgz2jvj4erd45fbkeolioaddu3lmu6uxm3ilb7a",
args: {
data: "{{ cid:bafybeiejevluvtoevgk66plh5t6xiy3ikyuuxg3vgofuvpeckb6eadresm }}",
x: 150,
Expand All @@ -305,15 +305,15 @@ export const workflowTwoPromised = WorkflowBuilder.workflow({
WorkflowBuilder.rotate90({
name: "rotate90",
resource:
"ipfs://bafybeiczefaiu7464ehupezpzulnti5jvcwnvdalqrdliugnnwcdz6ljia",
"ipfs://bafybeig6u35v6t3f4j3zgz2jvj4erd45fbkeolioaddu3lmu6uxm3ilb7a",
args: {
data: "{{needs.crop.output}}",
},
}),
WorkflowBuilder.grayscale({
name: "grayscale",
resource:
"ipfs://bafybeiczefaiu7464ehupezpzulnti5jvcwnvdalqrdliugnnwcdz6ljia",
"ipfs://bafybeig6u35v6t3f4j3zgz2jvj4erd45fbkeolioaddu3lmu6uxm3ilb7a",
args: {
data: "{{needs.rotate90.output}}",
},
Expand Down
9 changes: 1 addition & 8 deletions examples/websocket-relay/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,11 @@ fn main() -> Result<()> {
// daemon. Typically, these would be started separately.
let ipfs_daemon = ipfs_setup();

info!(
subject = "settings",
category = "homestar_init",
"starting with settings: {:?}",
settings,
);
info!("starting with settings: {:?}", settings,);

let db = Db::setup_connection_pool(settings.node(), None).expect("to setup database pool");

info!(
subject = "database",
category = "homestar_init",
"starting with database: {}",
Db::url().expect("database url to be provided"),
);
Expand Down
12 changes: 6 additions & 6 deletions flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 1 addition & 2 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@
cargo-expand
cargo-nextest
cargo-sort
cargo-spellcheck
cargo-unused-features
cargo-udeps
cargo-watch
Expand Down Expand Up @@ -101,7 +100,7 @@

devRunServer = pkgs.writeScriptBin "cargo-run-dev" ''
#!${pkgs.stdenv.shell}
cargo run --no-default-features --features dev -- start -c homestar-runtime/config/settings.toml
cargo run --no-default-features --features dev -- start
'';

doc = pkgs.writeScriptBin "doc" ''
Expand Down
2 changes: 1 addition & 1 deletion homestar-core/src/test_utils/workflow.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ use std::collections::BTreeMap;
use url::Url;

const RAW: u64 = 0x55;
const WASM_CID: &str = "bafybeiczefaiu7464ehupezpzulnti5jvcwnvdalqrdliugnnwcdz6ljia";
const WASM_CID: &str = "bafybeig6u35v6t3f4j3zgz2jvj4erd45fbkeolioaddu3lmu6uxm3ilb7a";

type NonceBytes = Vec<u8>;

Expand Down
4 changes: 2 additions & 2 deletions homestar-core/src/workflow/instruction.rs
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,7 @@ mod test {
(
RESOURCE_KEY.into(),
Ipld::String(
"ipfs://bafybeiczefaiu7464ehupezpzulnti5jvcwnvdalqrdliugnnwcdz6ljia".into()
"ipfs://bafybeig6u35v6t3f4j3zgz2jvj4erd45fbkeolioaddu3lmu6uxm3ilb7a".into()
)
),
(OP_KEY.into(), Ipld::String("ipld/fun".to_string())),
Expand All @@ -357,7 +357,7 @@ mod test {
"func": "join-strings"
},
"nnc": "", "op": "wasm/run",
"rsc": "ipfs://bafybeiczefaiu7464ehupezpzulnti5jvcwnvdalqrdliugnnwcdz6ljia"});
"rsc": "ipfs://bafybeig6u35v6t3f4j3zgz2jvj4erd45fbkeolioaddu3lmu6uxm3ilb7a"});

let instruction = Instruction::<Unit>::try_from(ipld.clone()).unwrap();
let instr_cid = instruction.to_cid().unwrap();
Expand Down
2 changes: 1 addition & 1 deletion homestar-core/src/workflow/task.rs
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ mod test {
(
"rsc".into(),
Ipld::String(
"ipfs://bafybeiczefaiu7464ehupezpzulnti5jvcwnvdalqrdliugnnwcdz6ljia".into(),
"ipfs://bafybeig6u35v6t3f4j3zgz2jvj4erd45fbkeolioaddu3lmu6uxm3ilb7a".into(),
),
),
("op".into(), Ipld::String("ipld/fun".to_string())),
Expand Down
12 changes: 11 additions & 1 deletion homestar-functions/test/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,11 @@ wit_bindgen::generate!({
});

use base64::{engine::general_purpose, Engine};
use std::io::Cursor;
use std::{
collections::hash_map::DefaultHasher,
hash::{Hash, Hasher},
io::Cursor,
};

pub struct Component;

Expand Down Expand Up @@ -115,6 +119,12 @@ impl Guest for Component {
.unwrap();
Self::rotate90(decoded)
}

fn hash(s: String) -> Vec<u8> {
let mut hash = DefaultHasher::new();
s.hash(&mut hash);
hash.finish().to_be_bytes().to_vec()
}
}

#[cfg(test)]
Expand Down
1 change: 1 addition & 0 deletions homestar-functions/test/wit/host.wit
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@ world test {
export grayscale-base64: func(data: string) -> list<u8>
export rotate90: func(data: list<u8>) -> list<u8>
export rotate90-base64: func(data: string) -> list<u8>
export hash: func(data: string) -> list<u8>
}
3 changes: 2 additions & 1 deletion homestar-runtime/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ config = { version = "0.13", default-features = false, features = ["toml"] }
console-subscriber = { version = "0.2", default-features = false, features = [
"parking_lot",
], optional = true }
const_format = "0.2"
crossbeam = "0.8"
dagga = "0.2"
dashmap = "5.5"
Expand Down Expand Up @@ -218,7 +219,7 @@ wait-timeout = "0.2"

[features]
default = ["wasmtime-default", "ipfs", "monitoring", "websocket-notify"]
dev = ["ansi-logs", "console", "ipfs", "monitoring", "websocket-notify"]
dev = ["ansi-logs", "ipfs", "monitoring", "websocket-notify"]
ansi-logs = ["tracing-logfmt/ansi_logs"]
console = ["dep:console-subscriber"]
ipfs = ["dep:ipfs-api", "dep:ipfs-api-backend-hyper"]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
//! Wrapper around [crossbeam::channel] and [flume::bounded] to provide common
//! interfaces for sync/async bounded and non-tokio "oneshot" channels.
//! Wrapper around [crossbeam::channel] and [flume] to provide common
//! interfaces for sync/async (un)bounded and non-tokio "oneshot" channels.

use crossbeam::channel;

Expand Down
2 changes: 2 additions & 0 deletions homestar-runtime/src/cli/show.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
//! Styled, output response for console table.

use std::{
fmt,
io::{self, Write},
Expand Down
Loading

0 comments on commit 480311a

Please sign in to comment.