Skip to content
Permalink
Browse files

new core architecture

  • Loading branch information
syntacticsugarglider committed Nov 30, 2019
1 parent 70dd37d commit 1d5e918d56a15cda472d87ea335a14eb266bbedf
@@ -11,10 +11,10 @@ The basic building block of Vessels is the *vessel*. A vessel is a WebAssembly b

The communication between vessels, locally and across different machines, is based on a system of *Kind*s. A Kind is essentially an interface for types with serializable representations, but with bidirectional communication permitted (though not necessarily used) for all Kinds. This communication happens over a dedicated *Channel*, preventing crosstalk, and the architecture is generically abstracted over the mechanism for delegating channels, the mechanism of serialization and deserialization (JSON / CBOR / etc), and the transport used (network / IPC / vessel-host boundary shared memory / etc). The two operations on Kind are construct and deconstruct, both of which operate on a channel, and both of which are able to read or write on the channel. Channels can be *fork*ed to create a subchannel, which is used to allow Kinds to send sub-Kinds (for example, having a list of Kinds also itself be a Kind).
A vessel itself must export a single Kind. This is called *satisfying* the Kind exported. A vessel can be provided across boundaries in a manner similar to “passing by reference” in which calls to the vessel are sent over the channel back to the deconstructor, which calls the original concrete instance and sends the return values back. Capabilities received from the orchestrator are generally Kinds provided in this manner, such as to provide access to abstracted hardware features or vessel orchestration. For example, a universally available capability is [`Executor`](https://noocene.github.io/vessels/vessels/core/executor/trait.Executor.html), an erased trait object that provides the ability to spawn futures. Requesting the executor will always return a valid execution context, but that concrete execution method may be different depending on the vessel in question or the target feature support for, for example, WASM atomics. A vessel can also be sent in a manner similar to “passing by move” where the WASM binary itself is sent over the wire and instantiated transparently by the recipient (assuming that recipient possesses the requisite capabilities to request the orchestrator to instantiate this new vessel). For example, a request for a video transcoder capability could either be replied to with a “passed by reference” vessel providing access to NVENC or similar in hardware locally, or with a “passed by value” vessel that provides software transcoding and is instantiated by the recipient. All use of vessels is transparent to the developer and requires no boilerplate additional to that which would be required for equivalent concrete types. In fact, vessels are used directly as the concrete Kind they export, appearing transparently equivalent to that underlying type.
A vessel itself must export a single Kind. This is called *satisfying* the Kind exported. A vessel can be provided across boundaries in a manner similar to “passing by reference” in which calls to the vessel are sent over the channel back to the deconstructor, which calls the original concrete instance and sends the return values back. Capabilities received from the orchestrator are generally Kinds provided in this manner, such as to provide access to abstracted hardware features or vessel orchestration. A vessel can also be sent in a manner similar to “passing by move” where the WASM binary itself is sent over the wire and instantiated transparently by the recipient (assuming that recipient possesses the requisite capabilities to request the orchestrator to instantiate this new vessel). For example, a request for a video transcoder capability could either be replied to with a “passed by reference” vessel providing access to NVENC or similar in hardware locally, or with a “passed by value” vessel that provides software transcoding and is instantiated by the recipient. All use of vessels is transparent to the developer and requires no boilerplate additional to that which would be required for equivalent concrete types. In fact, vessels are used directly as the concrete Kind they export, appearing transparently equivalent to that underlying type.
Vessels also provides an advanced reflection and dynamic casting system for interface erasure. Not only does Vessels provide a mechanism, the Kind derive macro, for implementing Kind for arbitrary structs and enums with no additional effort, it also provides the object macro which implements Kind for trait objects. These trait objects can be erased to a single type, reflected for individual methods, introspected for supertraits, and cast back into both their own concrete object type and concrete objects of their supertraits in a fully checked manner. This allows elegant dynamically-typed patterns for runtime composability, with vessel components implementing various traits and lenses and prisms permitting their appropriate interpretation for various ecosystem contexts.
#### Current Status
Kind is implemented for many common types as well as other crucial constructs such as boxed functions and futures/streams, derivation systems are fully working, use of Kinds over Channels is fully working. The reference Channel implementation, [`IdChannel`](https://noocene.github.io/vessels/vessels/channel/id_channel/struct.IdChannel.html), is fully working. Kinds can be exported from WebAssembly binaries, i.e. vessels, but the infrastructure required for their convenient use is not yet implemented. The core provider is implemented but does not yet provide an orchestrator due to the prior point, however [`Executor`](https://noocene.github.io/vessels/vessels/core/struct.Executor.html) is available and vessels can schedule tasks. The reflection engine is fully functional but, at the moment, [`Erased`](https://noocene.github.io/vessels/vessels/reflect/trait.Erased.html) is not yet a Kind object. This should be resolved shortly. Feature parity for all enumerated above exists across web and native and such parity will continue to be a goal. The current priority is the finalization of the core abstractions, with mostly orchestration systems and hardware abstraction remaining at this point, and the completion of reference systems, mostly [`IdChannel`](https://noocene.github.io/vessels/vessels/channel/id_channel/struct.IdChannel.html), such that demos and the first primitives of a growing ecosystem can be implemented.
Kind is implemented for many common types as well as other crucial constructs such as boxed functions and futures/streams, derivation systems are fully working, use of Kinds over Channels is fully working. The reference Channel implementation, [`IdChannel`](https://noocene.github.io/vessels/vessels/channel/id_channel/struct.IdChannel.html), is fully working. Kinds can be exported from WebAssembly binaries, i.e. vessels, but the infrastructure required for their convenient use is not yet implemented. The core provider is implemented but does not yet provide an orchestrator due to the prior point, however a global executor is available and vessels can schedule tasks. The reflection engine is fully functional but, at the moment, [`Erased`](https://noocene.github.io/vessels/vessels/reflect/trait.Erased.html) is not yet a Kind object. This should be resolved shortly. Feature parity for all enumerated above exists across web and native and such parity will continue to be a goal. The current priority is the finalization of the core abstractions, with mostly orchestration systems and hardware abstraction remaining at this point, and the completion of reference systems, mostly [`IdChannel`](https://noocene.github.io/vessels/vessels/channel/id_channel/struct.IdChannel.html), such that demos and the first primitives of a growing ecosystem can be implemented.
@@ -34,9 +34,8 @@ pub fn build(block: TokenStream) -> TokenStream {
unsafe {
let len = Box::from_raw(data.sub(size_of::<usize>()) as *mut usize);
let data = slice::from_raw_parts_mut(data, *len).to_vec();
let mut executor = ::vessels::core::<::vessels::core::Executor>().unwrap();
use ::vessels::futures::SinkExt;
executor.spawn(async move {
::vessels::core::spawn(async move {
DATA.lock().await.0.clone().send(data).await.unwrap();
});
}
@@ -54,27 +53,28 @@ pub fn build(block: TokenStream) -> TokenStream {
let data = format!("panic {}: {}", info, cause);
unsafe { _EXPORT_panic(data.as_bytes().as_ptr(), data.len()) };
}));
let _export_initializer: fn() -> ::vessels::kind::Future<_> = || Box::pin(async {
#block
});
let _export_initializer: fn(::vessels::core::Handle) -> ::vessels::kind::Future<_> = |handle| {
::vessels::core::register_handle(handle);
Box::pin(async {
#block
})
};
let _test_shims: () = {
trait export_helper {
type Kind: ::vessels::Kind;
}
struct export<T: ::vessels::Kind>(fn() -> ::vessels::kind::Future<T>);
struct export<T: ::vessels::Kind>(fn(::vessels::core::Handle) -> ::vessels::kind::Future<T>);
impl<T: ::vessels::Kind> export_helper for export<T> {
type Kind = T;
}
export(_export_initializer);
};
use ::vessels::{channel::IdChannel, OnTo, futures::{StreamExt, SinkExt, TryFutureExt}, format::{ApplyEncode, Cbor}, core, core::{Executor}};
let mut executor = core::<Executor>().unwrap();
let vessel = Box::new(_export_initializer) as ::vessels::core::Vessel<_>;
executor.spawn(async move {
use ::vessels::{channel::IdChannel, OnTo, futures::{StreamExt, SinkExt, TryFutureExt}, format::{ApplyEncode, Cbor}, core};
let vessel = Box::new(_export_initializer) as ::vessels::core::Constructor<_>;
::vessels::core::spawn(async move {
let (sink, mut stream) = vessel.on_to::<IdChannel>().await.encode::<Cbor>().split();
let receiver = DATA.lock().await.1.take().unwrap();
let mut exec = core::<Executor>().unwrap();
exec.spawn(receiver.map(Ok).forward(sink).unwrap_or_else(|_| panic!()));
::vessels::core::spawn(receiver.map(Ok).forward(sink).unwrap_or_else(|_| panic!()));
while let Some(item) = stream.next().await {
_EXPORT_safe_output(item);
}

This file was deleted.

@@ -1,7 +1,6 @@
use vessels::{
channel::IdChannel,
core,
core::Executor,
core::run,
format::{ApplyDecode, ApplyEncode, Cbor},
kind::using,
log, Kind, OnTo,
@@ -24,7 +23,7 @@ enum Enum<T: Kind> {
}

fn main() {
core::<Executor>().unwrap().run(async move {
run(async move {
let encoded = Enum::StructLike {
item: "hello".to_owned(),
another_kind: WithSerde { test: 10 },
@@ -1,7 +1,6 @@
use vessels::{
channel::IdChannel,
core,
core::Executor,
core::run,
format::{ApplyDecode, ApplyEncode, Cbor},
kind::Future,
log, OnTo,
@@ -12,7 +11,7 @@ type Call = Box<dyn Fn() -> Future<String> + Send + Sync>;
fn main() {
let call: Call = Box::new(|| Box::pin(async move { "hello".to_owned() }));

core::<Executor>().unwrap().run(async move {
run(async move {
let encoded = call.on_to::<IdChannel>().await.encode::<Cbor>();
let decoded: Call = encoded.decode::<IdChannel, Cbor>().await.unwrap();
log!("{}", (decoded)().await);
@@ -1,7 +1,6 @@
use vessels::{
channel::IdChannel,
core,
core::Executor,
core::run,
format::{ApplyDecode, ApplyEncode, Cbor},
kind::Future,
log, OnTo,
@@ -12,7 +11,7 @@ type Call = Box<dyn Fn(Vec<u8>) -> Future<String> + Send + Sync>;
fn main() {
let call: Call = Box::new(|data| Box::pin(async move { format!("{:?}", data) }));

core::<Executor>().unwrap().run(async move {
run(async move {
let encoded = call.on_to::<IdChannel>().await.encode::<Cbor>();
let decoded: Call = encoded.decode::<IdChannel, Cbor>().await.unwrap();
log!("{}", (decoded)(vec![5, 6, 7]).await);
@@ -1,7 +1,6 @@
use vessels::{
channel::IdChannel,
core,
core::Executor,
core::run,
format::{ApplyDecode, ApplyEncode, Cbor},
kind::Stream,
log, OnTo,
@@ -14,7 +13,7 @@ type Call = Box<dyn Fn() -> Stream<u8> + Send + Sync>;
fn main() {
let call: Call = Box::new(|| Box::pin(iter(1..10)));

core::<Executor>().unwrap().run(async move {
run(async move {
let encoded = call.on_to::<IdChannel>().await.encode::<Cbor>();
let decoded: Call = encoded.decode::<IdChannel, Cbor>().await.unwrap();
let mut stream = (decoded)();
@@ -1,13 +1,12 @@
use vessels::{
channel::IdChannel,
core,
core::Executor,
core::run,
format::{ApplyDecode, ApplyEncode, Cbor},
log, OnTo,
};

fn main() {
core::<Executor>().unwrap().run(async move {
run(async move {
let encoded = "Hello there"
.to_owned()
.on_to::<IdChannel>()
@@ -1,9 +1,8 @@
use vessels::{
channel::IdChannel,
core,
core::{
orchestrator::containers::{native::NativeContainers, Containers},
Executor, Vessel,
run, Constructor, Core
},
format::{ApplyDecode, Cbor},
log,
@@ -14,12 +13,11 @@ use std::fs::read;
pub fn main() {
let binary =
read("../../target/wasm32-unknown-unknown/debug/examples/test_vessel.wasm").unwrap();
let mut executor = core::<Executor>().unwrap();
executor.run(async move {
run(async move {
let mut containers = NativeContainers;
let module = containers.compile(binary).await;
let instance = containers.instantiate(&module).await;
let data: Vessel<String> = instance.decode::<IdChannel, Cbor>().await.unwrap();
log!("{}", data().await);
let data: Constructor<String> = instance.decode::<IdChannel, Cbor>().await.unwrap();
log!("{}", data(Core::new().as_handle()).await);
});
}
@@ -1,17 +1,18 @@
use vessels::{
channel::IdChannel,
core,
core::{hal::network::Client, Executor},
replicate::collections::List,
core::{hal::network::Client, run},
format::Cbor,
log,
replicate::collections::List,
};

pub fn main() {
core::<Executor>().unwrap().run(async move {
run(async move {
let mut network = Client::new().unwrap();
let mut data = network
.connect::<Box<dyn List<String>>, IdChannel, Cbor>("ws://127.0.0.1:61200".parse().unwrap())
.connect::<Box<dyn List<String>>, IdChannel, Cbor>(
"ws://127.0.0.1:61200".parse().unwrap(),
)
.await
.unwrap();
data.push("test".to_owned()).await;
@@ -1,7 +1,6 @@
use vessels::{
channel::IdChannel,
core,
core::Executor,
core::run,
format::{ApplyDecode, ApplyEncode, Cbor},
kind::Future,
log, object, Kind, OnTo,
@@ -23,7 +22,7 @@ impl<T: Kind + Display> ExampleObject<T> for Implementor {
}

fn main() {
core::<Executor>().unwrap().run(async move {
run(async move {
let encoded = (Box::new(Implementor) as Box<dyn ExampleObject<String>>)
.on_to::<IdChannel>()
.await
@@ -1,15 +1,15 @@
use vessels::{
channel::IdChannel,
core,
core::{hal::network::Server, Executor},
core::{hal::network::Server, run},
format::Cbor,
replicate::{collections::List, Share, Shared},
};

type Collection = Shared<dyn List<String>>;

pub fn main() {
core::<Executor>().unwrap().run(async move {
run(async move {
let collection = Collection::new(Box::new(vec![]));
let mut server = Server::new().unwrap();
server
@@ -1,7 +1,6 @@
use vessels::{
channel::IdChannel,
core,
core::{spawn, Executor},
core::{run, spawn},
format::{ApplyDecode, ApplyEncode, Json},
kind::Sink,
log, OnTo,
@@ -12,7 +11,7 @@ use futures::{channel::mpsc::channel, future::pending, stream::iter, SinkExt, St
fn main() {
let (sender, mut receiver) = channel(0);
let sender: Sink<u32, ()> = Box::pin(sender.sink_map_err(|_| panic!()));
core::<Executor>().unwrap().run(async move {
run(async move {
spawn(async move {
while let Some(item) = receiver.next().await {
log!("{}", item);
@@ -1,9 +1,8 @@
use vessels::{
channel::IdChannel,
core,
core::{
orchestrator::containers::{web::WebContainers, Containers},
Executor, Vessel,
run, Constructor,
},
format::{ApplyDecode, Cbor},
log,
@@ -16,11 +15,11 @@ const WASM_DATA: &'static [u8] =
#[wasm_bindgen(start)]
pub fn main() {
std::panic::set_hook(Box::new(console_error_panic_hook::hook));
core::<Executor>().unwrap().run(async move {
run(async move {
let mut containers = WebContainers;
let module = containers.compile(WASM_DATA).await;
let instance = containers.instantiate(&module).await;
let data: Vessel<String> = instance.decode::<IdChannel, Cbor>().await.unwrap();
let data: Constructor<String> = instance.decode::<IdChannel, Cbor>().await.unwrap();
log!("{}", data().await);
});
}
@@ -24,8 +24,7 @@ use std::{

use crate::{
channel::{Channel, Context as IContext, Fork as IFork, ForkHandle, Waiter},
core,
core::{spawn, Executor},
core::spawn,
kind::{Future, Sink},
Kind, SerdeAny, Target,
};
@@ -240,9 +239,8 @@ impl<'a, K: Kind> IShim<'a, IdChannel, K> for Shim<K> {
};
let fork = channel.get_fork::<K>(ForkHandle(0));
let (sender, receiver) = channel.split();
let mut executor = core::<Executor>().unwrap();
executor.spawn(receiver.map(Ok).forward(sink).unwrap_or_else(|_| panic!()));
executor.spawn(
spawn(receiver.map(Ok).forward(sink).unwrap_or_else(|_| panic!()));
spawn(
stream
.map(Ok)
.forward(sender)
@@ -466,14 +464,13 @@ impl<
context,
in_channels: Arc::new(Mutex::new(in_channels)),
};
let mut executor = core::<Executor>().unwrap();
executor.spawn(
spawn(
receiver
.map(move |v| Ok(Item::new(handle, Box::new(v), ct.clone())))
.forward(csender)
.unwrap_or_else(|_| panic!()),
);
executor.spawn(
spawn(
kind.deconstruct(IdChannelFork {
o: Box::pin(oi),
i: Box::pin(oo),

0 comments on commit 1d5e918

Please sign in to comment.
You can’t perform that action at this time.