New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC] WIT support for wasmCloud (wRPC) #1128
Comments
I think that as a foundation for one of the biggest changes we'll make before 1.0.0 this RFC looks great and well thought out. I've tried out the wRPC example (and it worked!) and I'm very happy with the developer experience of the examples. I'll have a followup RFC to publish here in the next few days regarding some higher level changes for links, RPC subscriptions for components/providers, etc etc. For any developers looking for how custom WIT and actor to actor calls will work in the WIT/componentized world, this RFC enables that 🚀 |
How do you encode |
Ah, good point, forgot to mention that one - strings are UTF-8-encoded and |
How do you handle partial type definition in Wube? Like |
yes, indeed. By the way, a few updates to the encoding to better match Wasm encoding:
|
I assume that enums and variants are now also encoded using LEB128, so you don't need to know the total number of possible values any more? |
that's the thinking, yes! |
Wasn't |
I moved this issue to Completed for the wasmCloud 1.0 roadmap to signify that the |
RFC: WIT support for wasmCloud
Introduction
This RFC proposes a set of changes, which will add support for distributed WIT interface implementations to wasmCloud.
Motivation
Existing wasmCloud RPC protocol and linking are incompatible with component model and WIT. Only a subset of functionality provided by WIT can currently be utilized by wasmCloud applications and wasmCloud as a product would very soon become unusable for the purpose of running real-world modern Wasm components.
For example, once
wasi:http
interfaces are integrated in the various Rust ecosystem crates, with the current protocol in place, running such applications within wasmCloud would produce undefined behavior at runtime breaking applications in unpredictable ways due to the fact that wasmCloud today cannot support e.g. bidirectional streaming of values required bywasi:http
package interfaces.Almost all of WASI is built on top of Resource types, which wasmCloud does not support today.
Detailed Design
Overview
WIT
Most of WIT specification is defined at https://component-model.bytecodealliance.org/design/wit.html (currently outdated). Most notably, the specification in its current state lacks definition of Resource types, which are a core primitive on top of which all of WASI functionality is built. Resources can be thought of as an opaque combination of a set of behaviors and (optional) state - in some ways, Resources are similar to Go interfaces or Rust dynamic trait objects. Resource state and behavior may not be decoupled, meaning that in order to be sent over the network, both of these components (resource state and resource behavior) must be represented in some way on the wire.
WIT interfaces are asynchronous in nature and rely on two-way communication between caller and callee in the context of a single call. For example, current definition of
wasi:http/outgoing-handler.handle
(https://github.com/WebAssembly/wasi-http/blob/2befff608aa362aef20d79888b993787acd2f010/wit/handler.wit#L39-L42) relies on simultaneous asynchronous request body streaming from the caller to callee and asynchronous streaming of response body from the callee to the caller. The result of this function is yet another asynchronous value - a result, which the caller polls for.wRPC v0.0.1 specification
wRPC protocol is designed for asynchronous transmit of WIT interface function calls and associated values over network.
As WIT interfaces and associated values are asynchronous in nature, callers and callees require ability to asynchronously, bidirectionally transfer (portions of) function parameter and return value data. For example, caller of
wasi:http/outgoing-handler.handle
function MUST be able to simultaneously send data over the passedoutput-stream
and well as receive data from the returnedinput-stream
.Transport
Primary transport used by wRPC is NATS, however this protocol can be implemented in terms of other transports as well, e.g. HTTP/2 or QUIC. Following this design decision, endpoint names are URL-safe. In URL form,
.
characters must be replaced by/
.wRPC protocol operates under assumption that globally-unique IDs can be generated by the caller (client). No particular type of identifier is required by wRPC by specification, but in case of NATS transport, the common NATS inbox concept is assumed to be used throughout this specification.
The default data encoding format is assumed to be
wube
defined in the appendix of this specification, implementations are free to use a different format, however, which can be specified usingContent-Type
header and/or additional out-of-band configuration mechanism.wRPC NATS topics (or URLs) assume to be rooted at a particular (optional) prefix, this prefix is configured out-of-band from perpective of this specification.
Invocations
wRPC follows client-server model, where peers may serve function (servers) and method calls invoked by the other peers (clients).
An optional concept of a "session" is used to facilitate function or method call invocations, which require asynchronous two-way data transfer.
On a high level, lifecycle of an arbitrary wRPC invocation looks the following:
T
corresponding to WIT function or methodF
served by itselfT
carrying, optionally truncated, encoded parameters to functionF
and reply topicR_c
R_c
with a reply topicR_s
- this initiates a "session"R_s
subtopics concurrently with the rest of the invocation flow.F
execution.F
parameters contain server-writable async values, server sends data onR_c
subtopics concurrently with the rest of the invocation flow.F
returns:R_c.results
.F
return values contain async values, server sends data onR_c.results
subtopics concurrently.F
traps or execution is not possible for some other reason:R_c.error
Structured data field addressing
To facilitate structured data transfer, a reflective zero-indexed numeric (u32-sized) type path is used.
Headers
wRPC packets may carry headers, which are defined in this section. All headers are optional unless specified otherwise.
Content-Type
This header is used to override default encoding format, it's not set for packets using
wube
encoding defined in this specification.See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type
Examples:
application/json
application/msgpack
application/octet-stream
text/plain
Content-Range
This header MUST be set for messages carrying partial payload.
See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range
Examples:
bytes 0-100/*
bytes 100-200/300
Content-Range: bytes 0-9/100
header MUST be set.Content-Range: bytes 10-19/*
header MUST be set.Transfer-Encoding
See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
Examples:
chunked
gzip, chunked
Function invocation
Topic:
[<prefix>.]?wrpc.0.0.1.<wit-namespace>:<wit-package>/<wit-interface>[@<wit-package-version>]?.[<function>|<wit-resource>.<function>]
URL:
example.com/[<prefix>/]?wrpc/0.0.1/<wit-namespace>:<wit-package>/<wit-interface>[@<wit-package-version>]?/[<function>|<wit-resource>.<function>]
Topic examples:
MBGL42DWFPGIEI63P333NCZW5BAGYJGSGLAIB6U7PPXKSXKJK74QTUZM.wrpc.0.0.1.wasi:http/outgoing-handler.handle
NARNEZWUJIOUEDOHI6BDRRFST5W6SHMTQXX5CVOBJC7Z4BQ63S2DKZH6.wrpc.0.0.1.wasi:http/outgoing-handler.handle
VD7C7DD6H5XSIL737EEVTHF7G6EYTMIPQLVOE2BLQDC7TEOGTUZECJYF.wrpc.0.0.1.wasi:http/outgoing-handler.handle
VD7C7DD6H5XSIL737EEVTHF7G6EYTMIPQLVOE2BLQDC7TEOGTUZECJYF.wrpc.0.0.1.wasi:http/outgoing-handler@0.2.0-rc-2023-11-10.handle
default.wrpc.0.0.1.wasi:http/outgoing-handler.handle
default.wrpc.0.0.1.wasi:http/types/fields.constructor
default.wrpc.0.0.1.wasi:http/types@0.2.0-rc-2023-11-10.fields.constructor
custom.wrpc.0.0.1.wasi:http/types@0.2.0-rc-2023-11-10.fields.constructor
URL examples:
example.com/wrpc/0.0.1/wasi:http/types@0.2.0-rc-2023-11-10/fields.constructor
Messages sent on this topic represent invocations of WIT interface and static resource functions.
Messages sent on this topic MUST specify the reply inbox subject.
The request payload represents the encoded function call parameter tuple.
The caller (client) may truncate the payload and only send first N bytes of the payload, including sending no bytes at all.
Method invocation
Topic:
<resource-topic>.wrpc.0.0.1.<method>
URL:
<resource-url>/wrpc/0.0.1/<method>
Messages sent on this topic represent invocations of WIT methods on previously-constructed resources managed by the remote parties (identified by
resource-topic
orresource-url
respectively).Messages sent on this topic MUST specify the reply inbox subject.
The request payload represents the encoded method call parameter tuple.
The caller (client) may truncate the payload and only send first N bytes of the payload, including sending no bytes at all.
Invocation response
Once the function or method invoked finishes execution, the server responds on a topic derived from the reply inbox specified in the initial invocation.
<reply-inbox>.results
as the response topic, on which the (possibly truncated) encoded result tuple is transmitted according to data transfer procedure defined in this specification. Client and server may still exchange data as part of the session associated with the invocation after the result tuple has been fully transmitted to the client.<reply-inbox>.error
as the response topic, on which the (possibly truncated) error string is transmitted according to data transfer procedure defined in this specification. The session is immediately assumed to be closed on transmit of any message on this topic, meaning that the server MUST discard all session data associated with the invocation.Invocation session
Once the (possibly truncated) function/method parameter tuple is received by the server, it MAY immediately start executing the function/method indicated by the topic, even if full contents of the tuple are not available yet.
In cases where the function/method parameter tuple was truncated or contained pending asynchronous values and potential execution has not resulted in a trap (yet), the server MUST communicate a reply inbox to the client in one of the two ways:
If the invocation payload was truncated, the client sends the rest of the payload on the reply inbox negotiated in the above procedure.
Partial data transfer
In all cases with the exception of transmit of initial invocation payload, which is defined in previous section, when encoded value is required/desired to be truncated, the truncated part of the encoding is sent on the original topic and the rest is sent on the same topic, following the first transmit. This procedure is repeated as many times as necessary until the whole data is transmitted.
Asynchronous data transfer
If an asynchronous value is not fully-available (e.g. pending future) at time of encoding, it is encoded as "pending" (refer to encoding specification) in the structured data (e.g. parameter tuple) and sent asynchronously as it becomes available on a reply topic negotiated as part of the session with a reflective
/
-delimited structured value path suffix.Examples:
_INBOX.WMZAFf1AjlpSF3r5e65nFe.dOssD7ON.0
- this could be first parameter or return value_INBOX.WMZAFf1AjlpSF3r5e65nFe.dOssD7ON.0/0/1
_INBOX.WMZAFf1AjlpSF3r5e65nFe.dOssD7ON.0/0
_INBOX.WMZAFf1AjlpSF3r5e65nFe.dOssD7ON.0/1/0
_INBOX.WMZAFf1AjlpSF3r5e65nFe.dOssD7ON.0/1/1
_INBOX.WMZAFf1AjlpSF3r5e65nFe.dOssD7ON.0/1
_INBOX.WMZAFf1AjlpSF3r5e65nFe.dOssD7ON.1/2/1
Resources
wRPC protocol implementations are encouraged to "adapt" interfaces utilising resources to interfaces using e.g. records for performance.
For example of this approach, consider existing
wasi:http/outgoing-handler.handle
function defined at https://github.com/WebAssembly/wasi-http/tree/v0.2.0-rc-2023-11-10/wit (comments omitted, type aliases resolved, relevant resource dependencies moved close):An adaptation of the following interface could look like this:
WebAssembly Untyped Binary Encoding (
wube
)wube
encoding is a simple binary encoding optimized for:wube
is designed to fit well with stream-based APIs and technologies, it performs no compression, however users are encouraged to rely on streaming compression/decompression technologies (like DEFLATE, defined at https://datatracker.ietf.org/doc/html/rfc1951) to transferwube
values over network.Although
wube
-encoded values could potentially be speculatively decoded without access to the WIT associated with the original encoded value, it is assumed that both encoder and decoder rely on exactly the same WIT definitions when working withwube
values, and all the type information is communicated between parties out-of-band.wube
represents all numeric values as little-endianNumeric types
All numeric values are encoded as little endian with zero padding appropriate for the numeric value byte size.
For example:
Type:
u32
Value:
1
Encoding:
[0x01, 0x00, 0x00, 0x00]
Type:
u64
Value:
1
Encoding:
[0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00]
Characters
Characters are encoded as UTF-8 32-bit unsigned number.
For example:
a
Encoding:
[0x61, 0x00, 0x00, 0x00]
Enums
Enums are encoded as their discriminant encoded as an unsigned number of bit size matching nearest byte boundary of maximum value of the descriminant (in other words, discriminants of enums containing at most 255 options are encoded as an 8-bit unsigned number).
For example:
Type:
enum test { foo, bar, baz }
Value:
foo
Encoding:
[0x00]
Type:
enum test { foo, bar, baz }
Value:
bar
Encoding:
[0x01]
Variants
Variants are encoded as their discriminant encoded as an unsigned number of bit size matching nearest byte boundary of maximum value of the descriminant (in other words, discriminants of variants containing at most 255 options are encoded as an 8-bit unsigned number), followed by encoding of their values, if such is defined.
For example:
Type:
enum test { foo, bar(bool), baz(option<bool>) }
Value:
foo
Encoding:
[0x00]
Type:
enum test { foo, bar(bool), baz(option<bool>) }
Value:
bar(true)
Encoding:
[0x01, 0x01]
Type:
enum test { foo, bar(bool), baz(option<bool>) }
Value:
baz(none)
Encoding:
[0x01, 0x00]
Type:
enum test { foo, bar(bool), baz(option<bool>) }
Value:
baz(some(true))
Encoding:
[0x01, 0x01, 0x01]
Booleans
Booleans are encoded as values of
enum bool { false, true }
typeValue:
false
Encoding:
[0x00]
Value:
true
Encoding:
[0x01]
Options
Options are encoded as values of
variant option { none, some(value) }
typeResults
Results are encoded as values of
variant result { error(error), ok(value) }
typeRecords
Records are encoded as concatenated encoding of their fields in definition order.
For example:
record example { foo: bool, bar: u32 }
Value:
{ foo: true, bar: 1}
Encoding:
[0x01, 0x01, 0x00, 0x00, 0x00]
Flags
Flags are encoded as byte-aligned bit masks matching flag definition order.
For example:
Type:
flags example { foo, bar, baz }
Value:
{ foo, bar }
Encoding:
[0b11000000]
Type:
flags example { one, two, three, four, five, six, seven, eight, nine }
Value:
{ two, nine }
Encoding:
[0b01000000, 0b10000000]
Type:
flags example { one, two, three, four, five, six, seven, eight, nine }
Value:
{ one, two }
Encoding:
[0b11000000, 0b00000000]
Tuples
Tuples are encoded as concatenating encoding of their contained elements in the order of their definition.
For example:
tuple<bool, bool>
Value:
(first: true, second: false)
Encoding:
[0x01, 0x00]
Lists
Lists are encoded as their length encoded as 32-bit unsigned number followed by concatenated encoding of their elements in the list order.
For example:
list<bool>
Value:
[true, false]
Encoding:
[0x02, 0x00, 0x00, 0x00, 0x01, 0x00]
Futures
Futures are transmitted as
variant future { pending, ready(value) }
Streams
Streams are transmitted as
future<list<value>>
as part of structured data and aslist<value>
during field transfer.Content-Range
header (or other out-of-band mechanism) must be used to indicate the end of the stream during field transfer.Resources
In the general case, resources are represented as their unique URIs, e.g. a NATS inbox subject handling resource methods via wRPC, encoded as strings.
Implementations are encouraged to use alternate, optimized, representations of resource values (for example, a reference uniquely identifying the resource being owned by a particular party).
Function parameters and return values
Function parameters and return values are encoded as tuples of their contained values.
example: func(first: bool) -> (a: u8)
Value:
example(true) -> 2
Encoding:
[0x01] -> [0x02]
Well-known types and interfaces in wRPC 0.0.1
wasi:io@0.2.0-rc-2023-11-10
poll.pollable
values are encoded and transmitted asfuture<()>
values.streams.input-stream
andstreams.output-stream
are encoded and transmitted asstream<u8>
values.wasi:http@0.2.0-rc-2023-12-05
Functionality is transmitted using the following WIT contract:
Owned values of:
wasi:http/types.outgoing-request
type are transmitted aswrpc:http/types.outgoing-request
wasi:http/types.incoming-request
type are transmitted aswrpc:http/types.incoming-request
wasi:http/types.future-incoming-response
type are transmitted asfuture<wrpc:http/types.incoming-response>
wasi:http/types.request-options
type are transmitted asfuture<wrpc:http/types.request-options>
wasi:http/types.fields
type values are transmitted aslist<tuple<string,list<u8>>>
HTTP extensibility
Because of the flexibility of wRPC protocol, fully-specified, HTTP-to-NATS-lattice and NATS-lattice-to-HTTP bridges can be built. Potential use case for this could be a multi-tenant HTTP microservice deployed outside the lattice, which could match the tenant based on the invocation signature and perform some action, e.g. serve as bridge between two separate lattices.
Assumptions
wasi:http
orwasi:keyvalue
) - since we already have functionality likeprovider-wit-bindgen
, this would probably be the easiest point of entry for building a native binary capability provider plugging into NATS lattice directly.Backwards Compatibility
Potentially the existing top-level invocation topic could be reused, but since we're planning to release 1.0, we should try to make changes sooner rather than later
Alternatives Considered
Not supporting component model and/or resources and WASI cloud WIT
Unresolved Questions
Linking has to be reworked, a separate RFC incoming (refs #458)
A feature similar to invocation signing could be added later, but we need to first focus on communication layer, especially considering how little we gain by actually using it
All external metadata (like trace context), really should be sent via headers.
The text was updated successfully, but these errors were encountered: