Notes On The Design Of An Ocap Kernel

Notes On The Design Of An Ocap Kernel

@Chip Morningstar

Weaselly Disclaimer

This document has been posted primarily to make it available to the Ocap Kernel team, both to help my teammates stay aware of what I'm doing and to get their feedback. However, I still appreciate feedback from anybody else who has the time and energy to go down into the rabbit hole, regardless of who they are or what part of the organization they are in. That said, keep in mind that this is very much a work in progress. The content here is likely to vary wildly in form, substance, and coherence from one day to the next, as the writing here is not merely a record of my emerging design but very much a part of how I work out my thoughts as to what the design even is. If what you seek is some kind of global understanding of the design itself you might want to wait until it's more cooked; rest assured that I won't be quiet about it once some kind of good-enough-to-criticize threshold is reached (if you want to be actively notified at that point, just drop me a note). If what you seek is design details suitable for implementation or any kind of actual close scrutiny, you definitely should wait until it's more complete -- in its current state this is much more of a think piece than a specification, however much it might at times resemble the latter.

Names and Words

"The naming of names is a difficult matter, it isn't just one of your holiday games." -- not T. S. Eliot

We probably should pick a proper label for this thing we are creating. We've been referring to it as the "ocap kernel", but I think it's more than that. While our team is labeled "the ocap kernel team" (and I suspect the higher level management discussions that lead to our being organized and tasked with doing this probably used this language as well), there's a distinction to be made between the platform as a whole and the kernel per se. In particular, I expect we are headed for some kind of kernel+vats architecture. This suggests that there's almost certainly going to at least be a component that is the trusted supervisory code that lives inside the vat but above any user code. This component is clearly part of our platform but definitionally is not part of the kernel.

(Though I will note in passing that "ocap kernel" conveniently abbreviates as "OK", which might be a source of amusement as we contemplate the Naming Of Things.)

A further complication is that each user who is hosting some set of vats will be running a separate instance of our platform, and such instances probably also need a term to label them. An object inside a vat may exchange messages not only with objects in other vats on its same platform instance but also with objects in vats hosted by other users elsewhere on the internet. Consequently, we'll probably want a word meaning "all the vats hosted by some user". Also, I'm being a little loose here with the term "user". Our default mental model is of vats being hosted inside some person's web browser, and it's natural to think of that person as "the user", but of course we also want to consider vats being hosted by independent servers in some data center somewhere whose purpose is to provide services unrelated to any particular human -- it's the internet, after all. You could think of these servers (or their operators) as "users" in some sense, though at that point the language gets a little strained. And of course that's before we even start talking about message traffic exchanged with entities entirely outside our paradigm (via, say, HTTP or whatever), even if those entities remain part of the ecosystem we're concerned with.

So I'm just going to start inventing terminology and making up names for things. Don't get overly fixated on the language that I've chosen (despite the fact that I'm obsessing over it quite a bit here myself) but rather consider all these words provisional, subject to revision, elimination, or addition as we refine our design and get clearer in our own minds what we are doing. In this document I'm going to try to follow the convention writing any new bit of jargon in BOLD ALL CAPS the first time it appears (hopefully in the context of some kind of definition or explanation) and thereafter follow the Germanic convention of Capitalizing These Words when they're used.

Also, as I've been going I've been encountering unanswered design questions that I don't already have a strong opinion on. These are marked with OPEN DESIGN QUESTION in the text.

The Big Picture

Everything here takes as a given that we're building on a common base of Hardened JavaScript, the ocap security model, and various related concepts such as compartments, eventual messaging, and so on, as provided by the SES shim or the XS engine, plus various supporting packages in the Agoric endo repository for things like bundling or marshaling. For purposes of this discussion, I'm going to presume that everybody generally knows what these things are and is on board with this approach, rather than stopping to explain or justify these bits, except when there are nuances that merit deeper discussion (ultimately, of course, there will be more outward reaching documentation that explains All The Things, but that's for later).

At the bottom we have OBJECTS. Because we are building in Hardened JavaScript, these are all just JavaScript objects, but for our purposes we further divide them into three varieties: DATA OBJECTS, BEHAVIORAL OBJECTS, and ORDINARY OBJECTS. This distinction is relevant in the context of marshaling and serialization, and in the places those are used, namely messaging and persistence.

• Data Objects are passed by value; they are selfless (that is, they have no identity) and are described entirely by the bits from which they are composed. All own properties of Data Objects must be either (a) JavaScript primitive values (e.g., number, string, boolean), (b) References to Behavioral Objects, or (c) recursively nested Data Objects. Since Data Objects are passed by value, they cannot have methods (i.e., own properties that are functions) because these can't be serialized. The prototype of a Data Object must be null or the JavaScript built-in object Object.

• Behavioral Objects are passed by reference. They have identity and may have methods, but can have no visible state (i.e., no JavaScript own properties other than methods). An empty object (i.e., one with no own properties at all) is considered a Behavioral Object since References to empty object instances are extremely useful as passable tokens (i.e., things that have identity and essentially nothing else).

• Ordinary Objects are regular JavaScript objects that remain inside the Vat. They may have any mixture of JavaScript properties whatsoever but are not passable.

Each Object resides within some VAT, an isolated process that can send and receive messages. When a message is sent from an Object in a Vat to the world outside that Vat, the message may contain OBJECT REFERENCES that designate Behavioral Objects (which are typically, but by no means necessarily, contained within the Vat sending the message). When a message is received by a Vat from the outside world, the message may contain Object References that designate Behavioral Objects contained by other Vats. Such References can be used by code within the Vat as targets for future messages it might send.

Broadly speaking, a Vat consists of two parts: a SUPERVISOR and USER CODE.

The Supervisor is trusted, privileged code that is part of our platform. It is responsible for:

• receiving, deserializing, and queueing messages inbound from other Vats, then, when appropriate according to our code execution model, dequeueing, demarshaling, and dispatching these to User Code.

• accepting, marshaling, and queueing messages sent from User Code to other Vats, then, when appropriate, dequeueing, serializing, and transmitting them to the outside world (specifically, transmitting them via the Kernel; more on this shortly).

• overseeing persistent storage, both for the Supervisor's own use (e.g., holding the inbound and outbound message queues) and for the storage of persistent objects belonging to User Code.

• providing other system level services to User Code as we see fit to provide, potentially including process termination, access to other communications channels (e.g., HTTP), clocks, timers, and so on. The list of possible services that might be included here is open ended and potentially large, though of course we'll want to constrain and thoroughly specify it in the final design.

User Code is a bundle of arbitrary, user-provided JavaScript that is executed within a compartment that the Supervisor maintains for that purpose. In general, User Code is untrusted, except insofar as we might have some kind of vetting and certification procedure in place for a limited number of special, privileged applications to be granted special endowments that provide capabilities to perform selected sensitive operations that are normally closely held (e.g., access to the clock or to the browser DOM).

The Vats running on a given host are managed by a KERNEL. The Kernel and Vats together are called a CLUSTER (note: of all the vocabulary I've been defining here, "Cluster" is the word I'd most like to have a better alternative for). The Kernel is responsible for:

• creating, loading, starting, pausing, resuming, and stopping the Vats in its Cluster (note that a Vat might be created as the result of extrinsic user action or by the intrinsic action of another Vat via a service that the Kernel makes available to it for this purpose)

• managing the granting of powers to specific Vats under the control of the user whose Cluster it is

• providing a KERNEL MANAGEMENT API, enabling a user interface by which its associated user can control or configure the Kernel's operation and that of its Vats

• conveying messages between the Vats that it is managing

• receiving messages inbound from other Clusters and forwarding them to the appropriate Vats

• accepting messages outbound from Vats in its Cluster that are targeted at objects hosted in other Clusters, and transmitting them over the network to those Clusters

• translating inbound and outbound object references (contained in messages) between Vat-relative and global forms

• Possibly: holding and managing the received message queues for any unsettled promises for which one of its Vats is the decider (OPEN DESIGN QUESTION: should responsibility for queueing messages to an unsettled promise be vested in the Kernel or in the deciding Vat? As a piece of Vat-associated state, there's obvious sense in having the Vat do it, but to the extent that promises can (and possibly often will) resolve to objects external to the deciding Vat, it might make sense to decouple this component of promise state from the Vat itself, in order to enable more efficient message forwarding, to limit certain kinds of potential Vat misbehavior, and ultimately to support automatic shortening of promise chains.)

A Cluster will be hosted either by a browser extension (where the primary UX will be presented in the browser itself via the web interface) or by a standalone Node executable (where the primary UX will most likely be command line arguments and configuration files). Whether it is presented as an interactive web page or as a CLI, we'll refer to the the controlling entity as the CONSOLE. The Console governs the operation of the Cluster via the Kernel. A Console itself consists of two parts:

• A CONSOLE DAEMON, an independent process that maintains persistent console state independent of the execution state of the Kernel or any Vats. It interacts with the Kernel via the Kernel Management API. Note that the Console Daemon needs to be independent of the Kernel because one of its jobs will be launching the Cluster in the first place, prior to which there is no Kernel. The Console Daemon in turn presents an API to the:

• A CONSOLE USER INTERFACE, a possibly ephemeral process (or a series of such processes over time) that actually implements the user facing controls for the cluster, whether via a web interface, a CLI, or something else.

One of the things that the Console Daemon maintains is the user's collection of PETNAMES, which are human readable, human meaningful, user assigned labels for those Objects within the Cluster's Vats that have been made available for the user's direct manipulation (using the Console User Interface) as the result of operations invoked from the Console.

We anticipate there will be a single Console Daemon implementation that serves for all purposes, but at least two and possibly more Console User Interface implementations that vary depending on the particulars of the type of control UX that is being realized.

Typically, a given machine will only host a single Cluster, with the notable exception of multiple Clusters on a machine in service of testing and debugging (i.e., not the normal use case but one that is still important to support as part of the DX).

Object Addressing

The ultimate affordance of an Object Reference is its use as the target destination for a message send, though the use of Object References as identity bearing tokens (i.e., values that can be compared with === and !==) is also supported.

At bottom, any given message is sent from one Object to another. However, to describe how things actually work we also need to consider the role of Vats and Clusters. In particular, a message sent to an Object in another Vat within the same Cluster as the sender needs to be handled rather differently from a message sent to an Object in a Vat in a different Cluster.

Within a Cluster, a message sent by an Object in Vat A to a different Object in Vat B is transferred from the sender to the receiver via the Kernel. The Kernel needs to be able to determine, based solely on the Object Reference itself, which Vat to deliver the message to, and in what form to deliver it such that the receiving Vat's Supervisor will understand which of its Objects is to be the actual recipient.

The primary way (aside from bootstrapping, about which more in a moment) that a Vat could know the identity of an Object outside itself is if the reference to that Object had previously been imported into the Vat as an argument of an earlier message, which in turn means that that Object Reference had to have been exported by the originating Vat as part of (or prior to) sending that earlier message. Export preceding import means that the Kernel is always able to associate any Object Reference with the Vat that exported it, and thus knows where to deliver any subsequent message addressed to that same Reference from a different Vat.

In order for this process of Reference begetting Reference not to lead to infinite regress, some kind of bootstrap mechanism is required, else there'd be no way for any Vat to ever know about anything outside itself in the first place and thus no way for it to ever send messages anywhere. To bootstrap the reference graph, whenever a Vat is created, its initialization creates within it an initial ROOT OBJECT, a Reference to which is exported to the creator as the result of the Vat creation operation. If the Vat was created by the action of User Code in another Vat, this Root Object Reference is delivered to the creator Object as the return value from its invocation of the Vat creation operation. If the Vat was created by an action of the user using the Console, the Root Object Reference is placed into the user's Petname table for potential future use in operations commanded via the Console (see the discussion of Petnames below).

The logic for messages between Objects in separate Clusters does not map directly to that for messages between Objects in Vats within the same Cluster. The first important difference is that an observer outside a given Cluster has no way to know which of that Cluster's Vats a given Object Reference feeds into. On the other hand, there's no particular reason for this outside observer to know or care about which specific Vat it is either. Instead, we consider the internal organization of a Cluster to be the Cluster's private business. Outside an Object's containing Cluster we only associate a Reference to it with the Cluster itself. Another way of putting this is: since you don't have visibility into the inner workings of a Cluster, you must treat all the Objects within the Cluster as if the Cluster was a single giant Vat, and that any finer-grained distinction within that Cluster that you might become aware of is an illusion presented to you by the Cluster's Kernel. The logic of References begetting References via message passing in the intra-Cluster case is recapitulated in the inter-Cluster case, but at the granularity of Clusters rather than the granularity of Vats.

A second, and probably more important, difference is that between Clusters there is no entity analogous to a Kernel possessing visibility onto all the Object Reference relationships that are in play between Clusters. Instead, Clusters connect to each other in an arbitrary peer-to-peer network rather than via the star topology that the Kernel organizes its Vats into. This has important implications with respect to how connectivity between Objects is bootstrapped across Cluster boundaries.

Whereas Vats are created by other Vats via the Kernel or by the user via the Console, new Clusters are created by the autonomous actions of users, external to any pre-existing Cluster (e.g., a user boots a new Cluster instance on their machine and connects it to the internet). In particular, the Cluster per se has no notion of a Root Object, since in its initial state a Cluster need not have any Vats at all. Once the Cluster does have Vats (either through later user action via the Console or as a consequence of configuration information provided as part of Cluster creation), these Vats (or rather, the Objects within them) have no a priori connection to anything, so initial connectivity between Clusters cannot be established purely by reference passing. Instead, we require some means for a Cluster's user, or software running on the user's behalf, to publish the equivalent of an Object Reference in a form that can be communicated out of band to the users of other Clusters, who in turn need a means to accept these and convert them into regular Object References of their own. To this end, the Kernel has the capability to translate between Object References, which have meaning in the context of messages between Objects as represented in the implementation of the inter-object message passing scheme, and OCAP URLS, which are text strings that can be communicated via pretty much any means you like (email, SMS, PostIt notes, QR codes on the sides of buses, etc.). The two directions of translation (from Ocap URL to Object Reference and vice versa) are treated as capabilities that are made available to the user via the Console. From the Console these capabilities may be used directly or granted to selected Objects to which the user has access.

Taken all together, the arrangements discussed above imply five different possible forms an Object Reference may take depending on context:

• A VREF (short for "vat reference") designates an Object within the scope of the Objects known to a particular Vat. It is used across the Kernel/Vat boundary in the marshaling of messages delivered into or sent by that Vat. A Vref is generated and assigned by the Kernel when importing an Object Reference into a Vat for the first time and by the Vat when exporting an Object Reference from it for the first time.

• A KREF (short for "kernel reference") designates an Object within the scope of the Kernel itself. It is used in the translation of References between one Vat and another. A Kref is generated and assigned by the Kernel whenever an Object reference is imported into or exported from a Vat for the first time. Krefs are strictly internal to the Kernel implementation. (Note: the differentiation between Vrefs and Krefs is important because it enables the Kernel to maintain 2N Reference mapping tables for N Vats rather than having to potentially maintain N² mapping tables.)

• An RREF (short for "remote reference") designates an object within the scope of an established point-to-point communications Channel between two Clusters (more on Channels below). An Rref does not survive the Channel it is associated with. An Rref is generated when the Kernel for one Cluster exports an Object Reference into the Channel connecting it to the Kernel for another Cluster.

• An OCAP URL is an externalizable string that designates an object globally in a form suitable for transmission out of band. It incorporates addressing information that allows Kernel identify the Cluster containing the Object it refers to and create a Channel to it if one does not already exist or to obtain the existing Channel if one does. It designates an Object in a way that is independent of whether the Cluster containing the Object is currently available or not and independent of the Channel used to establish connectivity. It cannot be used to address messages directly, but can be used to establish connectivity and ultimately obtain an Rref which does permit messaging.

• A PETNAME is a human readable, human meaningful, user assigned label that designates an Object within the scope of a given Cluster's Console. Internally, the Console maintains a table mapping these labels to Krefs. Petnames may manifest as textual names, such as a user might type or paste into a text input box on a web form, or draggable labels (with distinguishing text, icons, or both) that can be manipulated as part of an interactive Console GUI, or they might take some other form depending on the Console UX design, as long as they have the property of being directly meaningful to the user. One user's Petname for an Object is unrelated to another user's Petname for the same Object. (Note that the two users might happen to pick the same Petname string, perhaps deriving it from some functional role, known to both of them, that the Object plays in their world. However, such coincidences cannot be relied on and even when they do happen no deeper meaning can safely be inferred from them.)

Inter-Cluster Communication and Ocap URLs

A CHANNEL is a communications connection between two Clusters that provides a context for message traffic between them. In particular, it establishes a scope for the interpretation of shared Rrefs. A Channel is a sturdy version of a TCP-like session connection. It may, in fact, be transported over an ordinary TCP (actually, more likely TLS) connection, but it can survive loss and reestablishment of the underlying transport session, as well as termination and restart of the Kernel or Vat processes at either end, while preserving all of the Object References stretched over the connection and preserving unbroken message streams back and forth between the two endpoints.

Connecting one Cluster to another requires solving two important sub-problems: (a) knowing or learning where to make a network connection (i.e., what IP address and port number to connect to) in order to interact with the Cluster, and, (b) once having connected, verifying that what one has connected to is, in fact, the Cluster one actually intended to connect to (or, indeed, if it is a Cluster at all).

In addition, the actual mechanism of network connectivity needs to account for our intention to host Clusters inside users' web browsers. This is significant because browsers commonly run on machines that aren't in a position to directly accept incoming network connections due to network address translation, dynamic IP address assignment, firewalls, proxies, and all the other myriad ways that the IT organizations of the world have gummed up the proper functioning of TCP/IP networks over the years. Unless and until proven otherwise, I'm going to assume we can use one of the several pre-existing solutions to this problem that clever people have devised rather than trying to invent our own workaround. However, we do need to acknowledge that whatever solution we adopt will most likely end up as a key part of our network stack, since it will almost certainly need to be baked into our design at a fairly fundamental level. Consequently, for the time being I'm just going to treat this as a solved problem, making the naive but useful presumption that one computer can make a connection to another at will, while handwaving away all the complications.

An Ocap URL contains several distinct pieces of information:

• Network addressing information that indicates how to contact the Cluster that contains the object it designates.

• Cluster identification information that enables the party establishing a Channel to verify that it is validly in contact with the proper Cluster.

• An opaque encoding of the object's identity in a way that can be decoded by the target Cluster but which by itself does not uniquely reveal the Object's identity. In particular, multiple different Ocap URLs may all encode the same Object Reference, and it should generally be considered a best practice to vend a new, unique Ocap URL each time an Object Reference is externalized from its containing Cluster.

Verifying that a connection has been made to the proper endpoint as part of setting up a secure communications pathway is a job traditionally performed by TLS. However, in order to authenticate that a communications counterparty is the one expected, in normal operation TLS leverages DNS: when connecting to a server via TLS, the server provides a cryptographic certificate attesting to the association between the DNS name of the server and the public key used in the TLS handshake to generate the session key. This certificate is signed by a certificate authority (CA) whose validity is verified either by another certificate signed by a higher level CA (repeating this process recursively) or by being found in a collection of root CAs that the browser is pre-configured to automatically trust. The determination of the ultimate authenticity of the root CAs is highly centralized, leading to a number of points of vulnerability -- the authorities maintaining the root CA list could be compromised, individual CAs could be compromised without their parent CA's knowledge, revocation of a compromised CA's certificate might not have reached the browser (for many different possible reasons), or the root CA list held by the browser could itself be compromised (either within a given user's environment or at some point in the browser's implementation, build, and distribution path). Many of these failures have, in fact, occurred repeatedly in the history of the web. DNS itself is also a point of vulnerability: in the event that the operator of a server loses control over the DNS registry for its domain name, some other party may be positioned to substitute its own server address and corresponding TLS certificate chain. These sorts of compromises have also happened in practice. Because the root of the CA hierarchy is highly centralized, there remains a risk that some of these compromises could happen at scale. A further problem is that even without security failures, establishing the connection requires that the entity connected to possess the requisite certificates in the first place, which is itself administratively complex and highly error prone (and whether or not DNS is involved merely varies the modes of complexity and failure).

An alternative approach, which I want to advocate here, is for our Ocap URLs to employ something like Tyler Close's YURL scheme. While there are lots of details and possible variations, the basic idea behind them all is simple: embed the fingerprint of the contactee's public key into the URL itself, so that when a connection is made its authenticity can be validated directly. Tyler's documentation (just linked) shows how this can be accomplished with existing, mature TLS implementations, requiring only small and relatively simple changes to the way TLS is used. In the 20+ years since YURLs were first proposed they have seen only experimental adoption, in large part because of the huge entrenched installed base built around TLS, HTTPS, and the CA hierarchy, but also because YURLs and their brethren involve URLs that are large and unwieldy, giving them very poor ergonomics in many web use cases. However, because we are already handling Object References using Petnames combined with automation, our typical use cases do not generally require exposing representations of Object References to humans. YURLs do lend themselves very well to being handled via automation in such a fashion, which will allow us to have much better decentralization, avoid a large swath of potential security vulnerabilities, and bypass a lot of complicated, difficult to understand, and error prone administrative procedures (the latter being particularly important if we want this to be used by ordinary users in their browsers rather than just by professional IT people).

(OPEN DESIGN QUESTION: should we have a distinct Ocap URL scheme such as ocap for these URLs or are we better off sticking to some convention for using, say, https? Related: should we look into the cap URL scheme that Mark Miller registered years ago for E?)

(OPEN DESIGN QUESTION: should the host address in an Ocap URL be regarded as the actual site to contact to reach the Object in question or should it be regarded as a hint as to where one might look, with the URL providing entry into a directory service if it doesn't reach the Object directly? In some of the work with E we referred to this as a "redirectory", drawing analogy to a web server that might serve up a page or might serve up an HTTP redirect.)

Theory of Operation

With a couple of exceptions (which will be described when we get to them), we conceive of the interfaces between the various communicating components of the overall system in terms of asynchronous messaging between loosely coupled actors. Note that use of the term "message" here can be a source of confusion, since among the most common and important kinds of things being transported by messages between components are messages between Objects, but these are "messages" at two different levels of abstraction. When helpful for clarity I will sometimes refer to SYSTEM MESSAGES or OBJECT MESSAGES to disambiguate what is being talked about. All the traffic on the interfaces described in this section consists of System Messages; some of these System Messages will carry Object Messages in their payloads.

Typically, connections between processes on the same machine will be carried by interprocess pipes. Channels, which nominally connect across machine boundaries, will be carried via a network protocol layered over TCP/IP. In any case, these connections are always asynchronous and bidirectional. We will of course hide all that under an abstraction layer so that the components themselves are shielded from having to know the particulars of how they talk to each other.

Between any two kinds of components we define an interface that specifies what the allowed messages in each direction are and what operations they invoke. Every component implements some set of methods, some of which may be made available on more than one of its interfaces. In this sense, methods and interfaces are slightly orthogonal, despite most methods being interface-specific. Consequently, we will describe the some of the components below in terms of the methods they provide (and the operations they realize) and then separately indicate which interfaces those methods are exposed on.

Message Translation

One operation that is needed in various places is message translation, wherein a marshaled Object Message is converted from the form needed in one frame of reference to that needed in another. For example, transferring a message from the Kernel to some Vat requires converting all the Krefs it contains to that Vat's corresponding Vrefs. While this minimally includes simply mapping the various Object Reference strings through a table, importing an the Object into some context where it was previously unknown may also require generating a new Reference identifier for it in the receiving context.

Although this design is not yet committing to a specific scheme for marshaling messages and related data values, any acceptable scheme will need to make it easy and efficient to find and translate Object References embedded in a message's encoded representation. Agoric's capdata and smallcaps schemes both satisfy this requirement, as does OCapN. For expository purposes I'll adopt the approach (and accompanying terminology) used by both of Agoric's schemes. These schemes break the representation of a Data Object into two parts, labeled body and slots.

• slots is an array of strings, each of which is one of the Object References found in the Data Object being marshaled (nominally deduplicated, though this is not strictly required).

• body is an encoded representation of the structure and content of the Data Object itself, in which each of the embedded Object References is encoded by an index into the slots array.

With this arrangement, Object Reference translation can be performed by simply mapping over the slots array without having to parse or otherwise examine the potentially much more complicated and idiosyncratic body.

An Object Message is represented by a Data Object that contains all the message elements in a standardized form. For our purposes, a message consists of:

• method - a string labeling the method to be invoked on the message target
• arguments - an array of zero or more values that are the arguments to the method call. These values can be primitive data values, Object References, or more complex Data Objects that may have further Object References or Data Objects embedded within them.
• result - a singular Object Reference indicating a promise that, when settled, will constitute the result of the method invoked

A minor complication to the translation story stems from the fact that the message target (i.e., the Object to which the message is being sent) is not considered to be one of these message elements. It's not in the message per se, since a given message may pass through multiple handlers before reaching the Object that will ultimately act upon it directly. In some cases, a message may need to be processed prior to the target being known. In other cases, a given message may be reused, for example to send it to multiple recipients. The target is instead generally represented as a separate parameter when appropriate, such as when invoking a message send or message delivery operation. However, at the point we translate a message we will generally also need to translate the target Reference. Fortunately, this is not a significant problem, since the translation operation itself will be invoked from a context where the target Reference was a parameter, and therefor will be in hand at the point it requires translation.

Kernel

The Kernel interacts with three different (kinds of) entities via three different interfaces:

• Console (via the Console Daemon) to Kernel: KERNEL MANAGEMENT API
• Vats (via a Vat's Supervisor) to Kernel: SYSCALL API
• Remote Cluster (via a Channel) to Kernel: REMOTE API

In the individual method descriptions, each method will be tagged with markers indicating which interface or interfaces expose it and how:

• C - Console: the Kernel Management API
• V - Vat: the Syscall API
• R - Remote: the Remote API
• I - internal, meaning that in addition to its appearance in one or more of the regular Kernel interfaces, the method is also employed as a compositional building block in the realization of other methods
• O - Ocap, meaning the method may be made available to selected Vats as an endowment (i.e., provided to the Vat's User Code at the time the Vat is launched), in the form of a (potentially revocable) object capability that allows the method to be invoked from within the Vat to which the capability has been specifically granted

Inbound to the Kernel, messages are consumed eagerly (though subject to rate limiting to protect against attacks on availability) from the connection upon which they arrive, then queued within the Kernel itself. The Kernel maintains a separate queue for each connection to simplify prioritization of traffic handling. In general, the Console has priority over Vats which in turn have priority over Channels, while contending traffic from multiple Vats or multiple Channels are serviced round-robin. More complicated scheduling strategies are possible and might be worth exploring, but I think that at this stage in our design process we know too little about our requirements and our likely operating ecosystem to have actionable opinions about what, if any, alternatives might be desirable.

Outbound from the Kernel, messages are transmitted immediately, though subject to whatever buffering the host operating system might employ. In the fullness of time we probably need to consider how flow control might back up into actually blocking further processing on the sending end of an interface, but that seems like an advanced consideration that should not hold up engineering progress right now.

Kernel Methods

For expository purposes, the various Kernel methods are presented in functional groups. This grouping does not imply any deeper semantics.

Cluster Management

• StartCluster(storage)* [C] -- Launch a new Kernel process for a Cluster

  1. Establish a connection to Kernel's Console Daemon.
  2. Interrogate Kernel's persistent storage to determine the state of its Cluster. If the Kernel is being launched as a part of the launch of a new Cluster (indicated by absence of such state), initialize persistent state with:
     1. New Cluster ID
     2. Empty extant Vats table
     3. Empty extant Channels table
  3. If there was already a running Kernel process with this Cluster's Cluster ID:
     1. Send error feedback to the Console
     2. Exit
  4. For each extant Vat vatID:
     1. If the corresponding Vat process is already running: connect to the running Vat process
     • If not: invoke RestartVat(vatID)
  5. For each extant Channel channelID:
     1. Invoke RestartChannel(channelID)
     2. If this succeeds: resynchronize the message streams to and from each of the corresponding remote Clusters according to the Ken protocol.
     • If this fails:
       1. Invoke TerminateChannel(channelID)
       2. Send appropriate error feedback to the Console

  * Note: this not actually a method, and it is not exposed on any Kernel interface, since it cannot be invoked by sending a message to the Kernel: by definition, at the time it is initiated there is no Kernel to send a message to. Instead, it is the operation implied by launching the Kernel application at the operating system level in the first place. Parameters governing its operation are passed as command line arguments rather than as message arguments.

• StopCluster() [C] -- Shutdown all of a Cluster's processes in way that can be restarted

  1. For each extant Channel, stop that Channel
  2. For each extant Vat:
     1. Invoke StopVat() on that Vat
     2. Await ReceiveCrankCompletion resulting from the StopVat method
  3. Checkpoint Kernel state
  4. Exit

• TerminateCluster() [C] -- Permanently shutdown all of a Clusters's processes and reclaim its resources

  1. For each extant Channel channelID, invoke TerminateChannel(channelID)
  2. For each extant Vat, invoke TerminateVat() for that Vat
  3. Delete Kernel's persistent state
  4. Exit

Vat Management

• LaunchVat(isFirstTime, bundleSpec, config, endowments) [C, O] -- Create a new Vat and start it running in a process

  1. Start an instance of the Supervisor application, passing it:
     • a flag indicating that it should start a new Vat instance
     • a pathname or bundle that the Vat is to execute as its User Code
     • any additional configured initialization parameters for the Vat instance
     • any special ocaps that the party launching the Vat wishes it to have
  2. Add the newly launched Vat to the Kernel's extant Vats table

• StopVat(vatID) [C] -- Shutdown a Vat process, retaining its persistent state

  1. Invoke StopVat() on the Vat
  2. Await ReceiveCrankCompletion resulting from the StopVat method
  3. Update the Vat's run state in the extant Vats table to note that it is stopped

• UpdateVat(vatID, bundleSpec, updateConfig) [C, O] -- Replace a Vat's code without loss of persistent state

  1. Start an instance of the Supervisor application, passing it:
     • a flag indicating that it should restart an existing Vat instance
     • a pathname or bundle that the Vat is to execute as its User Code
     • any additional configured update parameters for the Vat instance
  2. Update the Vat's run state in the extant Vats table to note that it is running

• RestartVat(vatID) [C, I] -- Restart a previously shutdown Vat from its persistent state

  1. Start an instance of the Supervisor application, passing it:
     • a flag indicating that it should restart an existing Vat instance
  2. Update the Vat's run state in the extant Vats table to note that it is running

• TerminateVat(vatID) [C, I, O] -- Shutdown a Vat process and discard its persistent state

  1. Invoke StopVat() on the Vat
  2. For each Cluster the the Vat has exported remote References to: invoke DropExports([refs]) on all such References, via the corresponding Channel
  3. For each Cluster the the Vat has imported remote References to: invoke DropImports([refs]) on all such References, via the corresponding Channel
  4. Remove the Vat from the extant Vats table

Channel Management

• StartChannel(clusterSpec) [I] -- Create a Channel to some remote Cluster

  1. If there is already a Channel to the Cluster, fail
  2. Open a connection to the Cluster
  3. If this succeeds:
     1. Construct a new Channel with the connection
     2. Add the new Channel to the Kernel's extant Channels table
  • If this fails, fail

• StopChannel(channelID) [I] -- Disconnect a Channel, retaining the option to reconnect it

  1. Drop the Channel's connection
  2. Update the Channel's state in the extant Channels table to note that it is closed

• RestartChannel(channelID) [I] -- Reconnect a stopped Channel

  1. Open a connection to the Cluster
  2. If this succeeds: update the Channel's state in the extant Channels table to note that it is open
  • If this fails, fail

• TerminateChannel(channelID) [I] -- Permanently disconnect a Channel with no option to reconnect it

  1. Send a TerminateChannel message over the Channel to the remote Cluster
  2. Drop the Channel's connection
  3. For each Vat exporting References to the remote Cluster: invoke RetireExports([refs]) on all such exports from that Vat
  4. For each Vat importing References from the remote Cluster: invoke RetireImports([refs]) on all such imports from the Vat
  5. Remove the Channel from the extant Channels Table

  OPEN DESIGN QUESTION: Should these methods also be exposed on the Console interface (which in an earlier draft they were)? AGAINST: the management of Channels should be automatic and the Channels themselves should be transparent, so enabling them to be fiddled with externally seems questionable. FOR: being able to fiddle with them could be helpful in the context of development and debugging.

Code Execution

• OcapURLToObjectReference(ref) [C, O] -- Given an Object Reference, provide an equivalent Ocap URL

• ObjectReferenceToOcapURL(url) [C, O] -- Given an Ocap URL, provide the corresponding Object Reference

• DeliverObjectMessage(targetRef, message) [C, V, R] -- Deliver an Object Message to the appropriate Vat

• NotifyPromiseSettlements([[promiseRef, value], ...]) [C, V, R] -- Deliver a batch of promise resolutions/rejections to the appropriate Vat

• ReceiveCrankCompletion(status, completionRecord) [V] -- Accept the results from a Vat of processing an Object Message or promise settlement, consisting of: result status, Object Message sends, and promise settlements

• DropImports([refs]) [V, R, I] -- Note that a Vat will no longer Reference some Object

• RetireImports([refs]) [V, R, I] -- Note that a Vat will no longer recognize some Object

• DropExports([refs]) [V, R, I] -- Note that a Vat no longer contains some Object

• RetireExports([refs]) [V, R, I] -- Note that a Vat may no longer Reference some Object

  Note: While the signature of each of these methods is the same regardless of which interface it appears on, Object References (which are simply strings) contained within arguments are interpreted in the context of that interface: - C (Kernel Management API): Krefs - V, O (Syscall API): Vrefs - R (Remote API): Rrefs - I (Internal): inherited from whoever called it

Debugging and Development

• ListVats() [C] -- Provide a list of all extant Vats, with accompanying descriptive metadata
• ListChannels() [C] -- Provide a list of all extant Channels, with accompanying descriptive metadata
• DebugVat(vatID) [C] -- Command a Vat to make itself available to the/a JavaScript debugger
• DumpVatInfo(vatID) [C] -- Provide an info record about a Vat, including its c-lists and persistent state
• DumpKernelInfo() [C] -- Provide an info record about the Kernel, including its persistent state
• DumpChannelInfo(channelID) [C] -- Provide an info record about a Channel, including its c-lists

Vats

A Vat is made up of the Supervisor and the User Code. Unlike most of the other components, the Supervisor and the User Code are together regarded as a single intentional unit, the Vat. While the User Code actually does whatever job that particular Vat has been put in place to do, the User Code itself is (usually) untrusted, since it can contain unknown code coming from anywhere. The role of the Supervisor is to provide a trusted intermediary between the Kernel and the User Code.

Operation of a Vat involves three interfaces:

• Kernel to Supervisor: VAT MANAGEMENT API
• Supervisor to User Code: USER CODE INTERFACE
• User Code to Supervisor: VAT SERVICES API

The Vat Management API is how the outside world talks to a Vat. The Supervisor handles all interactions with the Kernel on behalf of its Vat. It receives System Messages from the Kernel via the Vat's Vat Management API, while it sends messages to the Kernel via the Kernel's Syscall API interface.

The User Code Interface and Vat Services API are strictly internal to the Vat.

Unlike almost all the other interfaces in this design, the User Code Interface is an asynchronous function call interface rather than a serialized data stream. The Supervisor dequeues an event from its run queue (put there by System Messages received via the Vat Management API) and dispatches it to the User Code, which then processes it to completion and returns to the Supervisor.

The Vat Services API is a synchronous function call interface that the Supervisor provides to the User Code as part of the Vat startup procedure. It is how the Supervisor provides various services to the User Code, such as the ability to send Object Messages.

All Supervisor activity aside from I/O (i.e., reading System Messages from the Kernel to Vat connection and capturing them in the Vat's run queue) happens when the Supervisor has agency, either between invocations of the User Code or during execution of one of the service functions in the Vat Services API. At any given time either the Supervisor has agency or the User Code does, but never both. This is important because while Supervisor code is running it cannot be interfered with by concurrent execution User Code. More importantly, once the User Code has completed processing an event the Supervisor can be assured that no additional User Code will run until such a time as the Supervisor chooses to allow it.

Unlike the Kernel's interfaces, the sets of methods exposed on the three Vat interfaces are disjoint. In other words, each interface presents a unique and distinct group of methods.

Kernel to Supervisor Methods: Vat Management API

• LaunchVat(isFirstTime, storage, bundleSpec, config, endowments)* -- Start an instance of the Supervisor application, passing it:

  • a flag indicating whether it is launching a new Vat instance or restarting an existing Vat instance
  • access to the persistent storage that the running Vat will use
  • a pathname or bundle that the Vat is to execute as its User Code, or a null indicating that an existing Vat instance should deploy the User Code it already has.
  • any additional configured initialization or update parameters for the Vat instance
  • any special ocaps that the party launching the Vat wishes it to have

  * Note: this is not actually a method, since it cannot be invoked by sending a message to the Vat: by definition, at the time it is initiated there is no Vat to send a message to. Instead it is the operation implied by launching the Supervisor application at the operating system level. Parameters governing its operation are passed as command line arguments rather than as message arguments.

• DeliverObjectMessage(targetRef, message) -- Deliver an Object Message to this Vat

• NotifyPromiseSettlements([[promiseRef, value], ...]) -- Deliver a batch of promise resolutions/rejections to this Vat

• StopVat() -- Shutdown this Vat's process, retaining its persistent state

• TerminateVat() -- Shutdown this Vat's process and discard its persistent state

• DumpVatInfo() -- Provide an info record about this Vat, including its persistent state

Supervisor to User Code Methods: User Code Interface

• Init(startMode, syscall, configParameters) -- Initialize the User Code. The actual initialization that will be done is specific to whatever the User Code is implementing; the interface merely specifies a standard set of parameters including:

  • a flag indicating whether this is the initial launch, an ordinary restart, or a restart after upgrade
  • a reference to the Supervisor services (syscall) object
  • configured initialization/upgrade parameters (including endowments)

• Invoke(target, args)* -- Invoke a function or method inside the User Code.

• Settle(promise, successFlag, value)* -- Resolve or reject a promise contained within the User Code's state, possibly triggering execution of then handlers registered on that promise

  * Note: the Invoke and Settle are not actually methods exposed on the User Code Interface per se. Instead, they represent the actual direct method invocation, or promise resolution/rejection on the relevant targets. This is possible because the Supervisor and the User Code execute in the same address space. Externally initiated invocations or settlements are only possible for targets that have previously been exported by the User Code by earlier outbound messages or promise settlements.

User Code to Supervisor Methods: Vat Services API

• SendObjectMessage(target, args) -- Send an Object Message to some target external to the Vat

• SettlePromise(promise, successFlag, value) -- Notify the world outside the Vat of the settlement of some previously exported promise

• Exit(status) -- Permanently shutdown this Vat

• Persistent data store methods TBD

  In all cases the actions initiated by User Code invocations of Vat Services API methods do not take place immediately. Instead, they are accumulated into the crank completion record for whatever crank the Vat is currently executing, and will be acted on collectively when the User Code finishes and surrenders agency to the Supervisor. At that time the actions will be executed by the Supervisor or added to the completion record that will be relayed to the Kernel via the Kernel's ReceiveCrankCompletion method as part of executing the ken protocol on the Vat's behalf.

Console

As described above, the Console consists of two components, the Console Daemon and the Console UI. In terms of the overall system design, we have little to say in this section about the Console UI aside from the interface that the Console Daemon exposes to it. This is not to say the Console UI design is not vitally important, but in considering its design we are principally concerned with usability rather than fundamental semantics. Moreover, as previously explained, we expect there to be multiple flavors of Console UI for different kinds of users and for different purposes, so even in a mature system there's no definitive design to specify.

The Console Daemon exists mainly to mediate access to the Cluster in terms of Petnames rather than raw, low-level Object References. It runs as a separate process so that the mapping table can be managed independently from the comings and goings of Console UI instances. OPEN DESIGN QUESTION: Might the Console Daemon be better conceptualized not as a background process but as a Kernel interface library (to be linked directly into individual Console UI implementations) that accesses a shared Petname table database? This might simplify some things, though it would require adopting a database that supports concurrent access by multiple readers and writers. I have an intuition that this latter approach is a potential footgun, and in particular is risky from a security perspective, but I don't at this time feel like I have an articulable justification for that intuition. However, for the time being I'm going to proceed with the daemon process approach until we have a chance to discuss the question more deeply.

The Console Daemon nominally presents two interfaces, but really only one:

• Console UI to Console Daemon CONSOLE SERVICES INTERFACE
• Kernel to Console Daemon

The Console Services Interface is essentially a tamed version of the Kernel Management API, exposing those Kernel methods that are suitable for direct manipulation by a user in form adapted to such use.

The Kernel to Console Daemon interface isn't really an interface in the sense of all the other interfaces we're defining here. There are no actual methods, as all information that flows from the Kernel to the Console does so in the context of responses to Kernel Management API methods rather than the Kernel initiating an operation itself -- in principle the Kernel should be able run on its own for extended periods of time without any Console attached, so it only speaks to the Console when the Console speaks to it.

Console UI to Console Daemon Methods: Console Services Interface

Essentially all of the methods described above for the Kernel that are tagged with C are replicated on the Console Services Interface. The key difference is that Object References (as found, for example, in message deliveries) are expressed in terms of Petnames.

Note that most of these Kernel methods are quite inappropriate for direct exposure to human users, but are instead made available so that the Console UI can implement higher-level operations that are appropriate to give to people directly (without taking a position as to what those appropriate operations might or ought to be). The exception is debugging and development tools, which might provide affordances for doing all kinds of things manually that normally would only be done by code. However, since these are both powerful and potentially dangerous, the availability of such affordances should be gated by security checkpoints or at least buried deep in some menu, so that unsophisticated users (which is to say, most users) don't get entangled with them by accident when they don't know what they're doing, just as you wouldn't provide an ordinary user with a hex debugger that lets them poke directly into memory.

The remaining methods that are unique to the Console Services Interface are concerned with managing the Console's Petname table, which maps between user-provided Petnames and Krefs.

• NameObject -- Assign a Petname (and optionally description) to an Object. Note that a given Object can have more than one Petname.
• RenameObject -- Change the Petname of an Object to a different string
• RemovePetname -- Remove an entry from the Petname table. Note that this does not remove the Object itself, simply makes it inaccessible to direct manipulation via the name being dropped.
• ListPetnames -- Return a list of the currently extant Petnames (and optionally the descriptions that go with them)

[WIP todo notes]

get straight on import/export drop/retire logic

ken protocol - what and how

object reference representation details

Channel and inter-Cluster connection logic

handling promises and eventual send

persistent storage services for kernel and vat

three party handoff - priority? necessary? threat or menace?