Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,6 @@ cargo test -- --nocapture

# Check for linting issues
cargo clippy -- -W warnings

# When adding a new package to the workspace
deno run -A scripts/cargo/update_workspace.ts
```

### Docker Development Environment
Expand Down
Empty file added docs/engine/ACTOR_KV.md
Empty file.
111 changes: 111 additions & 0 deletions docs/engine/ACTOR_LIFECYCLE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Actor Lifecycle Flow Diagram

```mermaid
---
config:
theme: mc
look: classic
---
sequenceDiagram
participant A as API
participant U as User
participant G as Gateway

participant R as Runner
participant RWS as Runner WS
participant RWF as Runner Workflow
participant AWF as Actor Workflow

critical runner connection
R->>RWS: Connect
RWS->>RWF: Create runner workflow
end

critical actor creation
U->>A: POST /actors

A->>AWF: Create actor workflow
A->>U:
end

critical initial request
U->>G: Request to actor
note over G: Await actor ready

critical actor allocation
note over AWF: Allocate

AWF->>RWF: Send StartActor

RWF->>RWS:
RWS->>R:
note over R: Start actor
R->>RWS: Actor state update: Running
RWS->>RWF:
RWF->>AWF:
note over AWF: Publish Ready msg
end
AWF->>G: Receive runner ID

G->>RWS: Tunnel ToClientRequestStart
RWS->>R:
note over R: Handle request
R->>RWS: ToServerResponseStart
RWS->>G:
G->>U:
end

critical second request
U->>G: Request to actor
note over G: Actor already connectable
G->>RWS: Tunnel ToClientRequestStart
RWS->>R:
note over R: Handle request
R->>RWS: ToServerResponseStart
RWS->>G:
G->>U:
end

note over A, AWF: Time passes

critical actor sleep
R->>RWS: Actor intent: Sleep
RWS->>RWF:
RWF->>AWF:
note over AWF: Mark as sleeping
AWF->>RWF: Send StopActor
RWF->>RWS:
RWS->>R:
note over R: Stop actor
R->>RWS: Actor state update: Stopped
RWS->>RWF:
RWF->>AWF:
note over AWF: Sleep
end

critical request to sleeping actor
U->>G: Request to actor
note over G: Actor sleeping
G->>AWF: Wake
note over G: Await actor ready
critical actor allocation
note over AWF: Allocate
AWF->>RWF: Send StartActor
RWF->>RWS:
RWS->>R:
note over R: Start actor
R->>RWS: Actor state update: Running
RWS->>RWF:
RWF->>AWF:
note over AWF: Publish Ready msg
end
AWF->>G: Receive runner ID

G->>RWS: Tunnel ToClientRequestStart
RWS->>R:
note over R: Handle request
R->>RWS: ToServerResponseStart
RWS->>G:
G->>U:
end
```
Empty file added docs/engine/GASOLINE.md
Empty file.
Empty file added docs/engine/GATEWAY.md
Empty file.
41 changes: 41 additions & 0 deletions docs/engine/GUARD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Rivet Guard

Guard facilitates HTTP communication between the public internet and various internal rivet services, as well as tunnelling connections to actors.

## Routing

Guard uses request path and/or headers to route requests.

### Actors

Guard routes requests to actors when:
- the path matches `/gateway/{actor_id}/{...path}`
- the path matches `/gateway/{actor_id}@{token}/{...path}`
- when connecting a websocket, `Sec-Websocket-Protocol` consists of comma delimited dot separated pairs like `rivet_target.actor,rivet_actor.{actor_id}`
- otherwise, when the `X-Rivet-Target` header is set to `actor` and `X-Rivet-Actor` header is set to the actor id

### Runners

Guard accepts runner websocket connections when:
- the path matches `/runners/connect`
- `Sec-Websocket-Protocol` consists of comma delimited dot separated pairs like `rivet_target.runner`

### API Requests

Guard routes requests to the API layer when the `X-Rivet-Target` header is set to `api-public` or is unset.

## Proxying

Guard acts as a proxy for requests and websockets to actors:

- Internally, the websocket connects to a websocket listener running on the Rivet Engine
- Rivet Engine transmits HTTP requests and websocket messages via the runner protocol to the actor's corresponding runner's websocket
- The runner has a single websocket connection open to Guard which is independent from any client websocket connection
- This single connection multiplexes all actor requests and websocket connections
- The runner delegates requests and websockets to actors
- The runner sends HTTP responses and websocket messages back to Rivet through is websocket via the runner protocol
- Rivet transforms the runner protocol messages into HTTP responses and websocket messages

### Websocket Hibernation

This proxy/tunnel approach allows us to implement hibernatable websockets (see HIBERNATING_WS.md) for actors. We can keep a client's websocket connection open while simultaneously allowing for actors to sleep, resulting in 0 usage when there is no traffic over the websocket. The actor is automatically awoken when a websocket message is transmitted to Guard.
25 changes: 25 additions & 0 deletions docs/engine/HIBERNATING_WS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Hibernating Websockets

## Lifecycle

1. Client establishes a websocket connection to an actor via Rivet, which is managed by Guard (see GUARD.md)
2. Guard checks to see if the actor is awake. If it is, skip step 3
3. If the actor is not awake, send a Wake signal to its workflow. This will make the actor allocate to an existing runner, or in the case of serverless, start a new runner and allocate to that
4. Guard sends the runner a ToClientWebSocketOpen message a via the runner protocol
5. The runner sends back ToServerWebSocketOpen to acknowledge the connection
- The runner must set `.canHibernate = true` for hibernation to work
6. At this point the websocket connection is fully established and any websocket messages sent by the client are proxied through Guard to the runner to be delegated to the actor
7. Should the actor go to sleep, the runner will close the websocket by sending ToServerWebSocketClose with `.hibernate = true` via the runner protocol
8. Guard receives that the websocket has closed on the runner side and starts hibernating. During hibernation nothing happens.
9.
- If the actor is awoken from any other source, go to step 6. We do not send a ToClientWebSocketOpen message in this case
- If the client sends a websocket message during websocket hibernation, go to step 2
- If the client closes the websocket, the actor is rewoken (if not already running) and sent a ToClientWebSocketClose

## State

To facilitate state management on the runner side (specifically via RivetKit), each hibernating websocket runs a keepalive loop which periodically stores a value to UDB marking it as active.

When a client websocket closes during hibernation, this value is cleared.

When a runner receives a CommandStartActor message via the runner protocol, it contains information about which hiberating requests are still active.
17 changes: 17 additions & 0 deletions docs/engine/RUNNER_SHUTDOWN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Runner Shutdown

## 1. Self initiated shutdown

1. Runner sends ToServerStopping to runner WS
2. Runner WS proxies ToServerStopping to runner WF
3. Runner WF sets itself as "draining", preventing future actor allocations to it
4. Runner WF sends GoingAway signal to all actor WFs
5. Once the runner lost threshold is passed, runner WF sends ToClientClose to runner WS
6. Runner WS closes connection to runner, informing it not to attempt reconnection

## 2. Rivet initiated shutdown

1. Runner WF receives Stop signal
2. Runner WF sends GoingAway signal to all actor WFs
3. Once the runner lost threshold is passed, runner WF sends ToClientClose to runner WS
4. Runner WS closes connection to runner, informing it not to attempt reconnection
Loading
Loading