Skip to content

Commit

Permalink
Squashed 'crates/service-protocol/service-protocol/' changes from 29b…
Browse files Browse the repository at this point in the history 
…28f98..a4a8596e

a4a8596e Now side effect is part of the protocol (restatedev#87)
494f765c Format
fec17c9e Change x-restate-user-agent to x-restate-server, to be similar to https://www.rfc-editor.org/rfc/rfc9110#field.server
98bd326c Entry names + specify the entry that caused the failure in ErrorMessage (restatedev#86)
c47e65b0 Describe x-restate-user-agent (restatedev#85)
d43de04c Describe side effects in protocol (restatedev#84)

git-subtree-dir: crates/service-protocol/service-protocol
git-subtree-split: a4a8596eab8c554dce0f8f0f1020d5ddc623a305
  • Loading branch information
@slinkydeveloper
slinkydeveloper committed Apr 16, 2024
1 parent 4ad4622 commit e72ee23
Show file tree
Hide file tree
Showing 2 changed files with 112 additions and 22 deletions.
70 changes: 65 additions & 5 deletions dev/restate/service/protocol.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,14 @@ message ErrorMessage {
string message = 2;
// Contains a verbose error description, e.g. the exception stacktrace.
string description = 3;

// Entry that caused the failure. This may be outside the current stored journal size.
// If no specific entry caused the failure, the current replayed/processed entry can be used.
uint32 related_entry_index = 4;
// Name of the entry that caused the failure. Empty if no name was set.
string related_entry_name = 5;
// Entry type. 0 if unknown.
uint32 related_entry_type = 6;
}

// Type: 0x0000 + 4
Expand All @@ -90,13 +98,16 @@ message EndMessage {
// --- Journal Entries ---

// Every Completable JournalEntry has a result field, filled only and only if the entry is in DONE state.
// Depending on the semantics of the corresponding syscall, the entry can represent the result field with any of these three types:
//
// For every journal entry, fields 12, 13, 14 and 15 are reserved.
//
// The field 12 is used for name. The name is used by introspection/observability tools.
//
// Depending on the semantics of the corresponding syscall, the entry can represent the completion result field with any of these three types:
//
// * google.protobuf.Empty empty = 13 for cases when we need to propagate to user code the distinction between default value or no value.
// * bytes value = 14 for carrying the result value
// * Failure failure = 15 for carrying a failure
//
// The tag numbers 13, 14 and 15 are reserved and shouldn't be used for other fields.

// ------ Input and output ------

Expand All @@ -107,6 +118,9 @@ message InputEntryMessage {
repeated Header headers = 1;

bytes value = 14;

// Entry name
string name = 12;
}

// Completable: No
Expand All @@ -117,6 +131,9 @@ message OutputEntryMessage {
bytes value = 14;
Failure failure = 15;
};

// Entry name
string name = 12;
}

// ------ State access ------
Expand All @@ -132,6 +149,9 @@ message GetStateEntryMessage {
bytes value = 14;
Failure failure = 15;
};

// Entry name
string name = 12;
}

// Completable: No
Expand All @@ -140,19 +160,27 @@ message GetStateEntryMessage {
message SetStateEntryMessage {
bytes key = 1;
bytes value = 3;

// Entry name
string name = 12;
}

// Completable: No
// Fallible: No
// Type: 0x0800 + 2
message ClearStateEntryMessage {
bytes key = 1;

// Entry name
string name = 12;
}

// Completable: No
// Fallible: No
// Type: 0x0800 + 3
message ClearAllStateEntryMessage {
// Entry name
string name = 12;
}

// Completable: Yes
Expand All @@ -167,6 +195,9 @@ message GetStateKeysEntryMessage {
StateKeys value = 14;
Failure failure = 15;
};

// Entry name
string name = 12;
}

// ------ Syscalls ------
Expand All @@ -183,6 +214,9 @@ message SleepEntryMessage {
Empty empty = 13;
Failure failure = 15;
}

// Entry name
string name = 12;
}

// Completable: Yes
Expand All @@ -203,6 +237,9 @@ message InvokeEntryMessage {
bytes value = 14;
Failure failure = 15;
};

// Entry name
string name = 12;
}

// Completable: No
Expand All @@ -224,6 +261,9 @@ message BackgroundInvokeEntryMessage {

// If this invocation has a key associated (e.g. for objects and workflows), then this key is filled in. Empty otherwise.
string key = 6;

// Entry name
string name = 12;
}

// Completable: Yes
Expand All @@ -235,6 +275,9 @@ message AwakeableEntryMessage {
bytes value = 14;
Failure failure = 15;
};

// Entry name
string name = 12;
}

// Completable: No
Expand All @@ -245,9 +288,26 @@ message CompleteAwakeableEntryMessage {
string id = 1;

oneof result {
bytes value = 5;
Failure failure = 6;
bytes value = 14;
Failure failure = 15;
};

// Entry name
string name = 12;
}

// Completable: No
// Fallible: No
// Type: 0x0C00 + 5
// Flag: RequiresRuntimeAck
message SideEffectEntryMessage {
oneof result {
bytes value = 14;
dev.restate.service.protocol.Failure failure = 15;
};

// Entry name
string name = 12;
}

// --- Nested messages
Expand Down
64 changes: 47 additions & 17 deletions service-invocation-protocol.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ The system is composed of two actors:
- SDK, which contains the implementation of the Restate Protocol
- User business logic, which interacts with the SDK to access Restate system calls (or syscalls)

Each service method invocation is modeled by the protocol as a state machine, where state transitions can be caused
either by user code or by _Runtime events_.
Each invocation is modeled by the protocol as a state machine, where state transitions can be caused either by user code
or by _Runtime events_.

Every state transition is logged in the _Invocation journal_, used to implement Restate's durable execution model. The
journal is also used to suspend an invocation and resume it at a later point in time. The _Invocation journal_ is
Expand All @@ -32,11 +32,12 @@ The state machine is summarized in the following diagram:
```mermaid
sequenceDiagram
Note over Runtime,SDK: Start
Runtime->>SDK: HTTP Request to /invoke/{service}/{method}
Runtime->>SDK: HTTP Request to /invoke/{service}/{handler}
Runtime->>SDK: StartMessage
Note over Runtime,SDK: Replaying
Runtime->>SDK: [...]EntryMessage(s)
Note over Runtime,SDK: Processing
SDK->>Runtime: HTTP Response headers
loop
SDK->>Runtime: [...]EntryMessage
Runtime->>SDK: CompletionMessage and/or EntryAckMessage
Expand Down Expand Up @@ -102,9 +103,9 @@ protocol mandates the following messages:

### Message stream

In order to execute a service method invocation, service deployment and restate Runtime open a single stream between the
runtime and the service deployment. Given 10 concurrent service method invocations to a service deployment, there are 10
concurrent streams, each of them mapping to a specific invocation.
In order to execute an invocation, service deployment and restate Runtime open a single stream between the runtime and
the service deployment. Given 10 concurrent invocations to a service deployment, there are 10 concurrent streams, each
of them mapping to a specific invocation.

Every unit of the stream contains a Message serialized using the
[Protobuf encoding](https://protobuf.dev/programming-guides/encoding/), using the definitions in
Expand All @@ -119,10 +120,23 @@ in two modes:
runtime. Once the service deployment starts sending messages to the runtime, the runtime cannot send messages anymore
back to the service deployment.

When opening the stream, the request method MUST be `POST` and the request path MUST have the following format:
A message stream MUST start with `StartMessage` and MUST end with either:

- One [`SuspensionMessage`](#suspension)
- One [`ErrorMessage`](#failures)
- One `EndMessage`

If the message stream does not end with any of these two messages, it will be considered equivalent to sending an
`ErrorMessage` with an [unknown failure](#failures).

The `EndMessage` marks the end of the invocation lifecycle, that is the end of the journal.

### Initiating the stream

When opening the stream, the HTTP request method MUST be `POST` and the request path MUST have the following format:

```
/invoke/{fullyQualifiedServiceName}/{methodName}
/invoke/{serviceName}/{handlerName}
```

For example:
Expand All @@ -133,19 +147,24 @@ For example:

An arbitrary path MAY prepend the aforementioned path format.

In case the path format is not respected, or `fullyQualifiedServiceName` or `methodName` is unknown, the SDK MUST close
the stream replying back with a `404` status code.
In case the path format is not respected, or `serviceName` or `handlerName` is unknown, the SDK MUST close the stream
replying back with a `404` status code.

A message stream MUST start with `StartMessage` and MUST end with either:
In case the invocation is accepted, `200` status code MUST be returned.

- One [`SuspensionMessage`](#suspension)
- One [`ErrorMessage`](#failures)
- One `EndMessage`
Additionally, the header `x-restate-server` MAY be sent back, with the following format:

If the message stream does not end with any of these two messages, it will be considered equivalent to sending an
`ErrorMessage` with an [unknown failure](#failures).
```http request
x-restate-server: <sdk-name> / <sdk-version>
```

The `EndMessage` marks the end of the invocation lifecycle, that is the end of the journal.
E.g.:

```http request
x-restate-server: restate-sdk-java/0.8.0
```

This header is used for observability purposes by the Restate observability tools.

### Message header

Expand Down Expand Up @@ -275,6 +294,11 @@ index of the corresponding entry.
| Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

#### Entry names

Every Journal entry has a field `string name = 12`, which can be set by the SDK when recording the entry. This field is
used for observability purposes by Restate observability tools.

### Journal entries reference

The following tables describe the currently available journal entries. For more details, check the protobuf message
Expand All @@ -294,6 +318,7 @@ descriptions in [`protocol.proto`](dev/restate/service/protocol.proto).
| `SetStateEntryMessage` | `0x0800` | No | No | Set the value of a service instance state key. |
| `ClearStateEntryMessage` | `0x0801` | No | No | Clear the value of a service instance state key. |
| `ClearAllStateEntryMessage` | `0x0802` | No | No | Clear all the values of the service instance state. |
| `SideEffectEntryMessage` | `0x0C05` | No | No | Run non-deterministic user provided code and persist the result. |

#### Awakeable identifier

Expand Down Expand Up @@ -365,6 +390,11 @@ additional features to the users.
The protocol allows the SDK to register an arbitrary entry type within the journal. The type MUST be `>= 0xFC00`. The
runtime will treat this entry as any other entry, persisting it and sending it during replay in the correct order.

Custom entries MAY have the entry name field `12`, as described in [entry names](#entry-names).

The field numbers 13, 14 and 15 MUST not be used, as they're reserved for completable journal entries, as described in
[completable journal entries](#completable-journal-entries-and-completionmessage).

**Header**

0 1 2 3
Expand Down

0 comments on commit e72ee23

Please sign in to comment.