Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add stream rollup section #610

Merged
merged 5 commits into from
Sep 6, 2023
Merged

Add stream rollup section #610

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f13ae0c
Add stream rollup section
bruth Aug 23, 2023
0034ea7
Add stream message headers
bruth Aug 23, 2023
a2a9dff
Move headers to separate page
bruth Aug 23, 2023
67642b9
Add headers reference for JetStream
bruth Aug 23, 2023
56831e1
Update headers.md
bruth Sep 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions SUMMARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
- [Key/Value Store Walkthrough](nats-concepts/jetstream/key-value-store/kv_walkthrough.md)
- [Object Store](nats-concepts/jetstream/object-store/obj_store.md)
- [Object Store Walkthrough](nats-concepts/jetstream/object-store/obj_walkthrough.md)
- [Headers](nats-concepts/jetstream/headers.md)
- [Subject Mapping and Partitioning](nats-concepts/subject_mapping.md)
- [NATS Service Infrastructure](nats-concepts/service_infrastructure.md)
- [NATS Adaptive Deployment Architectures](nats-concepts/adaptive_edge_deployment.md)
Expand Down
55 changes: 55 additions & 0 deletions nats-concepts/jetstream/headers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Headers

Message headers are used in a variety of JetStream contexts, such de-duplication, auto-purging of messages, metadata from republished messages, and more.

## Publish

Headers that can be set by a client when a message being published.

| Name | Description | Example | Version |
| :------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------- | :------ |
| `Nats-Msg-Id` | Client-defined unique identifier for a message that will be used by the server apply de-duplication within the configured `Duplicate Window`. | `9f01ccf0-8c34-4789-8688-231a2538a98b` | 2.2.0 |
| `Nats-Expected-Stream` | Used to assert the published message is received by some expected stream. | `my-stream` | 2.2.0 |
| `Nats-Expected-Last-Msg-Id` | Used to apply optimistic concurrency control at the stream-level. The value is the last expected `Nats-Msg-Id` and the server will reject a publish if the current ID does not match. | `9f01ccf0-8c34-4789-8688-231a2538a98b` | 2.2.0 |
| `Nats-Expected-Last-Sequence` | Used to apply optimistic concurrency control at the stream-level. The value is the last expected sequence and the server will reject a publish if the current sequence does not match. | `328` | 2.2.0 |
| `Nats-Expected-Last-Subject-Sequence` | Used to apply optimistic concurrency control at the subject-level. The value is the last expected sequence and the server will reject a publish if the current sequence does not match for the message's subject. | `38` | 2.3.1 |
| `Nats-Rollup` | Used to apply a purge of all prior messages in the stream or at the subject-level. | `all` for stream, `sub` for subject | 2.6.2 |

## RePublish

Headers set messages that are republished.

| Name | Description | Example | Version |
| :------------------- | :--------------------------------------------------------------------------------------------------------------------- | :---------------------------- | :------ |
| `Nats-Stream` | Name of the stream the message was republished from. | `Nats-Stream: my-stream` | 2.8.3 |
| `Nats-Subject` | The original subject of the message. | `events.mouse_clicked` | 2.8.3 |
| `Nats-Time-Stamp` | The original timestamp of the message. | `2023-08-23T19:53:05.762416Z` | 2.8.3 |
| `Nats-Sequence` | The original sequence of the message. | `193` | 2.8.3 |
| `Nats-Last-Sequence` | The last sequence of the message having the same subject, otherwise zero if this is the first message for the subject. | `190` | 2.8.3 |

## Sources

Headers that are implicitly added to messages sourced from other streams.

| Name | Description | Example | Version |
| :------------------- | :---------------------------------------------------------- | :---------- | :------ |
| `Nats-Stream-Source` | Specifies the origin stream name, the subject and the sequence number plus the subject filter and destination transform of the message being sourced. | `my-stream` | 2.2.0 |

## Headers-only

Headers added to messages when the consumer is configured to be "headers only" omitting the body.

| Name | Description | Example | Version |
| :-------------- | :----------------------------------- | :------ | :------ |
| `Nats-Msg-Size` | Indicates the message size in bytes. | `1024` | 2.6.2 |

## Mirror

Headers used for internal flow-control messages for a mirror.

| Name | Description | Example | Version |
| :---------------------- | :---------- | :------ | :------ |
| `Nats-Last-Consumer` | | | 2.2.1 |
| `Nats-Last-Stream` | | | 2.2.1 |
| `Nats-Consumer-Stalled` | | | 2.4.0 |
| `Nats-Response-Type` | | | 2.6.4 |
19 changes: 10 additions & 9 deletions nats-concepts/jetstream/streams.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Below are the set of stream configuration options that can be defined. The `Vers
| NoAck | Disables acknowledging messages that are received by the Stream | 2.2.0 | Yes |
| [Retention](#retentionpolicy) | Declares the retention policy for the stream. | 2.2.0 | No |
| [Discard](#discardpolicy) | The behavior of discarding messages when any streams' limits have been reached. | 2.2.0 | Yes |
| Duplicates | The window within which to track duplicate messages, expressed in nanoseconds. | 2.2.0 | Yes |
| Duplicate Window | The window within which to track duplicate messages, expressed in nanoseconds. | 2.2.0 | Yes |
| [Placement](#placement) | Used to declare where the stream should be placed via tags and/or an explicit cluster name. | 2.2.0 | Yes |
| [Mirror](#mirror) | If set, indicates this stream is a mirror of another stream. | 2.2.0 | No (if defined) |
| [Sources](#sources) | If defined, declares one or more streams this stream will source messages from. | 2.2.0 | Yes |
Expand All @@ -41,7 +41,7 @@ Below are the set of stream configuration options that can be defined. The `Vers
| Sealed | Sealed streams do not allow messages to be deleted via limits or API, sealed streams can not be unsealed via configuration update. Can only be set on already created streams via the Update API. | 2.6.2 | Yes (once) |
| DenyDelete | Restricts the ability to delete messages from a stream via the API. | 2.6.2 | No |
| DenyPurge | Restricts the ability to purge messages from a stream via the API. | 2.6.2 | No |
| AllowRollup | Allows the use of the `Nats-Rollup` header to replace all contents of a stream, or subject in a stream, with a single new message. | 2.6.2 | Yes |
| [AllowRollup](#allowrollup) | Allows the use of the `Nats-Rollup` header to replace all contents of a stream, or subject in a stream, with a single new message. | 2.6.2 | Yes |
| [RePublish](#republish) | If set, messages stored to the stream will be immediately _republished_ to the configured subject. | 2.8.3 | No (if defined) |
| AllowDirect | If true, and the stream has more than one replica, each replica will respond to _direct get_ requests for individual messages, not only the leader. | 2.9.0 | Yes |
| MirrorDirect | If true, and the stream is a mirror, the mirror will participate in a serving _direct get_ requests for individual messages from origin stream. | 2.9.0 | Yes |
Expand Down Expand Up @@ -166,6 +166,14 @@ One functional difference from a mirror is that a stream with sources defined ca

The fields per source stream are the same as defined in mirror above.

### AllowRollup

If enabled, the `AllowRollup` stream option allows for a published message having a `Nats-Rollup` header indicating all prior messages should be purged. The scope of the _purge_ is defined by the header value, either `all` or `sub`.

The `Nats-Rollup: all` header will purge all prior messages in the stream. Whereas the `sub` value will purge all prior messages for a given subject.

A common use case for rollup is for state snapshots, where the message being published has accumulated all the necessary state from the prior messages, relative to the stream or a particular subject.

### RePublish

If enabled, the `RePublish` stream option will result in the server re-publishing messages received into a stream automatically and immediately after a succesful write, to a distinct destination subject.
Expand All @@ -177,10 +185,3 @@ The fields for configuring republish include:
- `Source` - An optional subject pattern which is a subset of the subjects bound to the stream. It defaults to all messages in the stream, e.g. `>`.
- `Destination` - The destination subject messages will be re-published to. The source and destination must be a valid [subject mapping](../../nats-concepts/subject_mapping.md).
- `HeadersOnly` - If true, the message data will not be included in the re-published message, only an additional header `Nats-Msg-Size` indicating the size of the message in bytes.

Every message that is republished will have a set of headers set providing metadata about the source:

- `Nats-Stream` - name of the stream the message was republished from.
- `Nats-Subject` - The original subject of the message
- `Nats-Sequence` - The original sequence of the message in the stream
- `Nats-Last-Sequence` - The last sequence of the message having the same subject, otherwise zero if this is the first message for the subject.