From 4ce71cf7db73510e8958d4e897af13054a0dc855 Mon Sep 17 00:00:00 2001 From: Tomasz Pietrek Date: Sat, 21 Jan 2023 12:38:23 +0100 Subject: [PATCH 1/6] Add metadata ADR proposal Signed-off-by: Tomasz Pietrek --- adr/ADR-00.md | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 adr/ADR-00.md diff --git a/adr/ADR-00.md b/adr/ADR-00.md new file mode 100644 index 00000000..db91f126 --- /dev/null +++ b/adr/ADR-00.md @@ -0,0 +1,41 @@ +# Metadata for Stream and Consumer + +|Metadata|Value| +|--------|-----| +|Date |2023-01-23| +|Author |@Jarema| +|Status |Approved| +|Tags |jetstream, client, server| + +## Context and Problem Statement + +Until now, there was no way to easily add additional information about Stream or Consumer. +The only solution was using `Description` field, which is a ugly workaround. + +## Server PR +https://github.com/nats-io/nats-server/pull/3797 + +## Design + +The solution is to add new `metadata` field to both `Consumer` and `Stream` config. +The `metadata` field would by a map of `string` keys and `string` values. + +The map would be represented in json as object with nested key/value pairs, which is a default +way to marshal maps/hashmaps in most languages. + +### Example +```json +{ + "durable_name": "consumer", + ... // other consumer/stream fields + "metadata": { + "owner": "nack", + "domain": "product" + } +} + +``` + +## Open questions + +Do we want to limit the size of `metadata`? From e3299b024f897b7c0d7463b58b0aa704786726be Mon Sep 17 00:00:00 2001 From: Tomasz Pietrek Date: Mon, 23 Jan 2023 14:13:29 +0100 Subject: [PATCH 2/6] Declare max size of metadata Signed-off-by: Tomasz Pietrek --- adr/ADR-00.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/adr/ADR-00.md b/adr/ADR-00.md index db91f126..1be39132 100644 --- a/adr/ADR-00.md +++ b/adr/ADR-00.md @@ -4,13 +4,13 @@ |--------|-----| |Date |2023-01-23| |Author |@Jarema| -|Status |Approved| +|Status |Proposed| |Tags |jetstream, client, server| ## Context and Problem Statement Until now, there was no way to easily add additional information about Stream or Consumer. -The only solution was using `Description` field, which is a ugly workaround. +The only solution was using `Description` field, which is a not ergonomic workaround. ## Server PR https://github.com/nats-io/nats-server/pull/3797 @@ -23,6 +23,9 @@ The `metadata` field would by a map of `string` keys and `string` values. The map would be represented in json as object with nested key/value pairs, which is a default way to marshal maps/hashmaps in most languages. +To avoid abuse of the metadata, the size of it will be limited to 1MB. +Size will be counter for lenght of keys + values. + ### Example ```json { @@ -36,6 +39,4 @@ way to marshal maps/hashmaps in most languages. ``` -## Open questions -Do we want to limit the size of `metadata`? From 97c853ee02bf52c5583722f8d13ecc073d682598 Mon Sep 17 00:00:00 2001 From: Tomasz Pietrek Date: Mon, 23 Jan 2023 20:27:03 +0100 Subject: [PATCH 3/6] Improve ADR Signed-off-by: Tomasz Pietrek --- adr/ADR-00.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/ADR-00.md b/adr/ADR-00.md index 1be39132..0469bccb 100644 --- a/adr/ADR-00.md +++ b/adr/ADR-00.md @@ -23,7 +23,7 @@ The `metadata` field would by a map of `string` keys and `string` values. The map would be represented in json as object with nested key/value pairs, which is a default way to marshal maps/hashmaps in most languages. -To avoid abuse of the metadata, the size of it will be limited to 1MB. +To avoid abuse of the metadata, the size of it will be limited to 128kB. Size will be counter for lenght of keys + values. ### Example From 8edcebc66b19ff607f4743d8f26bd9b3178f92e6 Mon Sep 17 00:00:00 2001 From: Tomasz Pietrek Date: Mon, 23 Jan 2023 22:55:28 +0100 Subject: [PATCH 4/6] Update the ADR Signed-off-by: Tomasz Pietrek --- adr/ADR-00.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/adr/ADR-00.md b/adr/ADR-00.md index 0469bccb..41e4e5b8 100644 --- a/adr/ADR-00.md +++ b/adr/ADR-00.md @@ -20,11 +20,19 @@ https://github.com/nats-io/nats-server/pull/3797 The solution is to add new `metadata` field to both `Consumer` and `Stream` config. The `metadata` field would by a map of `string` keys and `string` values. +### JSON representation The map would be represented in json as object with nested key/value pairs, which is a default way to marshal maps/hashmaps in most languages. -To avoid abuse of the metadata, the size of it will be limited to 128kB. -Size will be counter for lenght of keys + values. +### Size limit +To avoid abuse of the metadata, the size of it is limited to 128kB. +Size is equal to len of all keys and values summed. + +### Reserved prefix +`_nats` is a reserved prefix. +Will be used for any potential internals of server or clients. +Server can lock its metadata to be immutable and deny any changes. + ### Example ```json @@ -33,7 +41,8 @@ Size will be counter for lenght of keys + values. ... // other consumer/stream fields "metadata": { "owner": "nack", - "domain": "product" + "domain": "product", + "_nats_created_version": "1.10.0" } } From 6ee4278e1594e84bab405a12ebc2f5c318123bce Mon Sep 17 00:00:00 2001 From: Tomasz Pietrek Date: Tue, 24 Jan 2023 12:17:09 +0100 Subject: [PATCH 5/6] Update ADR Signed-off-by: Tomasz Pietrek --- adr/ADR-00.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/adr/ADR-00.md b/adr/ADR-00.md index 41e4e5b8..b230149e 100644 --- a/adr/ADR-00.md +++ b/adr/ADR-00.md @@ -4,7 +4,7 @@ |--------|-----| |Date |2023-01-23| |Author |@Jarema| -|Status |Proposed| +|Status |Approved| |Tags |jetstream, client, server| ## Context and Problem Statement @@ -25,7 +25,7 @@ The map would be represented in json as object with nested key/value pairs, whic way to marshal maps/hashmaps in most languages. ### Size limit -To avoid abuse of the metadata, the size of it is limited to 128kB. +To avoid abuse of the metadata, the size of it is limited to 128KB. Size is equal to len of all keys and values summed. ### Reserved prefix From 5af48a9630cf72e0d26cfb46d8b0de413ad68d1d Mon Sep 17 00:00:00 2001 From: Tomasz Pietrek Date: Tue, 24 Jan 2023 21:55:18 +0100 Subject: [PATCH 6/6] Generate Readme.md Signed-off-by: Tomasz Pietrek --- README.md | 33 ++++++++++++++++++--------------- adr/{ADR-00.md => ADR-33.md} | 0 2 files changed, 18 insertions(+), 15 deletions(-) rename adr/{ADR-00.md => ADR-33.md} (100%) diff --git a/README.md b/README.md index 8412a047..c9d9965c 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,7 @@ This repo is used to capture architectural and design decisions as a reference o |[ADR-22](adr/ADR-22.md)|jetstream, client|JetStream Publish Retries on No Responders| |[ADR-31](adr/ADR-31.md)|jetstream, client, server|JetStream Direct Get| |[ADR-32](adr/ADR-32.md)|client|Service API| +|[ADR-33](adr/ADR-33.md)|jetstream, client, server|Metadata for Stream and Consumer| ## Jetstream @@ -50,6 +51,7 @@ This repo is used to capture architectural and design decisions as a reference o |[ADR-22](adr/ADR-22.md)|jetstream, client|JetStream Publish Retries on No Responders| |[ADR-28](adr/ADR-28.md)|jetstream, server|JetStream RePublish| |[ADR-31](adr/ADR-31.md)|jetstream, client, server|JetStream Direct Get| +|[ADR-33](adr/ADR-33.md)|jetstream, client, server|Metadata for Stream and Consumer| ## Kv @@ -79,21 +81,22 @@ This repo is used to capture architectural and design decisions as a reference o ## Server -| Index |Tags|Description| -|-------------------------|----|-----------| -| [ADR-1](adr/ADR-1.md) |jetstream, client, server|JetStream JSON API Design| -| [ADR-2](adr/ADR-2.md) |jetstream, server, client|NATS Typed Messages| -| [ADR-3](adr/ADR-3.md) |observability, server|NATS Service Latency Distributed Tracing Interoperability| -| [ADR-4](adr/ADR-4.md) |server, client|NATS Message Headers| -| [ADR-5](adr/ADR-5.md) |server, client|Lame Duck Notification| -| [ADR-6](adr/ADR-6.md) |server, client|Naming Rules| -| [ADR-7](adr/ADR-7.md) |server, client, jetstream|NATS Server Error Codes| -| [ADR-9](adr/ADR-9.md) |server, client, jetstream|JetStream Consumer Idle Heartbeats| -| [ADR-10](adr/ADR-10.md) |server, client, jetstream|JetStream Extended Purge| -| [ADR-28](adr/ADR-28.md) |jetstream, server|JetStream RePublish| -| [ADR-30](adr/ADR-30.md) |server|Subject Transform| -| [ADR-31](adr/ADR-31.md) |jetstream, client, server|JetStream Direct Get| -| [ADR-26](adr/ADR-26.md) |server|NATS Authorization Callouts| +|Index|Tags|Description| +|-----|----|-----------| +|[ADR-1](adr/ADR-1.md)|jetstream, client, server|JetStream JSON API Design| +|[ADR-2](adr/ADR-2.md)|jetstream, server, client|NATS Typed Messages| +|[ADR-3](adr/ADR-3.md)|observability, server|NATS Service Latency Distributed Tracing Interoperability| +|[ADR-4](adr/ADR-4.md)|server, client|NATS Message Headers| +|[ADR-5](adr/ADR-5.md)|server, client|Lame Duck Notification| +|[ADR-6](adr/ADR-6.md)|server, client|Naming Rules| +|[ADR-7](adr/ADR-7.md)|server, client, jetstream|NATS Server Error Codes| +|[ADR-9](adr/ADR-9.md)|server, client, jetstream|JetStream Consumer Idle Heartbeats| +|[ADR-10](adr/ADR-10.md)|server, client, jetstream|JetStream Extended Purge| +|[ADR-26](adr/ADR-26.md)|server|NATS Authorization Callouts| +|[ADR-28](adr/ADR-28.md)|jetstream, server|JetStream RePublish| +|[ADR-30](adr/ADR-30.md)|server|Subject Transform| +|[ADR-31](adr/ADR-31.md)|jetstream, client, server|JetStream Direct Get| +|[ADR-33](adr/ADR-33.md)|jetstream, client, server|Metadata for Stream and Consumer| ## When to write an ADR diff --git a/adr/ADR-00.md b/adr/ADR-33.md similarity index 100% rename from adr/ADR-00.md rename to adr/ADR-33.md