Stream Mirrors, Sources and Updates Support #1932
Conversation
Signed-off-by: Derek Collison <derek@nats.io>
Signed-off-by: Derek Collison <derek@nats.io>
Signed-off-by: Derek Collison <derek@nats.io>
Signed-off-by: Derek Collison <derek@nats.io>
Signed-off-by: Derek Collison <derek@nats.io>
Add in proper support for stream updates in clustered mode. Don't send API updates without subjects, caused GW parser errors. Stream internal loops use their own clients now. Signed-off-by: Derek Collison <derek@nats.io>
| // Headers for published messages. | ||
| const ( | ||
| JSMsgId = "Nats-Msg-Id" | ||
| JSExpectedStream = "Nats-Expected-Stream" | ||
| JSExpectedLastSeq = "Nats-Expected-Last-Sequence" | ||
| JSExpectedLastMsgId = "Nats-Expected-Last-Msg-Id" | ||
| JSStreamSource = "Nats-Stream-Source" |
There was a problem hiding this comment.
I saw these and wasnt sure what the user story this fixes? Would the user parse this subjet (its the ack subject) to figure out metadata? This would have original time stamp I guess, the sourced timestamp would be different?
There was a problem hiding this comment.
They could, we need it for sources to properly figure out where to pick back up for any given source. We walk backwards through our own stream and look at that header for stream sequence for the source stream. Then we recreate the consumer with that starting sequence +1.
There was a problem hiding this comment.
OK, its like 70 bytes overhead per message. We could pick out just the bits we need and put it in a header we dont show the user - and so with a much shorter name?
Usually I am not too concerned about messages sizes - a lot less than you :P - but 70/message is a lot.
There was a problem hiding this comment.
| Stream | Storage | Template | Consumers | Messages | Bytes | Lost | Deleted | Cluster |
+----------------+---------+----------+-----------+----------+---------+------+---------+----------------------+
| ORDERS_6 | File | | 5 | 0 | 0 B | 0 | 0 | n1-c1, n2-c1, n3-c1* |
| ORDERS_7 | File | | 5 | 0 | 0 B | 0 | 0 | n1-c1, n2-c1*, n3-c1 |
| ORDERS_2 | File | | 5 | 0 | 0 B | 0 | 0 | n1-c1, n2-c1*, n3-c1 |
| ORDERS_3 | File | | 5 | 0 | 0 B | 0 | 0 | n1-c1, n2-c1*, n3-c1 |
| ORDERS_4 | File | | 5 | 0 | 0 B | 0 | 0 | n1-c1*, n2-c1, n3-c1 |
| ORDERS_5 | File | | 5 | 0 | 0 B | 0 | 0 | n1-c1*, n2-c1, n3-c1 |
| ORDERS_8 | File | | 5 | 0 | 0 B | 0 | 0 | n1-c1, n2-c1*, n3-c1 |
| ORDERS_9 | File | | 5 | 0 | 0 B | 0 | 0 | n1-c1, n2-c1*, n3-c1 |
| ORDERS_0 | File | | 6 | 100 | 4.5 KiB | 0 | 0 | n1-c1, n2-c1*, n3-c1 |
| ORDERS_1 | File | | 6 | 100 | 4.5 KiB | 0 | 0 | n1-c1*, n2-c1, n3-c1 |
| ORDERS_ARCHIVE | File | | 0 | 200 | 27 KiB | 0 | 0 | n2-c2* |
+----------------+---------+----------+-----------+----------+---------+------+---------+----------------------+```
Worst case since my messages are tiny though
There was a problem hiding this comment.
After our discussion let me know if you still want me to look into shrinking or do we think this could be valuable as is to end users?
/cc @kozlovic
There was a problem hiding this comment.
I think its ok, it should be useful. 1 billion messages = 70GB of this header :P
Signed-off-by: Derek Collison <derek@nats.io>
derekcollison
force-pushed
the
mirrors2
branch
from
ee90822
to
8fe8b83
Compare
Signed-off-by: Derek Collison <derek@nats.io>
Signed-off-by: Derek Collison <derek@nats.io>
Signed-off-by: Derek Collison <derek@nats.io>
Remove panic on runAsLeader when not able to subscribe (which happens on shutdown) Gateway name access does not need lock since it is immutable. Will prevent deadlocks in some situations. Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Also various fixes, stability improvements, etc.
/cc @nats-io/core