Distributed atomic counter support in JetStream KV

[FalxDraco]

Feature Request

Add Distributed Atomic Counter support to JetStream KV.

Use Case:

Slack thread

Proposed Change:

• add supporting code to kv
• add command-line support
  • a new flag for kv add subcommand to specify whether the stream backing a bucket should be a file stream or a memory stream, e.g.:
    • nats kv add my_bucket --replicas 1 --history 5 --STORAGE=FILE
    • nats kv add my_bucket --replicas 1 --history 5 --STORAGE=MEMORY
  • two new subcommands:
    • incrby x, e.g.: nats kv INCRBY 20 my_bucket my_key_1
    • decrby x, e.g.: nats kv DECRBY 10 my_bucket my_key_1
• allow interacting with the counters programmatically
  • 1st phase: via publishing messages to dedicated subjects; this would allow the new feature to be used before the official clients catch up.
  • 2nd phase: via Synadia official JetStream client(s).

Who Benefits From The Change(s)?

• Synadia - by closing the feature gap between JetStream and more popular solutions.
• Synadia - by making clients and users happy. Who doesn't need or use counters in production?
• JetSstream users - who can say no to happiness ?

Alternative Approaches

• The competition
  • Apache Kafka
  • Apache Pulsar
• Other software
  • Redis
  • RDBMS
  • etc

[ajlane]

The Slack thread describing the use case has been archived, but I hope that the use cases are still easy enough to imagine.

I am currently doing this this by publishing +1s and -1s to a stream, then consuming that stream with another client that eventually puts the accumulated results into a kv store. It would be nice to skip that step.

[derekcollison]

We continue to think through some of these patterns.

For now, the client can read and update only if no updates got in between. If the update rate is high this might not be a good solution however.

[elichai]

Is anyone still working/thinking about this?

[ripienaar]

it's on the short term roadmap yes

[pySilver]

@ripienaar any chance this will be implemented in 2.11?

[ripienaar]

I believe it's still the plan yes

[sloveridge]

I have been checking for this in the 2.11 preview announcements but haven't seen it land yet. Could you let us know if this will be in 2.11 still?

Thank you!

[derekcollison]

Yes it will be. We are admittedly behind, but not for lack of effort and commitment, we just been helping out customers and partners. But it will be there.

[Ilyes512]

Any updates on this? Is this tight to a specific issue in the 2.11 milestone that we could track?

[derekcollison]

No updates but still planned, just running a bit behind.

[max13fr]

Hello,

Does anyone have found a workaround to wait this feature ?

We can read the value then update with the incremented value, but I didn't find any transactional command that avoid missing an increment if 2 get values are done before any update (so only incremented by one instead of two).

Thanks in advance,
Best regards, Max

[pySilver]

It will not be incremented by two because there is versioning

[Jarema]

@max13fr you can use kv.Update() method, which takes a revision as an argument (that you get with the kv value). It would fail if you missed an increment.