KeyValue Revision is KeyStore wide? How can we cherry pick from a single key history?

[andrea-tomassi]

Hello,
I'm not sure I understand the concept of revision.

kvStore.put("k1", System.currentTimeMillis());
kvStore.put("k2", System.currentTimeMillis());
kvStore.put("k2", System.currentTimeMillis());
kvStore.put("k3", System.currentTimeMillis());

System.out.println(kvStore.get("k1").getRevision());
System.out.println(kvStore.get("k2").getRevision());
System.out.println(kvStore.get("k3").getRevision());

If you run this super-simple test you get:

1
3
4

This means that the "revision" value is not "key-related", but it seems to be "keystore-wide" instead.
In fact the revision you got for a brand new key and value (i.e. k3) is different from 1 (except for the very first key).

Is that correct and someway expected?

If it is, how can we manage the following scenario:

kvStore.put("k1", System.currentTimeMillis());

kvStore.put("kX", System.currentTimeMillis());
kvStore.put("kY", System.currentTimeMillis());
[...] // unknown put number from other processes
kvStore.put("k1", System.currentTimeMillis());
kvStore.put("k1", System.currentTimeMillis());
kvStore.put("kX", System.currentTimeMillis());
kvStore.put("kY", System.currentTimeMillis());
[...] // unknown put number from other processes
kvStore.put("k1", System.currentTimeMillis());

etc

At this point I get the very last k1 entry and I find the revision == 2345

Now, I'd like the previous 3 values from K1 without selecting all the "max-history" entries. (Assume I set max-history to 64 and I want to avoid this overhead).

Unfortunately, I cannot assume 2344 and 2343 are the appropriate revision numbers for K1 to select from its history.

More in general, how the method

KeyValueEntry get(String key, long revision)
could be userd in an appropriate way ?

Thanks a lot

[ripienaar]

Yes, as you say revision is a bucket wide number, so you can't guess what the n-1 revision number is for any given key. Need to do a history call at present.

[andrea-tomassi]

Thank you so much for confirming this for me.
This "key-value" abstraction is very powerful. What a pity. Being able to read a smaller portion form history (i.e. latest n entries instead of the entire history) would unlock some interesting scenario (with larger history max size).

[ripienaar]

Do you have like very large entries, whats the use case for this?

We have not had requests for this kind of thing - and usually history operations are very infrequent so we been focussing on optmising the get path for perf not the history paths

[andrea-tomassi]

Yes, I have a scenario where history could be large (for sure it could be larger than 64).
I'm going to build a "persistent, temporal aware, distributed" graphDB on NATS keystore DB.

Vertex will be saved as key-value documents (json value for vertex attributes).
The key is a unique id for the vertex, the value is a json with some attribute.

On the other hand, I'm going to use an edgelist based notation for edges (both direct link and reverse edgelist to enable reverse queries).
The key is a starting vertex, the value is the list of vertices with a link to the first one.

I also implemented a custom biz logic to avoid updates with a value equals to the current value, to avoid useless usage of history entries.
I'm also able to recognize equivalent json objects, so the equality is not limited to plain string exact matching.
This means that two json strings with same keys and values in different order are considered as equivalent.

The handy thing here is that NATS is going to manage for me the change history for each vertex and edge, and I can also "watch" for graph updates, this give me the "streaming" version of the graph for free.

Getting the latest graph status is straightforward, however, history gives me the possibility to "rebuild" every vertex/edge/graph status in the past in any given timestamp. I think this would be a game-changing feature for modeling time-based networks and graph data.
If I'm interested in one specific Vertex I can monitor the specific history involving that vertex.
This is also true for sub-graphs.

This could be done with time-based snapshots of the graph as well, but snapshots force to a discrete sampling or you can use the transaction log to move across the graph history, but this could be an high cost operation, so I would giva a try to the NATS approach.

At this point, for my specific scenario, it would be great if NATS could:

• unlock the 64 limit for the few "highly updated" vertices (actually I already unlocked it by updating the underlying Stream cfg)
• trace revisions for the specific key (subject)
• move across revisions in a predictable way (because of the above key-related revision number)
• provide the timestamp of the creation of the key and the timestamp of the update as well as a parameter (now the date and time is forced to be the current one, instead imagine your dataset is a time-series in the past. At that point you would like to build a temporal graph from that dataset, just keeping the original timestamps)

I'm going to implement these features by myself because I understand these can be domain specific.
For example I'm forced to add the "actual" timestamp as an attribute instead of leveraging the already present "metadata" from NATS.

Sorry for the verbosity, hope this answer to your question.

Best,

Andrea Tomassi

[ripienaar]

Thanks for the detail, @derekcollison will find interesting also.

• I believe you can just stream edit the history property higher post creation, clients wont fail if they find a buket with higher history they just wont make one
• Revision per key will be very hard in our current storage model
• I dont believe we have a way to go 10 back in a subject in the underlying jetstream, interesting idea to have that though as a consumer start signal
• Timestamps as your request will be hard, I think its quite fundemantal that data points be stamped by the server. However we have considered some TSDB use cases that would make this a requirement but no movement on that for now.

dont worry, this is great detail thank you

[derekcollison]

You could use lightweight (R1), and now memory based consumers to pull all history for a given key fairly effectively, even further filter by start time (last 2hrs), if that is helpful.