About kvrocks add timeseries

[xiaobiaozhao]

1. I think the timeseries module can be introduced into kvrocks. This module can be realized very efficiently by taking advantage of the natural orderliness of the underlying data of rocksdb.
2. In order to implement this module, we need to add a new column family to rocksdb. The key of this column family is the key of timeseries, and the value is a list. Each element in the list is a pair. The first of the pair is timestamp, and the second is value.
3. The cluster can allocate keys to different nodes based on the sensor hash value of TS. Each node has a rocksdb instance, so that distribution can be achieved.
4. Related reference documents https://redis.io/docs/data-types/timeseries/quickstart/

I have made a demo to implement some ts-related functions, and compared kvrocks vs. other rocks-based DBs. When tested on the same data, kvrocks won completely.

As you can see ts* reads three times the qps than the others. The resource consumption of kvrocks such as CPU and disk is much smaller than other DVs.

[mapleFU]

Hmmm I'm glad to support more types, but firstly, wouldn't professional TSDB better in this case? (like IOTDB, InfluxDB..).

Since TimeSeries workload may having batch writing and analysis. Batch writing might be suitable in RocksDB, but I think analysis would consuming lots of CPU, and after compaction, we cannot make full uses of columnar data and fast computing on it.

[xiaobiaozhao]

Yes, we don't need to compare with a professional timing database, we just need to be able to access similar timing data via kvrocks.

[mapleFU]

Yeah, I think it's ok

Another problem is that. Generally, kvrocks don't eat lots of CPU. The most of CPU is used for compaction. But Agg and analysis is a CPU-bound operation. Which should be take care of.

[xiaobiaozhao]

You are absolutely right that cpu usage needs to be evaluated by the user. However, I test to do some simple calculations do not consume too much cpu time. Most of the time is wasted on business encodings and decodeings, such as json-encode json-decode.

[PragmaTwice]

I would be glad to see if RedisTimeSeries can be introduced.

I think the first step is to draft a detailed storage design, and you can refer to here to see how other data types are designed.

In order to implement this module, we need to add a new column family to rocksdb. The key of this column family is the key of timeseries, and the value is a list. Each element in the list is a pair. The first of the pair is timestamp, and the second is value.

IMHO it is too rough to get your point of storage design, and generally it need to be detailed to calculate the time/space complexity for each TS command, e.g. how efficient would it be for timestamp query / tag query / range query / aggregate function query?

[xiaobiaozhao]

Yes, we can redesign the data structure

[tanruixiang]

@xiaobiaozhao I have some experience with TSDB, please let me know when you have a more detailed design, and let me to join the new design discussion. I may be able to provide some help. 👍🏻

[git-hulk]

That's cool, I also prefer discussing the design first if possible.

[xiaobiaozhao]

it's very cool