中文README

CCKVServer

This project aims to practice building a KV server from scratch. It supports basic storage functions and publish-subscribe functionality.

Architecture

Features

• Storage: Supports multiple storage backends, including in-memory DashMap, embedded rocksdb, and sled.
• Encryption Protocols: Supports Tls protocol or Noise protocol (currently, only NN mode is supported for Noise protocol).
• Multiplexing: Supports Yamux protocol or Quic protocol.
• Monitoring and Measurement: Integrated with opentelemetry and jaeger.
• Custom Frame Data Encapsulation Format: Each data frame header occupies four bytes, containing information on length, compression status, and compression format.

Usage

Profile

The server and client will read configuration files generated by tool/gen_config.rs.

~/$ cargo run --bin gen_config -- -h
help generating client and server config

Usage: gen_config [OPTIONS] --protocol <PROTOCOL>

Options:
  -p, --protocol <PROTOCOL>          Tls files are required when using Quic and Tls.
                                     You may use tools/gen_cert.rs to generate these files. [possible values: tls, noise, quic]
  -a, --addr <ADDR>                  [default: 127.0.0.1:1973]
      --enable-jaeger                
      --enable-log-file              
      --log-level <LOG_LEVEL>        [default: info]
      --log-path <LOG_PATH>          [default: /tmp/kv-log]
      --log-rotation <LOG_ROTATION>  [default: daily] [possible values: hourly, daily, never]
      --storage <STORAGE>            [default: memtable]
  -h, --help                         Print help

Running Server

cargo run --bin kvs

Running Client

cargo run --bin kvc

Supported Commands

• get Get the value of a specified key.
• getall Get the values of all keys.
• mget Get the values of multiple specified keys.
• set Set the value of a specified key. If the key already exists, its value is updated.
• mset Set the values of multiple keys. If a key already exists, its value is updated.
• del Delete a specified key and its associated value.
• mdel Delete multiple specified keys and their associated values.
• exist Check if a specified key exists.
• mexist Check if multiple specified keys exist.
• Publish Publish a message to a specified channel.
• Subscribe Subscribe to a specified channel to receive messages.
• Unsubscribe Unsubscribe from a specified channel.

Benchmark

Pubsub

A benchmark test for the Publish operation was conducted with 100 subscribers. The results are as follows:

publishing              time:   [76.341 µs 76.824 µs 77.390 µs]
                        change: [-0.6231% +0.9935% +2.7754%] (p = 0.29 > 0.05)

The average time for each publish operation is 76.341 microseconds, meaning the server can handle approximately 13,097 publish operations per second.

🚧TODO🚧

• Replace the Subscribe implementation (currently based on Tokio channel) with a message queue (e.g., Kafka, RabbitMQ, or Redis Pub/Sub)
• Implement hash sharding mechanism for DashMap
• Expand the support modes for Noise protocol to include more modes like XX and IK.
• Add support for cluster mode.
• Fix the conflict between Noise protocol and Yamux protocol. Currently, data streams constructed by Noise protocol encounter EOF when placed in Yamux.