Feature: Zero-copy datastore key serialisation and deserialisation

[tobiemh]

Is your feature request related to a problem?

Currently when creating Vec<u8> keys for use in the datastore, we pass in references, which are then cloned before being serialized. In addition when deserializing a datastore key, the value is cloned to create owned data.

This is unnecessary as the datastore key is never used or held beyond the end of a local function.

Describe the solution

Zero-copy deserialization will ensure that we are not unnecessarily cloning &str values when serializing, and cloning data once again when deserializing.

With this improvement, writing to and reading from the key-value store should be quicker, with less memory allocation.

For this to work, we need to make the storekey deserializer accept borrowed data as can be seen in rmp-serde - https://github.com/3Hren/msgpack-rust/blob/master/rmp-serde/src/decode.rs#L909-L1003

#[derive(Clone, Debug, Eq, PartialEq, PartialOrd, Serialize, Deserialize)]
pub struct Ns<'a> {
    kv: &'a str,
    _a: &'a str,
    ns: &'a str,
}

impl<'a> Into<Vec<u8>> for Ns<'a> {
    fn into(self) -> Vec<u8> {
        self.encode().unwrap()
    }
}

impl<'a> From<Vec<u8>> for Ns<'a> {
    fn from(val: Vec<u8>) -> Self {
        deserialize(&val).unwrap()
    }
}

pub fn new<'a>(ns: &'a str) -> Ns<'a> {
    Ns::new(ns)
}

impl<'a> Ns<'a> {
    pub fn new(ns: &'a str) -> Ns {
        Ns {
            kv: BASE,
            _a: "!ns",
            ns,
        }
    }
    pub fn encode(&self) -> Result<Vec<u8>, Error> {
        Ok(serialize(self)?)
    }
    pub fn decode(v: &[u8]) -> Result<Ns, Error> {
        Ok(deserialize(v)?)
    }
}

Alternative methods

No alternative methods.

SurrealDB version

surreal 1.0.0-beta.5 for macos on aarch64

Contact Details

No response

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[tobiemh]

The serializer/deserializer implementation for SurrealDB keys is located in https://github.com/surrealdb/storekey

[tobiemh]

The keys themselves are all located in https://github.com/surrealdb/surrealdb/tree/main/lib/src/key

[demfabris]

Refactoring surrealdb-derive
This cannot be merged yet as it conflicts with Deserialize in storekey crate

[demfabris]

Refactoring storekey
surrealdb/storekey#1

[ChinYing-Li]

Hi there,
does this issue still need help? If so, I'd like to work on this.