Performance hits when using this crate

[ultrabug]

Hi (again)

Disclaimer : please bear with me, I plan on using (if you're okay) this discussion as a notebook to my findings to allow anyone to suggest / review / argue my methodology and thoughts instead of doing it on my own. My intention is to find ways to improve this useful crate while making sure we get the best experience and performance out of it.

While working around how to deserialize Avro messages in a consumer, I started measuring the latency of deserialization using vanilla avro-rs, struct mapped deserialization with vanilla avro-rs and then using the async_impl with EasyAvroDecoder

What I'm using as a baseline on my environment is:

• kafka topic filled with 80+M messages
• 20 partitions
• kubernetes + 20 replicas (1 reader per partition)
• prometheus histograms with the following buckets : vec![0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1], meaning that my lowest grain is 100 micro seconds

I'm measuring the max of P50, P75 and P95 of the deserialization function between my replicas

hardcoded schema + avro-rs

pub fn get_decoded_message(schema: &Schema, message: &BorrowedMessage<'_>) -> Result<Partner> {
    let mut reader = Cursor::new(&message.payload().unwrap()[5..]);
    let val = from_avro_datum(&schema, &mut reader, None).unwrap();
    let partner: Partner = match &val {
        Value::Record(inner) => from_value::<Partner>(&val).unwrap(),
        _ => unreachable!(),
    };
    Ok(partner)
}

schema_registry_converter async_impl + easy avro

pub async fn get_decoded_message_converter(avro_decoder: &EasyAvroDecoder, message: &BorrowedMessage<'_>) -> Result<Partner> {
    let decode_result = avro_decoder.decode(message.payload()).await;
    let decoded: Value = match decode_result {
        Ok(decoded) => decoded.value,
        Err(err) => unreachable!(),
    };
    let partner: Partner = from_value::<Partner>(&decoded).unwrap();
    Ok(partner)
}

First finding :

Using schema_registry_converter impacts my (max) overall throughput very badly : (~100K msg/s vs ~56K msg/s)

First lead :

EasyAvroDecoder uses a Mutex on the whole decode function, but so far I was under the impression that the lock is useful only for the cases when the schema is not in cache so : maybe we should only use a Mutex on cache misses?

I did not yet look at the code for this, but maybe there's already a strong argument and I would waste my time.

I hope this is helpful and constructive

Thanks

[gklijs]

Hi, thanks a lot for doing performance tests.
I assume something like you proposed is possible, after all if it's already in the cache, it only needs to be able to read.

I would very much appreciate a PR which will improve performance. I might look into it myself as well.

As long as indeed all the cache misses has some mutex, as you don't want to make a call to Schema Registry for the time between that ID was first encountered, till the result is cached. That's basically the main challenge.

[ultrabug]

I would very much appreciate a PR which will improve performance. I might look into it myself as well.

Yep, I'm not sure my Rust is good enough yet to do this properly but I'll keep on pushing my performance tests anyway.

[ultrabug]

As promised, I took some time to look into a fast implementation to outline what it could look like and compare that to the current EasyAvroDecoder implementation.

The results is there: #70

I hope you find it interesting!

[gklijs]

Dash map seems used a lot, and draws in only a few transitive dependencies. I think it makes most sense to replace all occurrences of hashmap with the dash map. I could pick this up in a few weeks, but if you have the time and adopt the PR that would be great.

[ultrabug]

My PR was to get your impression and approval, should I take it that:

• you are okay with using dashmap
• you think it could be used to replace all the hashmap cache usage
• the goal is to replace the internals of the current structs by dashmap
• and plan (or wish me) to do it?

Am I right to see that as a green light?

[gklijs]

Yes, and as I said, I'm also OK with doing it myself, just don't have the time for it the next few weeks. I appreciate all the work you did already.

[menneni]

Interesting!

[gklijs]

@menneni FYI the changes are on master. I'm mostly waiting for apache-avro to have a release, so I can cut the 3.0.0 release. As the API is slightly changed moving to Dashmap.