chore(bottlecap): reduce lock contention in logs #256
Conversation
it doesnt make sense to send thousands of events to the main thread, and then to the logs agent, inverting makes more sense
so the telemetry api can directly send the logs to it, insted of the main thread
…h://github.com/DataDog/datadog-lambda-extension into jordan.gonzalez/bottlecap/reduce-lock-contention
added some debug logs, found that issue is in serialization of huge payloads
update how we read from stream
| @@ -1,11 +1,10 @@ | |||
| use serde::Serialize; | |||
| use std::collections::VecDeque; | |||
| use crossbeam::queue::SegQueue; | |||
There was a problem hiding this comment.
FWIW SegQueue is an unbounded multi-producer, multi-consumer. Minor issue, the multi-consumer bit requires more coordination in the implementation so you're leaving performance on the table by not using a single-consumer. More pressing, this is an unbounded queue and represents a potential site for unbounded allocations.
There was a problem hiding this comment.
I'm aware of the tradeoffs, I have to clean this PR still, will mark it as draft – plan to use a bounded queue here
There was a problem hiding this comment.
SegQueue is also theoretically lock free but it uses Acquire/Release semantics internally and so the semantics are equivalent to a mutex. Except Acq/Rel done on atomic structures cuts the Linux scheduler out so your latency will be worse generally speaking, except in specialized circumstances where the acq/rel using software has the highest CPU priority anyway and is always scheduled first.
Tokio's mpsc is good, will play well with the Linux scheduler and also nicely between async/sync code.
|
Closing due to other PRs doing this |
What?
Aggregatorlock as soon as possible.