Repository navigation
Issue: HINCRBY commands taking 10–30 ms despite low memory usage #8261
Replies: 2 comments 1 reply
|
Thanks for the detailed report - the shard stats and replication counters help a lot. The key observation is that it's not just HINCRBY: |
|
@vyavdoshenko Thanks for the explanation. I did some additional checks on the host: The 16 Proactor threads are pinned one-to-one to CPUs 0–15. mpstat over 60 seconds shows the CPUs are generally 94–100% idle. %iowait is essentially 0%. %steal is generally low, although I observed occasional individual CPU samples up to ~2.94%. So there doesn't appear to be general CPU saturation or an obvious CPU affinity issue. I understand why SELECT, INFO ALL, and SLOWLOG GET can take 10–30 ms since they wait for all 16 Proactors and therefore reflect the slowest Proactor. What I'm trying to understand is specifically HINCRBY. My understanding is that HINCRBY is routed to a single owning shard/Proactor, rather than waiting for all 16 Proactors. However, I'm seeing similar 10–30 ms latency for HINCRBY as well. If the HINCRBY execution itself should only take microseconds and the shards are balanced, why would the owning Proactor take 10–30 ms to process it when there is plenty of CPU capacity available overall? What would be the best next diagnostic to determine whether this is caused by short scheduler/VM-steal delays or something else preventing a Proactor from running/responding? If there is a Dragonfly-specific metric, tracing option, or profiling method for identifying the slow Proactor during one of these events, I'd be happy to collect it. |
Uh oh!
There was an error while loading. Please reload this page.
Running DragonflyDB v1.40.0 with 16 shards/threads and are observing unexpectedly high command latency in SLOWLOG.
The commands are primarily
HINCRBYoperations on small hashes. The increment value is always0.For example:
The SLOWLOG latency is frequently in the 10–30 ms range:
Interestingly, this does not appear to be specific to
HINCRBY. We also see similar latency for other commands:The hash contains approximately 79 fields.
At the same time, overall memory usage is low:
There is also no OOM/eviction pressure.
Shard memory/key distribution also appears reasonably balanced:
Is it expected for simple commands such as
HINCRBY,SELECT, andINFOto take 10–30 ms under load?Is there any DragonflyDB metric or diagnostic command that can help identify which shard/thread is responsible for this latency?
Are there any recommended DragonflyDB configuration or debugging options for investigating this type of command latency?
We would appreciate any guidance on how to identify the underlying cause.
Thanks.
All reactions