nats_stream_total_messages grows, but no other metric follows it

[hpdobrica]

Hello, I was tracking nats_stream_total_messages metric for one stream, which usually grows and falls together with nats_consumer_num_pending metric in our system.

Sometimes it happens that nats_stream_total_messages grows a lot (until it fills up the whole stream), but no other metric I could find (except nats_stream_total_bytes) shows similar growth.

I've tried summing all of the metrics I found that seem to show number of messages for a stream:

• nats_consumer_num_pending
• nats_consumer_num_ack_pending
• nats_consumer_num_redelivered
• nats_consumer_num_waiting

but the sum of all these for one stream is below 1k, while nats_stream_total_messages is around 100k.

Is there a metric that i'm missing that could explain what these 100k messages are? This spike seems to only be observable in our system through nats_stream_total_messages and nats_stream_total_bytes.

Any ideas what I'm missing and how i can find out how to track these messages?

Thanks a lot for your help!

[somratdutta]

hey @hpdobrica,
let me know if this issue still persists, then I will investigate futher.
Thanks.