-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor(metrics): change gauge metrics to be scoped by worker id #9605
Conversation
ef3d229
to
50b87fd
Compare
Currently, we face several issues trying to keep resource metrics reasonable. For example, when a resource is re-created and has its metrics reset, but then its durable queue resumes its previous work and leads to strange (often negative) metrics. Instead using `counters` that are shared by more than one worker to manage gauges, we introduce an ETS table whose key is not only scoped by the Resource ID as before, but also by the worker ID. This way, when a worker starts/terminates, they should set their own gauges to their values (often 0 or `replayq:count` when resuming off a queue). With this scoping and initialization procedure, we'll hopefully avoid hitting those strange metrics scenarios and have better control over the gauges.
50b87fd
to
dc1e46c
Compare
} | ||
label { | ||
en: "Kafka Consumer" | ||
zh: "卡夫卡消费者" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
zh: "卡夫卡消费者" | |
zh: "Kafka 消费者" |
} | ||
label { | ||
en: "Kafka Consumer" | ||
zh: "卡夫卡消费者" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
zh: "卡夫卡消费者" | |
zh: "Kafka 消费者" |
consumer_kafka_opts { | ||
desc { | ||
en: "Kafka consumer configs." | ||
zh: "Kafka消费者配置。" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
zh: "Kafka消费者配置。" | |
zh: "Kafka 消费者配置。" |
en: "MQTT to Kafka" | ||
zh: "MQTT 到 Kafka" | ||
en: "Kafka Producer" | ||
zh: "卡夫卡制片人" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
zh: "卡夫卡制片人" | |
zh: "Kafka 生产者" |
} | ||
label { | ||
en: "Kafka topic" | ||
zh: "卡夫卡主题 " |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
zh: "卡夫卡主题 " | |
zh: "Kafka 主题 " |
Port of the gauge part of this for master (since e501 is a bit on hold): #9619 |
https://emqx.atlassian.net/browse/EMQX-8548
Currently, we face several issues trying to keep resource metrics
reasonable. For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.
Instead using
counters
that are shared by more than one worker tomanage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID. This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or
replayq:count
when resuming off a queue).With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
Related Wolff PR: kafka4beam/wolff#41