[Issue #7419] [WIP] Enable pulsar component to send their metrics to topics#7433
[Issue #7419] [WIP] Enable pulsar component to send their metrics to topics#7433KannarFr wants to merge 18 commits intoapache:masterfrom
Conversation
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/sender/PulsarMetricsSender.java
Outdated
Show resolved
Hide resolved
| public void start() { | ||
| final int interval = this.conf.intervalInSeconds; | ||
| log.info("Scheduling a thread to send metrics after [{}] seconds in background", interval); | ||
| this.metricsSenderExecutor.scheduleAtFixedRate(safeRun(this::getAndSendMetrics), interval, interval, TimeUnit.SECONDS); | ||
| getAndSendMetrics(); | ||
| } | ||
|
|
||
| @Override | ||
| public void getAndSendMetrics() { | ||
| List<Metrics> metricsToSend = this.pulsar.getBrokerService().getTopicMetrics(); | ||
|
|
||
| metricsToSend.forEach(metrics -> { | ||
| try { | ||
| log.info("Sending metrics [{}]", metrics.toString()); | ||
| this.producer.send(metrics.toString()); | ||
| } catch (PulsarClientException e) { | ||
| e.printStackTrace(); | ||
| } | ||
| }); | ||
| } | ||
| } |
There was a problem hiding this comment.
Should we embed a retry with backoff stuff?
KannarFr
changed the title
KannarFr
force-pushed
the
metrics
branch
3 times, most recently
from
3230d71 to
900c7ec
Compare
sijie
left a comment
sijie
left a comment
There was a problem hiding this comment.
Why know just write an agent to collect the stats and write the stats to a Pulsar topic? It doesn't have to write such code at the broker side.
|
@sijie pull-based collection requires a more complex infra, but also for the monitored system to buffer data during polling. So when it's under load, you'll choose between eating memory or losing data. While in push, the data is out ASAP and the collector makes that decision. So push scales much better than poll particularly in the broker case where monitoring data increases with topics number. WDYT? |
KannarFr
force-pushed
the
metrics
branch
2 times, most recently
from
735235b to
fe4db2a
Compare
sending each write to metrics sender producer
|
@KannarFr You can install a local agent per bookie to scrap the metrics from the local endpoint and push it to your metrics system. How is that different from adding a sender in Pulsar? |
2 participants
Fixes #7419
Motivation
Enable pulsar components to send their metrics to defined topics in a dedicated tenant.
Modifications
Create a MetricsSender interface and fill it with PulsarMetricsSender as the "to pulsar" sender.
Verifying this change
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
Documentation