Data Prepper Prometheus Endpoints

[devarshi2002]

Hello,

I'm setting up monitoring for Data Prepper and want to confirm the correct approach for Prometheus-based alerting.

From the docs, I understand Data Prepper exposes:

• /metrics/sys — JVM/system metrics (Prometheus scrape format)
• /metrics/prometheus — plugin metrics (Prometheus scrape format), default port 4900

My questions:

1. Are these the only two endpoints intended for Prometheus scraping, or is there a unified/aggregated endpoint recommended for alerting setups?
2. Beyond the common AbstractBuffer / AbstractProcessor / AbstractSink metrics (recordsIn, recordsOut, recordsInFlight, writeTimeouts, etc.), is there a full reference list of metric names per built-in plugin (e.g. opensearch-sink, otel_trace_source, grok processor)?
3. Are there any officially recommended Prometheus alerting rules / thresholds for Data Prepper (e.g. for recordsWriteFailed, writeTimeouts, or pipeline backlog via recordsInFlight)?
4. Does Data Prepper support any native alerting hooks, or is Prometheus + Alertmanager (or Grafana Alerting) the expected pattern for all alerting?

Any pointers to existing docs, sample Prometheus rule files, or Grafana dashboards the community uses would be great.

Thanks!

[ruslan-shaydullin]

In Data Prepper 2.16.0, /metrics/sys and /metrics/prometheus are aliases, not separate JVM/plugin feeds. Both are registered with the same handler and registry (source). An isolated HTTP → grok → stdout container test confirmed both returned JVM and plugin metrics; demo_grok_recordsIn_total increased from 0 to 3 on both after three events. Scraping one path—such as /metrics/prometheus—is sufficient. Scraping both under different jobs would collect the same measurements twice.

For plugin-specific references, the release includes metric sections for OpenSearch sink, OTel trace source, and grok. Use the scrape from your configured instance to get exact Prometheus names and labels: the test exposed demo_BlockingBuffer_recordsWritten_total, for example, rather than using the YAML plugin name bounded_blocking literally. Some metrics are registered only when their component/path is exercised.

One distinction for backlog alerts: recordsInBuffer tracks queued records; recordsInFlight tracks records already read and still being processed (definitions). Both gauges were present in the test. For cumulative failure/timeout counters, evaluate increases or rates rather than the lifetime total.

I did not find a maintained alert-rule/threshold pack in the release's monitoring docs or examples. For Data Prepper self-monitoring, I would use Prometheus + Alertmanager or Grafana Alerting for rule evaluation and notifications, with sustained backlog/error conditions calibrated to your pipeline. The metric-serving code does not itself evaluate alert rules; that is not a claim that no custom integration can exist.

Prepared with AI-assisted source inspection and an isolated Data Prepper 2.16.0 container check.