Built-in metrics exporter: push node/cluster stats as ss4o docs to a monitoring cluster via _bulk

[Vinz2168]

Proposal

Add a built-in, opt-in metrics exporter: a background task that periodically collects xerj's own node/cluster/index stats, serializes them as ss4o (Simple Schema for Observability) documents, and pushes them via _bulk to an external ES/OpenSearch-compatible "monitoring" target (or another xerj node).

Prior art: I built the same thing for OpenSearch as a plugin — https://github.com/Vinz2168/opensearch-monitoring-exporter (polls Health/NodesStats/IndicesStats/ClusterState, writes ss4o docs into ss4o_metrics-opensearch-yyyy.MM.dd on a dedicated monitoring cluster, no Data Prepper / OTel Collector hop). In xerj it would be engine code, not a plugin.

Why: today xerj exposes /v1/metrics (Prometheus text, pull) and _nodes/stats / _cluster/health (pull). Teams that already use OpenSearch Dashboards' Observability → Metrics UI (or another xerj node) as their monitoring store currently need a scraper + pipeline to land those numbers there. A push exporter writes them durably in the shape that UI already reads, with nothing extra to deploy.

What already exists (checked in the tree)

Need	Existing piece
Push _bulk to an external target with bounded batches, capped/jittered backoff, per-item read-back, healthy stats	wal_tap — engine/crates/xerj-engine/src/wal_tap.rs
Boot config + superuser-only runtime edit, persisted overlay	WalTapConfig (xerj-common/src/config.rs) + PUT /_xerj/wal_tap
Node/cluster numbers	es_compat::nodes_stats, es_compat::cluster_health — but computed inline in the HTTP handlers
Internal counters/gauges	Prometheus registry in xerj-common/src/metrics.rs (served at /v1/metrics)

wal_tap alone is not the answer — it replicates WAL contents (documents), not metrics — but its delivery half is reusable as-is.

Sketch

1. Extract collectors: move the bodies of nodes_stats / cluster_health (and per-index stats) into pure fn build_*(&AppState) -> Value, called by both the HTTP handlers and the exporter — one source of truth, no drift between what _nodes/stats says and what gets exported.
2. metrics_export module: tokio task, every interval_secs → collect → ss4o docs → shared _bulk sender factored out of wal_tap. Bounded queue, drop-oldest with a counter when the target is down (memory must not grow).
3. ss4o mapping, two shapes:
   • one doc per Prometheus-registry sample (name, kind, value, unit, attributes@… with .→@ like Data Prepper) — the most faithful ss4o/OTel shape;
   • category docs (cluster_health, node_os, node_fs, node_indices, per-index) for the _nodes/stats-style data.
4. Config [metrics_export]: enabled (default false), target_url, target_auth, index (default ss4o_metrics-xerj-%{yyyy.MM.dd}), interval_secs, max_batch_docs, request_timeout_secs, max_retry_backoff_secs; runtime PUT/GET/DELETE /_xerj/metrics_export + /_stats, superuser-only like wal_tap (it is an outbound channel to an arbitrary host).
5. TLS: wal_tap currently has no custom CA / client cert. Add ca_cert_path (PEM) and client_identity_path (mTLS) to the shared sender — wal_tap gets them for free.
6. Index template: idempotent PUT _index_template for the ss4o mapping on the target at arm time (opt-out flag).

Honest differences vs the OpenSearch plugin

• No JVM: heap/GC/JVM-thread metrics don't exist. Export process RSS, OS threads, rayon/tokio pool stats under their real names — not relabelled as "heap" (note: _nodes/stats currently fills heap_used_percent with RSS/host-memory).
• RSS/meminfo/CPU are read from /proc → Linux-only; on macOS those fields are 0/estimated today. The exporter should omit, not zero, what it can't measure.
• No keystore: secrets via env var or file, same as wal_tap.target_auth.
• Single-node: no "which node ships cluster-level docs" election problem.

Open questions for the maintainer

1. Is a push exporter in scope for the engine, or should this stay an external tool (e.g. a xerj subcommand / companion binary scraping /v1/metrics)?
2. Preferred doc shape: per-sample ss4o, category docs, or both?
3. OK to factor the _bulk delivery out of wal_tap into a shared module (and add CA/mTLS there) as a prerequisite PR?
4. Should the extraction of nodes_stats/cluster_health into pure functions land first as a standalone refactor?

Happy to implement once the shape is agreed.

---

Drafted with an AI coding agent (Claude Code) after reading the xerj source and the OpenSearch plugin; file pointers above were verified in the tree, nothing has been implemented or benchmarked yet.