Monitoring

Monitoring the forwarder

The forwarder exposes three complementary signals for operations: HTTP metrics, periodic status logs, and process health (via your supervisor or orchestrator).

1. Enable metrics

Set metrics.enabled: true in your config. The forwarder starts a small HTTP server that serves:

Endpoint	Purpose
GET /metrics	Prometheus scrape endpoint (OpenTelemetry)
GET /health	Liveness probe — returns {"status":"UP"}
GET /ready	Readiness probe (sink, buffer, hibernate)
GET /deadletters	Dead letter batch metadata only (when publish_batch.dead_letter.path is configured)

Both endpoints share the same metrics.host and metrics.port. /health is only available when metrics are enabled.

curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/metrics
curl http://127.0.0.1:8080/deadletters

GET /deadletters returns a JSON array of batch metadata (filename, created_at, event_count, bytes, failure_reason, sink_type, batch_attempts). It does not return log record bodies. Retrieve spilled content from the dead_letter.path volume (for example kubectl exec in Kubernetes). Do not expose the management port to untrusted networks without authentication.

Bind to 127.0.0.1 when Prometheus runs on the same host. Use 0.0.0.0 only if a remote scraper needs direct access, and restrict access at the network layer.

2. Scrape with Prometheus

Add a scrape job targeting the forwarder management port:

scrape_configs:
  - job_name: log-forwarder
    static_configs:
      - targets: ["localhost:8080"]
    metrics_path: /metrics
    scrape_interval: 15s

If you changed metrics.path in config, use that value for metrics_path.

3. Health and readiness checks

Use /health for liveness probes. It confirms the management server is running only; it does not verify sink connectivity after startup.

Use /ready for readiness probes when metrics.enabled: true. It returns 503 when:

• the sink fails its connectivity check (when the sink implements sink.Checker and metrics.readiness.sink_check is true)
• pipeline buffer depth / capacity exceeds metrics.readiness.buffer_threshold (default 0.8)
• metrics.readiness.require_files: true and no log files are being tailed
• the forwarder is in hibernate mode after a failed publish batch (reason: sink_hibernating)

Use /health for liveness only during hibernate: the process is still running and blocked on backpressure, but it should not receive new traffic until /ready is 200 again.

Kubernetes example:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 30
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Optional readiness tuning:

metrics:
  enabled: true
  host: 0.0.0.0
  port: 8080
  readiness:
    path: /ready
    buffer_threshold: 0.8
    sink_check: true
    require_files: false
    sink_check_timeout: 5s

Pair readiness with Prometheus alerts on log_forwarder_publish_failures and log_forwarder_pipeline_buffer_depth for sustained sink or backlog issues.

4. Log-based status

Configure periodic status logging to stderr or a log file:

logging:
  level: info
  format: json
  status_interval: 30s

Every status_interval, the forwarder logs a forwarder status line with watched_files — useful when metrics are disabled or as a secondary signal. Set status_interval: 0 to disable.

Other useful log lines:

Log message	Meaning
log forwarder started	Process is up; includes sources, topic, and metrics address when enabled
sink connectivity verified	Startup sink check passed
sink unavailable at startup	Forwarder refused to start — destination unreachable
publish failed, retrying	Transient publish error; check sink destination and network
forwarder stopped	Clean or error shutdown

5. Metrics reference

Forwarder metrics:

Metric	Description
log_forwarder_lines_read	Lines read from watched files
log_forwarder_lines_published	Lines published to the configured sink
log_forwarder_lines_filtered	Lines dropped by configured filters (after transform)
log_forwarder_lines_skipped	Lines dropped (transform.on_error: skip)
log_forwarder_pipeline_buffer_dropped	Lines dropped when pipeline.on_full: drop and buffer is full
log_forwarder_transform_errors	Transform failures
log_forwarder_timestamp_parse_failures	Records that fell back to processing time during timestamp normalization
log_forwarder_publish_failures	Failed sink publish attempts
log_forwarder_publish_truncations	Records truncated to fit pipeline.max_publish_bytes
log_forwarder_publish_batch_flushes	Publish buffer flushes (reason: size, timer, shutdown, wake; result: success, hibernate, dead_letter, error)
log_forwarder_publish_hibernating	1 when the forwarder is in sink hibernate mode after a failed batch flush
log_forwarder_publish_dead_letter_batches	Publish batches written to dead letter storage
log_forwarder_publish_consecutive_dlq_batches	Consecutive dead-letter batches without a successful sink publish
log_forwarder_publish_batch_size	Records per publish batch flush (histogram)
log_forwarder_publish_batch_bytes	Serialized JSON bytes per publish batch flush (histogram)
log_forwarder_publish_buffer_active_bytes	Serialized JSON bytes waiting in the publish buffer
log_forwarder_publish_retries	Retries after a publish failure
log_forwarder_publish_duration	Sink publish latency (histogram, seconds)
log_forwarder_files_watched	Files currently being tailed
log_forwarder_pipeline_buffer_depth	Events queued between watcher and pipeline
log_forwarder_pipeline_buffer_capacity	Configured pipeline.buffer_size

Process and runtime metrics:

Metric	Description
process_memory_usage	Process RSS in bytes
process_cpu_time	Process CPU time (user/system)
go_memory_used	Go runtime memory in use
go_memory_allocated	Heap memory allocated by the application
go_cpu_time	CPU time spent by the Go runtime
go_goroutine_count	Number of live goroutines

6. What to alert on

Signal	Suggested condition	Likely cause
Publish failures	rate(log_forwarder_publish_failures[5m]) > 0 sustained	Sink unreachable, auth/TLS issue, or network partition
Publish retries	rate(log_forwarder_publish_retries[5m]) rising	Intermittent sink or timeout pressure
Publish latency	histogram_quantile(0.95, rate(log_forwarder_publish_duration_bucket[5m])) high	Sink load, network latency, or slow endpoint
Buffer backlog	log_forwarder_pipeline_buffer_depth / log_forwarder_pipeline_buffer_capacity > 0.8 sustained	Pipeline slower than ingest; risk of backpressure
No files watched	log_forwarder_files_watched == 0 while logs are expected	Wrong watch paths, patterns, or permissions
Buffer drops	rate(log_forwarder_pipeline_buffer_dropped[5m]) > 0	pipeline.on_full: drop under overload — permanent log loss; not recoverable on restart
Read/publish gap	rate(log_forwarder_lines_read[5m]) >> rate(log_forwarder_lines_published[5m])	Transform skips, filter drops, buffer drops (on_full: drop), persistent publish failures, or pipeline stall
High filter rate	rate(log_forwarder_lines_filtered[5m]) high vs lines_read	Expected when filtering noisy logs; tune rules if too aggressive
Memory growth	process_memory_usage or go_memory_used trending up without stabilizing	Possible leak or sustained backlog
Process down	/health failing or scrape target up == 0	Crash, OOM kill, or misconfigured port

7. Quick checks