Troubleshooting

Troubleshooting

Forwarder refuses to start

sink unavailable at startup

The configured sink failed its connectivity check (Kafka broker/topic or HTTP endpoint unreachable; file sink directory not writable).

Fix: Ensure the destination is up and reachable, then restart. For HTTP http-noauth, something must be listening on the configured URL.

Config validation error

Examples:

• Watermark or file sink inside a watched directory
• Missing sink.kafka when type: kafka
• Invalid regex in transform.pattern or parser.start_pattern

Fix: Read the error message, correct YAML, restart.

Corrupt watermark file

The startup error mentions watermark file ... is corrupt or unreadable and suggests recovery steps.

Fix: Restart once with --reset-watermarks, or set watch.state.reset_on_corrupt: true in config. The corrupt file is archived as {path}.corrupt.{timestamp} and tailing starts from the beginning (re-ships already-processed lines). Alternatively, rename or delete the watermark file manually.

No logs appearing at the sink

Check	Action
Watch path	Confirm log files exist under watch.sources / watch.paths
Patterns	Filenames must match globs (*.log, etc.)
Watermark ahead of content	You may have already tailed past those lines — see Watermarks and Restarts
Transform mismatch	Lines may be wrapped as _raw with _error — consume sink output and inspect
on_error: skip	Unparseable lines are dropped silently (debug log only)

Enable metrics and check:

• log_forwarder_files_watched > 0
• log_forwarder_lines_read increasing (new appended lines only)
• log_forwarder_lines_published vs log_forwarder_lines_skipped
• log_forwarder_lines_replayed — non-zero after restart means at-least-once replay from stale watermark, not missing new data

Permanent log loss from pipeline.on_full: drop

Symptoms:

• rate(log_forwarder_pipeline_buffer_dropped[5m]) > 0
• log_forwarder_lines_read or log_forwarder_lines_replayed increases but log_forwarder_lines_published lags or stalls
• Gaps in the sink with no publish-failure alerts

Cause: With on_full: drop, the watcher reads each line and advances its in-memory file offset before enqueueing. If the pipeline buffer is full, the line is discarded and the watermark is not updated. Restart resumes from the last processed watermark — dropped bytes are skipped forever. Under sustained backpressure, the same drop loop can continue indefinitely.

Fix: Set pipeline.on_full: block (default), increase buffer_size, fix publish latency, or scale the sink. Do not use drop expecting backpressure relief without accepting permanent loss. See Configuration-Reference#pipelineon_full-block-vs-drop.

Duplicate records in Kafka / downstream

Usually caused by:

• Deleting watermarks and re-reading from the beginning
• Running two processes with the same watermark file
• Log rotation with unexpected inode behavior (rare)
• Crash or kill -9 after Kafka ack but before watermark flush — the built-in Kafka sink uses leader ack only (RequireOne), synchronous writes, and no idempotent producer keys. Watermarks are debounced (watch.state.flush_interval, default 1s). A restart can re-publish the last flush window of records. See Choosing a Sink#Kafka delivery semantics.

Fix: One watermark file per process; avoid resetting watermarks unless you intend to re-ship. Design consumers for at-least-once delivery (dedupe keys, idempotent sinks). Tighten watch.state.flush_interval (or set 0) to shrink the duplicate window at the cost of more disk I/O — this does not provide exactly-once. When metrics are enabled, log_forwarder_lines_replayed increments for bytes re-read after restart due to watermark lag; log_forwarder_lines_read stays at zero if no new lines were appended.

Spring Boot stack trace split across many records

You need the multiline parser. Use an example-spring-boot-*.yaml config, not a single-line setup.

HTTP sink: connection refused at startup

Expected if nothing listens on http_noauth.url. Start your ingest service first, or use file/Kafka sink for local testing.

HTTP sink: 4xx / 5xx on publish

Non-2xx responses are retried then fail. Check ingest service logs. The built-in http-noauth sink does not send auth headers.

Kafka: publish failures after startup

Startup only checks connectivity once. Later broker outages show as:

publish failed, retrying

Monitor log_forwarder_kafka_publish_failures and broker health.

Log rotation surprises

When logrotate creates a new file at the same path:

• New inode → forwarder tails the new file from the beginning
• Old file’s watermark does not apply to the new inode

This is usually what you want for rotated logs.

Getting help

1. Run with logging.level: debug (if supported in your build) for more detail
2. Check forwarder stderr / logging.file
3. Inspect watermark file and one sample source log line
4. Open an issue on GitHub with config (redacted secrets), forwarder version, and symptom