Skip to content

Logging

Jump to bottom
James Brucker edited this page Nov 24, 2025 · 1 revision

Some recommended practices for logging of containerized applications are:

  • Log to standard output (stdout) and let the container or orchestrator (Kubernetes) manage logs.
  • In a container, keep the logging configuration simple. One logger.
  • Uvicorn logs to stdout by default. If using Uvicorn, align the application's logs with Uvicorn's logs.
    • There is a command line option to request logging (--access-logs)?
  • Use structured logging, i.e. JSON
  • Let Docker, Kubernetes, or Systemd handle log collection.
  • Use a log collector and aggregator.

Viewing Container Logs

docker logs <container>

Logging Agents and Aggregators

Loki

  • Log aggregation system from Grafana Labs.
  • Stores logs efficiently by indexing only labels (not full log text).
  • Offers fast search and low storage cost.
  • Integrates well with Prometheus and Grafana.
  • Typically paired with Promtail, but also accepts logs from Fluent Bit and others.
  • Role: log storage + query engine (ElasticSearch alternative optimized for logs).

Promtail

  • Lightweight log shipper designed specifically for Loki.
  • Reads logs from:
    • Docker containers (stdout/stderr)
    • Files (e.g., nginx access/error logs)
    • Journald
  • Pushes them to Loki.
  • Simple, reliable, minimal resource usage.

Fluentd

  • CNCF log processor and forwarder. "General purpose log pipeline engine."
  • Very flexible with hundreds of plugins.
  • Can route logs to many destinations:
    • Loki
    • Elasticsearch
    • Kafka
    • S3
    • Cloud logging services
  • Often used in Kubernetes or multi-pipeline data environments.
  • Better suited for central/on-node log transformation and routing than Promtail.

Logstash

  • Part of the Elastic (ELK) stack.
  • Feature-rich ingestion and transformation pipeline.
  • Designed for complex parsing, enrichment, and routing.
  • Most commonly outputs logs to Elasticsearch.
  • Pros:
    • Enormous processing capability.
  • Cons:
    • Heavy compared to Promtail or Fluent Bit.
    • Usually unnecessary unless using Elasticsearch or complex log rules.

Fluent Bit

  • Lightweight log forwarder from the Fluentd ecosystem.
  • Written in C and designed for performance (Kubernetes, edge nodes).
  • Supports routing logs to:
    • Loki
    • Elasticsearch
    • Kafka
    • Fluentd
    • Cloud targets
  • Some filtering and parsing capabilities; more than Promtail but less than Fluentd.

Filebeat

  • Lightweight shipper from the Elastic Beats family.
  • Reads local logs and sends them to:
    • Elasticsearch (most common)
    • Logstash
    • Kafka
  • Provides metadata for Docker and Kubernetes logs.
  • Less flexible for heavy processing, usually paired with Logstash or Elasticsearch.
  • Filebeat is to Elasticsearch what Promtail is to Loki.

Elasticsearch

A distributed, scalable search and analytics engine commonly used for:

  • Log storage and querying
  • Full-text search
  • Analytics dashboards
  • Application search features
  • Original code is open-source and free to self-host. Current version is commercial & requires paid license.
  • Often used with Logstash and Kibana for visualization (the "ELK Stack").

Datadog Agent

  • Single agent that collects:
    • Logs
    • Metrics
    • APM traces
  • Sends to Datadog’s SaaS backend.
  • Very easy to install with Docker autodiscovery.
  • Proprietary and paid beyond trial/free limits.

Log Shippers to collect & forward logs

  • Promtail best for Loki users.
  • Fluent Bit best universal lightweight shipper supporting many outputs.
  • Filebeat best for Elastic Stack users.
  • Datadog Agent best when using the Datadog SaaS platform.

All can sit on a Docker host and ship container logs to a central destination.

Log Processing Pipelines (complex parsing and enrichment)

  • Fluentd - more lightweight and widely used in Kubernetes
  • Logstash - best in Elastic-based enterprise pipelines with heavy transformations.

Log Storage & Query Systems

  • Loki
  • Elasticsearch - commercial with "source available" version that can be self-hosted for free
  • OpenSearch - open-source version of Elasticsearch
  • Splunk (commercial)
  • Datadog logs (commercial)
  • AWS CloudWatch (commercial)

Best Options

The requirements for this application are

  1. Collect logs from containerized apps. Containers may be running on different servers.
    • FastAPI web service (stdout logs)
    • Nginx proxy for FastAPI app. Has access/error logs.
    • Postgres database logs (if available)
  2. Centralize the logs
  3. Minimal operational overhead
  4. (Desirable) Integrate with existing monitoring that is using Prometheus

Best Overall Choice: Loki + Promtail

Reasons:

  • Low storage cost.
  • Lightweight.
  • Integrates well with Prometheus and Grafana.
  • Install and configuration are simple in Docker or Docker Compose.
  • Perfect for remote, containerized environments.
  • Grafana can be used as interface to both metrics (Prometheus) and logs (Loki)

This is also the most common modern open-source stack for observability in Docker/Kubernetes setups.

A good alternative is Fluent Bit with Loki

  • more flexible and universal shipper
  • more routing/filtering ability than Promtail
  • good if logs may later go to multiple destinations (S3, Kafka, etc.)

Clone this wiki locally