diff --git a/docs/Contributing/reference/configuration-for-contributors.md b/docs/Contributing/reference/configuration-for-contributors.md index 69f564b7d74..b0d343d5abf 100644 --- a/docs/Contributing/reference/configuration-for-contributors.md +++ b/docs/Contributing/reference/configuration-for-contributors.md @@ -229,6 +229,8 @@ By default, OpenTelemetry is used. Set `tracing_type` to `elasticapm` only if yo Enables exporting logs to an OpenTelemetry collector in addition to stderr output. When enabled, logs are sent to the OTLP endpoint configured via the standard `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable. Logs are automatically correlated with traces via `trace_id` and `span_id` attributes. +> **Note:** All log levels, including debug, are always sent to the OpenTelemetry collector regardless of the `logging.debug` setting. The `logging.debug` flag only controls what appears in stderr output. + > **Note:** This option requires `logging.tracing_enabled` to be set to `true`. Fleet will fail to start if `otel_logs_enabled` is `true` but `tracing_enabled` is `false`. - Default value: `false` diff --git a/tools/README.md b/tools/README.md index 1e308f8dfc9..9ae29becb55 100644 --- a/tools/README.md +++ b/tools/README.md @@ -242,6 +242,7 @@ go run ./tools/run-scripts -scripts-disabled -content 'echo "Test"' | `percona/` | Percona testing | Percona MySQL testing configs - See [percona/test/README.md](percona/test/README.md) | | `sentry-self-hosted/` | Self-hosted Sentry | See [sentry-self-hosted/README.md](sentry-self-hosted/README.md) | | `smtp4dev/` | Local SMTP testing | SMTP4Dev server with TLS certs for email testing | +| `signoz/` | SigNoz for traces, metrics, and logs | See [signoz/README.md](signoz/README.md) | | `telemetry/` | Jaeger + Prometheus for tracing | `docker compose up` - See [telemetry/README.md](telemetry/README.md) | | `terraform/` | Terraform provider for Fleet teams | `make install && make apply` - See [terraform/README.md](terraform/README.md) | | **MDM Tools** | | | diff --git a/tools/signoz/README.md b/tools/signoz/README.md new file mode 100644 index 00000000000..8514591b3dc --- /dev/null +++ b/tools/signoz/README.md @@ -0,0 +1,86 @@ +# Running SigNoz locally with Fleet + +[SigNoz](https://signoz.io/) is an open-source observability platform that provides traces, metrics, and logs in a single UI. This guide explains how to run SigNoz locally for Fleet development with optimized settings for reduced latency. + +## Prerequisites + +- Docker and Docker Compose +- A locally-built Fleet server (see [Testing and local development](../../docs/Contributing/getting-started/testing-and-local-development.md)) + +## Setup + +1. Clone the SigNoz repository at a specific release: + +```bash +git clone --branch v0.110.1 --depth 1 https://github.com/SigNoz/signoz.git +cd signoz/deploy +``` + +2. Modify the SigNoz UI port to avoid conflict with Fleet (which uses port 8080): + +In `docker/docker-compose.yaml`, change the signoz service port mapping: + +```yaml +services: + signoz: + ports: + - "8085:8080" # Changed from 8080:8080 to avoid conflict with Fleet +``` + +3. (Optional) For reduced latency during development, modify `docker/otel-collector-config.yaml`: + +```yaml +processors: + batch: + send_batch_size: 10000 + send_batch_max_size: 11000 + timeout: 200ms # reduced from 10s for dev + # ... + signozspanmetrics/delta: + # ... + metrics_flush_interval: 5s # reduced from 60s for dev +``` + +4. Start SigNoz: + +```bash +cd docker +docker compose up -d +``` + +Give it a minute for all services to initialize. The SigNoz UI will be available at http://localhost:8085. + +## Configuring Fleet + +Start the Fleet server with OpenTelemetry tracing and logging enabled: + +```bash +export FLEET_LOGGING_TRACING_ENABLED=true +export FLEET_LOGGING_OTEL_LOGS_ENABLED=true +export OTEL_SERVICE_NAME=fleet +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 + +./build/fleet serve +``` + +> **Note:** All log levels (including debug) are always sent to SigNoz regardless of the `--logging_debug` flag. That flag only controls stderr output. + +### Low-latency configuration (optional) + +For faster feedback during development, you can reduce the batch processing delays on the Fleet side: + +```bash +# Batch span processor delay (default 5000ms) +export OTEL_BSP_SCHEDULE_DELAY=1000 + +# Log batch processor settings +export OTEL_BLRP_EXPORT_TIMEOUT=1000 +export OTEL_BLRP_SCHEDULE_DELAY=500 +export OTEL_BLRP_MAX_EXPORT_BATCH_SIZE=1 + +./build/fleet serve +``` + +## Using SigNoz + +After starting Fleet with the above configuration, you should start seeing traces, logs, and metrics in SigNoz UI at http://localhost:8085. diff --git a/tools/telemetry/README.md b/tools/telemetry/README.md index 940c8f267c7..6dfc6b75608 100644 --- a/tools/telemetry/README.md +++ b/tools/telemetry/README.md @@ -20,3 +20,7 @@ OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" \ ``` Afterwards, you can navigate to http://localhost:16686/ to access the Jaeger UI. + +## Alternatives + +For a more full-featured observability platform that includes traces, metrics, and logs in a single UI, see [SigNoz](../signoz/README.md).