sweetcorn

A DuckDB backend for OpenTelemetry data.

Features

• Logs
  • Basic insert
  • Basic query
  • Create and query views
• Spans
• Metrics
• HTTP server
• Exporter for open telemetry collector
• Focus completly on DuckDB data types, OTEL -> database schema transformation, etc.
• TTL for rows (duck db does not provide it)
  • Table specific TTL configuration
• Refresh views periodically
  • This way the schemas will remain up to date
• Add configuration parameters for duckdb and add to config.yaml
• Compression:
  • DuckDB has built-in compression with lightweight compression algorithms.

Development

make test

DuckDB

curl https://install.duckdb.org | sh

Telemetry generation

Mock telemetry data can be generated using telemetrygen.

telemetrygen is installed as a go tool (check the tools directive in go.mod for the exact version).

# example: generate logs for 5 seconds
go tool telemetrygen logs --otlp-http --otlp-insecure --otlp-endpoint localhost:8090 --duration 5s

Query

Logs

• Get log severity count time series.

SELECT TimestampTime as time, SeverityText, count() as count
FROM otel_logs
WHERE time >= NOW() - INTERVAL 1 HOUR
GROUP BY SeverityText, time
ORDER BY time;

• Find any log.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE TimestampTime >= NOW() - INTERVAL 1 HOUR
Limit 100;

• Find log with specific service.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE ServiceName = 'telemetrygen'
  AND TimestampTime >= NOW() - INTERVAL 1 HOUR
Limit 100;

• Find log with specific attribute.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE LogAttributes['container_name'] = '/example_flog_1'
  AND TimestampTime >= NOW() - INTERVAL 1 HOUR
Limit 100;

• Find log with body contain string token.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE 'message' IN Body
  AND TimestampTime >= NOW() - INTERVAL 1 HOUR
Limit 100;

• Find log with body contain string.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE Body LIKE '%mes%'
  AND TimestampTime >= NOW() - INTERVAL 1 HOUR
Limit 100;

• Find log with body regexp match string.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE BODY GLOB '*'
  AND TimestampTime >= NOW() - INTERVAL 1 HOUR
Limit 100;

• Find log with body json extract.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE JSONExtractFloat(Body, 'bytes') > 1000
  AND TimestampTime >= NOW() - INTERVAL 1 HOUR
Limit 100;

References

1. Observability 3
2. LogHouse