v1.6.1

A release about speed and getting the details right: the Logs page histogram and totals come from per-minute counts on TimescaleDB, ClickHouse and MongoDB, facets work on projects with tens of millions of logs, the client IP behind a reverse proxy is the real one, database and Redis passwords can contain any character, and project health no longer emails on every quiet night.

Before upgrading: migrations 075 to 077 run on start. On ClickHouse the first start migrates the traces table and needs every 1.6.0 backend and worker stopped first (see Changed). Behind a reverse proxy, set TRUST_PROXY (1 for one proxy, 2 with Cloudflare in front).

Added

• Traffic map from client IPs: the geo map dashboard panel needed the fields a GeoIP pipeline step writes at ingestion, so projects without that step showed "No geo data" even though their logs carry the client IP. When those fields are missing, the panel now locates the most frequent client IPs of the time range (up to 10,000 distinct addresses) with the GeoLite2 database the server already uses for security enrichment, and sums them per country or per place. It works on existing logs, stores nothing and sends nothing outside the server. The IP is read from a metadata field such as client_ip (what the Nginx and Apache parsers write), ip or remote_addr, or from the field set in the new "Client IP field" panel setting (ipField); forwarded-for lists and ip:port values are understood. Private and unknown addresses are skipped, and the panel says where its locations come from.
• Instance admin pages for extensions: extensions can now add pages for instance administrators, next to the ones LogTide ships. On the backend, api.auth.requireInstanceAdmin guards a route for instance administrators (re-checked on every request, so an administrator who loses the role or is disabled is refused right away), api.admin.audit() records their actions in the audit log as ext.admin_action with the real administrator as actor, and api.orgs.getMany() looks up to 500 organizations at once. On the frontend, adminNav adds pages to the admin navigation and the command palette (administrators only), and adminOrgPanels adds sections to the admin organization page; admin pages live under /dashboard/admin/ext/<name>/ and inherit the admin guard. The extension API stays at version 1, and an instance without extensions looks and behaves exactly as before. See docs/extensions.md.
• Database and Redis as separate settings (discussion #317): the backend and the worker accept DB_HOST, DB_PORT (default 5432), DB_USER, DB_PASSWORD and DB_NAME, and REDIS_HOST, REDIS_PORT (default 6379) and REDIS_PASSWORD, and build the connection URLs themselves with every part percent-encoded, so a password with #, %, @, /, ? or : works as it is. Before, the compose files put DB_PASSWORD into DATABASE_URL unencoded, and such a password stopped the backend at start with Cannot read properties of undefined (reading 'searchParams'). DATABASE_URL and REDIS_URL are still supported and take precedence when set, for existing installs and managed databases; an empty value counts as unset. DB_HOST without DB_USER or DB_NAME stops the server at start with an error naming what is missing.

Performance

• Log volume on MongoDB from per-minute counts: MongoDB has no incremental aggregate, and counting logs in its time-series collection unpacks every one of them (2.8 seconds for the histogram and 0.6 seconds for the count of 5 million logs). LogTide now keeps per-minute counts per project, level and service next to the logs collection (logs_volume_minutes, with logs_volume_state and logs_volume_dirty), filled by the worker every minute up to five minutes ago (refreshLogVolume; about 5 seconds for 5 million logs the first time, a few milliseconds after that). Histograms and counts of searches that filter only by project, level or service read them for those minutes: 71 ms and 61 ms. Logs written with an older timestamp and deletes (retention, project purge) mark the minutes they touch before writing, and those minutes are read from the logs until the next refresh has counted the finished write, so the numbers match the logs; several workers can refresh at once without double counting. Batches of logs newer than two minutes skip this bookkeeping. If the counts cannot be read, the logs are read as before.
• Log volume on ClickHouse from a minute projection: the logs table gets the aggregate projection proj_minute_volume (log count per minute, project, level and service), added and materialized in the background on the next start (2 seconds for 36 million logs; parts not yet materialized are read as logs meanwhile). Histograms and counts of searches that filter only by project, level or service read it for the whole minutes of the range and read the partial minutes at the edges as logs: 22 ms instead of 0.84 seconds and 12 ms instead of 0.42 seconds over 36 million logs. ClickHouse keeps the projection inside every part, so inserts, merges and retention deletes keep the numbers exact.
• Log volume on TimescaleDB from a per-minute aggregate: the histogram and the total of the Logs page scanned every log of the time range (6 seconds for the histogram and 2 seconds for the count of 5 million logs). A new continuous aggregate logs_minute_stats (migration 075, one row per minute, project, level and service) now answers the whole minutes of the last 30 days when the search filters only by project, level or service; the edge minutes, the newest minutes, anything older and any minute TimescaleDB still lists as changed since its last refresh (late logs, backfills, retention deletes) are read from the logs table, so the numbers are exactly the same as before. The same requests take 30 ms and 6 ms. Searches with text, metadata, host, trace or session filters work as before. The aggregate is filled in the background by its refresh policy (about 6 seconds for 5 million logs, once); until then requests cost what they did.
• Log pages on MongoDB: the logs collection is a time-series collection, where only a sort on time alone runs one bucket at a time; the log list sorted on time and id, which made MongoDB sort the whole time range for every page (about 5 seconds for 5 million logs, 10 seconds without a filter). Pages now read on a time-only sort and order logs sharing a timestamp by id in memory, with the same order and cursors as before: about 0.4 to 0.6 seconds.
• Facets on big projects: the facet sidebar of the Logs page (GET /api/v1/logs/facets) counted every log of the time range and read each metadata key from every row, so on large projects it took longer than the request could wait: over 30 seconds for 36 million logs on ClickHouse, 23 seconds for 5 million on TimescaleDB and over 60 seconds for 5 million on MongoDB. Facets are now counted over the newest matching logs, at most 100,000 of them, taken from the newest 2,000,000 logs of each project in the range; smaller ranges are still counted in full. The same requests now take about 0.5, 1.3 and 2 seconds. The response gains sampleSize (how many logs were counted) and the sidebar says "Top values of the newest N of M matching logs" when the facets cover only part of them. The affected services in the alert preview come from the same facets. In @logtide/reservoir, facets() accepts sampleSize and scanLimit (at most those defaults) and returns sampleSize.
• Field discovery on big projects: GET /api/v1/logs/fields, which picks the metadata keys shown as facets, sorted the metadata of the whole time range on ClickHouse (8 to 15 seconds for 36 million logs) and on MongoDB (5 seconds for 5 million) before keeping its 1,000 log sample. It now reads only the newest logs it samples, one project at a time: about 50 ms and 0.3 seconds.
• Trace ingestion writes every trace summary of a request at once (#309): POST /v1/otlp/traces wrote one trace summary per trace of the request, one awaited database round trip after the other. A request with spans of 500 traces took 206 seconds on ClickHouse (every summary waited for the async insert flush), 0.4 seconds on TimescaleDB and 0.5 seconds on MongoDB; it now takes 0.3, 0.08 and 0.1 seconds, with one multi-row upsert on TimescaleDB, one bulkWrite on MongoDB and one insert on ClickHouse. Collectors exporting large batches to ClickHouse no longer time out and retry. @logtide/reservoir gains upsertTraces(traces); upsertTrace stays and does the same for one trace. Note that RATE_LIMIT_MAX (1,000 requests per minute per API key by default) also applies to POST /v1/otlp/traces: exporters that send one span per request (a simple span processor instead of the batch processor) are capped at about 16 spans per second.

Fixed

• Project health emails for projects that do not send all the time: a project that sleeps at night, between batch runs or over the weekend opened a new "stopped sending data" issue for every pause, and every one of them sent an opened and a resolved notification (email, webhook, PagerDuty and in-app). A project that sends nothing for a while is not necessarily broken, so the "stopped sending data" check is now opt-in per project: turn it on in Project settings > Notifications ("Alert when this project stops sending data") for projects that should send data all the time, and it is a warning instead of critical. To know whether a service is up, use an uptime monitor (a heartbeat your app pings, or a log heartbeat on one service). On upgrade, projects whose silent threshold was changed from the default 60 minutes keep the check on (migration 077); on every other project it is off, and an open "stopped sending data" issue is resolved by the next health pass without a notification. The project page now shows a quiet project as "No recent data" instead of a passing "Receiving data". On top of that, any health check that opens again within 24 hours of its previous issue resolving is a recurrence: it is shown on the project page as usual but notified neither when it opens nor when it resolves, and a check that stayed quiet for more than 24 hours notifies again the next time it opens. The settings API gains silentEnabled (PUT /api/v1/projects/:id/notification-settings and the settings of GET /api/v1/projects/:id/health).
• Newest data missing from TimescaleDB charts and counters: the hourly and daily continuous aggregates were materialized-only and refreshed once an hour (hourly) or once a day (daily), one bucket behind. The metrics overview and the metric charts with a 1 hour or 1 day interval read only those aggregates, so metrics sent in the last one to three hours (up to two days for daily charts) did not show up there. The dashboard charts and counters read the aggregate before a cutoff (one hour or one day ago) and raw data after it, so the hours between the last refresh and the cutoff were missing from "Total logs today", the 24 hour log and detection charts and the top services (up to two hours, or most of a day for the top services). Migration 076 turns on real-time aggregation for every continuous aggregate (the rows not materialized yet are computed from the raw tables at query time) and refreshes them every 10 minutes (hourly) or every hour (daily), so that part stays small. The dashboard and the log search split aggregate and raw data at the start of the current UTC hour or day, so nothing is read twice. On a large install the first refresh after the upgrade materializes up to three days of data once in the background. After the upgrade these numbers go up on TimescaleDB installs: they now include the hours that were missing, so they are not comparable with what 1.6.0 showed.
• Incidents created by hand or from grouped detections had no "created" entry in their history: the incident timeline started with the first change instead of the creation. It now records who created the incident (no user for incidents grouped by the system). Incidents created before the upgrade keep their history as it is.
• "Add facet" was unreadable with many metadata fields: with more fields than the menu could fit, the rows shrank instead of scrolling and their text was cut in half. The rows keep their height and the list scrolls, and a filter at the top of the menu finds a field among hundreds.
• Client IP behind a reverse proxy: with the default TRUST_PROXY=false, every request that reached the backend through a proxy carried the proxy address (for example the Docker gateway 172.31.0.1), so the audit log showed the same IP for everyone and all clients shared the login, registration and password reset rate limits. TRUST_PROXY now also accepts the number of proxies in front of the backend (1 for one reverse proxy, 2 for Cloudflare in front of it) or a comma-separated list of proxy IPs and CIDRs; the client IP is then the address right before those proxies, so a client cannot fake it with its own X-Forwarded-For. true still trusts every hop, as before, and takes the left-most X-Forwarded-For entry, which the client controls. An invalid value stops the server at start with an error. Set it in your .env if LogTide runs behind a proxy (see .env.example).
• Dropdowns outside the design system: 47 dropdowns (metric aggregate/interval/service, trace and session filters, live tail size, monitor and status page forms, alert rule scope and renotify, SIEM filters, audit and delivery logs, PII masking scope, admin filters, invite role and others) were plain browser <select> elements, so they opened the operating system menu instead of the app's own, ignoring the theme and dark mode. They now use the same Select component as the rest of the app, with the same keyboard navigation.
• Traefik overlay did not start: docker/docker-compose.traefik.yml failed with container_name '' does not match pattern on current Docker Compose versions, and past that Traefik 3.2 cannot talk to Docker Engine 29 (which dropped the older API versions), so it found no service and answered 404 to everything. Its ports: [] also left the backend and frontend published on their own ports next to Traefik, and Traefik was not on the network of the services it routes to. The overlay now uses !reset (Docker Compose 2.24 or newer), Traefik 3.6 and the LogTide network, and sets TRUST_PROXY: "1" instead of "true", so the client IP is the one Traefik saw rather than whatever the client put in its own X-Forwarded-For. Use 2 there when Cloudflare or another proxy sits in front of Traefik. Checked with two backends and two workers behind it.
• Alert rule errors showed only the status code: saving an alert rule the server rejected showed "HTTP 400" instead of the reason (the alerts client read message, the server sends error).
• PostgreSQL shared memory in the Docker Compose files: Docker gives containers 64 MB of /dev/shm, which is too little for parallel queries and VACUUM on large TimescaleDB tables (could not resize shared memory segment ... No space left on device), so they failed or ran without parallel workers. The postgres service of every compose file now sets shm_size: 1gb (an upper bound, not memory reserved up front). Existing installs pick it up when the container is recreated (docker compose up -d).
• ClickHouse trace summaries were wrong with the async buffer (#309): the ClickHouse engine recomputed a trace's span count, time range and error flag from the spans table, but with RESERVOIR_BUFFER_ENABLED=true the spans of the request are usually still in the buffer at that point, so traces showed too few spans and a too short duration. The summary no longer reads the spans table on any engine: each request writes the summary of the spans it carried and the engine merges them, so the result does not depend on when the spans land.
• Late spans cleared the root operation of a trace on MongoDB and ClickHouse: a request that did not carry the root span (for example a child span exported after the root) overwrote the trace's root service and operation with empty values, so the trace list showed no operation for it. They are now kept once known, as TimescaleDB already did. Traces affected before the upgrade keep their missing values.
• BullMQ workers now apply job retention too: BullMQ falls back to the worker's removeOnComplete / removeOnFail for jobs that carry none of their own, and the workers had none, so such a job was kept in Redis forever. Workers now use the same retention as the queues (keep 100 completed jobs for 1 hour, 50 failed jobs for 24 hours). Jobs added through LogTide already carried this since 1.0.3; this covers the remaining paths.

Changed

• ClickHouse traces table migration (runs once on start): on ClickHouse the traces table moves from ReplacingMergeTree to AggregatingMergeTree, which merges the partial summaries of a trace. The first start of 1.6.1 copies the existing table into the new layout, checks that the trace and span totals match, swaps the tables and keeps the previous one as traces_backup_<timestamp>; drop it once you are happy with the upgrade. In our tests 20 million traces took about 10 seconds and 300 MiB of ClickHouse memory; the copy needs free disk for a second copy of the table. TimescaleDB and MongoDB need no migration. Before upgrading a ClickHouse install, stop every backend and worker of the older version: they keep writing to the old table, which makes the check fail and the migration stop without changing anything until they are gone. When the backend and the worker start together, one of them migrates and the others wait for it (at most 30 minutes). If the process that migrates dies, the next start finishes a swap that was already under way; if it died during the copy, the others refuse to start with an error naming it instead of copying the table twice. The traces table is unchanged in that case: check that no LogTide process is still running the migration, run DROP TABLE traces_migration_lock and start again. To go back to 1.6.0, run RENAME TABLE traces TO traces_1_6_1, traces_backup_<timestamp> TO traces with LogTide stopped; traces written after the upgrade are then only in traces_1_6_1.
• Docker Compose files: the backend and worker environment, the same in both, is now one shared x-backend-env block, so the two cannot drift apart. Both services get the database and Redis as separate settings (see Added) instead of a DATABASE_URL built in the compose file, and the redis service gets its password through the environment instead of its command line, so a Redis password with spaces, #, ; or & works too. Nothing changes in .env. If you edited the compose file to point at an external database, set DATABASE_URL (or DB_HOST) in the x-backend-env block. install.sh no longer writes a DATABASE_URL line, which the compose file never read.