diff --git a/CHANGELOG.md b/CHANGELOG.md index f9007e99..db2ef8a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,56 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.2.0] - 2026-07-23 + +### Security +- **Bumped echarts to 6.1.0** (Dependabot `GHSA-fgmj-fm8m-jvvx` / `CVE-2026-45249`). Versions before 6.1.0 render a raw HTML `series.data[i].name` through an `innerHTML` sink into the built-in tooltip of a Lines series when no custom `tooltip.formatter` is set, bypassing the automatic HTML escaping the built-in formatters otherwise apply and allowing script execution. The frontend does not use the Lines series, so LogTide was not exposed in practice, but the dependency is upgraded to clear the advisory and prevent future exposure. +- **Bumped `protobufjs` to 7.6.5** (Dependabot `GHSA-j3f2-48v5-ccww`, MODERATE): versions `>=7.5.0 <=7.6.4` (and the 8.x line up to 8.6.5) can enter an infinite loop while parsing `.proto` option syntax, hanging the thread (DoS). LogTide loads OTLP protobuf via `protobufjs` in the backend ingestion path; it never parses untrusted `.proto` text at runtime, so real exposure was limited, but the root pnpm override floor was raised from `>=7.6.3 <8` to `>=7.6.5 <8` to clear the advisory. Stays on the 7.x line (the `<8` bound is intentional). +- **Bumped `brace-expansion` to 5.0.7** (Dependabot `GHSA-3jxr-9vmj-r5cp` / `CVE-2026-13149`, HIGH): versions `>=3.0.0 <5.0.7` expand consecutive non-expanding `{}` groups in exponential time, so a short all-ASCII input (~90 bytes / 30 groups) can block the calling thread for minutes (DoS). Pulled in transitively (via glob/minimatch in tooling); the root pnpm override floor was raised from `>=5.0.6` to `>=5.0.7`. + +### Changed +- **Login failures classified by typed error code, not message strings** (auth provider follow-up): `authenticationService.authenticateWithProvider` now throws a typed `AuthError` carrying the provider's `AuthErrorCode` instead of a plain `Error` built from the user-facing message, and `POST /auth/login` maps that code to the HTTP status and audit reason. Previously the route matched exact message strings, so rewording any of them would have silently turned a wrong-password 401 into a 500 and dropped its `auth.login_failed` audit row. A login while the local provider is disabled or removed now returns a clean `503` (reason `provider_unavailable`) instead of a raw 500. A dedicated `AuthErrorCode.SSO_REQUIRED` distinguishes the SSO-only account case from a wrong password so the audit reason is derived from the code. The HTTP response body is unchanged (still the provider message), so anti-enumeration behavior is untouched. The redundant `usersService.login` (a parallel local-auth path with no production callers that never received the enumeration hardening) was removed and its tests consolidated onto the provider path and the shared `createTestSession` helper. +- **Local login routed through the auth provider registry** (#266, community contribution by @Justy116): `POST /auth/login` now authenticates via the same provider path used by OIDC and LDAP instead of the parallel `usersService.login` implementation, and registration auto-login rides that path too. If the auto-login after signup fails transiently, the endpoint returns 201 with the created user and no `session` (the account stays recoverable via a normal login) instead of propagating a 500 after the user row is already committed. The local provider also gained user-enumeration hardening: the password is verified before the disabled-account check, so a wrong password returns the same generic error whether or not the account is disabled, and only a correct password reveals the disabled state. New `docs/architecture/auth.md` documents the provider architecture. Follow-up fixes landed with the merge: the register page now handles the sessionless 201 (redirects to login with a "please sign in" toast instead of throwing), the disabled-account and SSO-required login rejections are now recorded as `auth.login_failed` audit events with a `reason` in the metadata (previously only wrong-password failures were audited; the reason never appears in the HTTP response, so anti-enumeration behavior is unchanged), and a failed post-registration auto-login is now logged through internal observability instead of being swallowed silently. Test-suite hardening that the registry routing made necessary: two auth test files deleted every `auth_providers` row including the migration-seeded `local` provider, which (now that login resolves through the registry) broke every login-based test that ran after them, masked intermittently by the registry's 5-minute cache; they now preserve the seeded row and the test setup re-seeds it defensively, since tracked migrations never re-run + +### Fixed +- **Persisted time-range / current-project stores keep their value when localStorage is blocked** (pre-release hardening): `timeRangeStore.get()` and `currentProjectStore.get()` re-read localStorage on every call, so in a session where localStorage is unavailable (private mode, disabled, quota) a `set()` still updated subscribers but `get()` kept returning null, breaking the intended cross-page persistence. Both stores now hold an in-memory source of truth (hydrated from localStorage once) that `get()` returns, so a selection survives within the session regardless of localStorage. +- **Digest dispatch passes `last_sent_at` through as a Date** (pre-release hardening): the hourly dispatch cast the already-`Date` column to `string` and re-parsed it (`new Date(x as unknown as string)`), a needless round-trip that risked locale-dependent parsing on the double-fire guard. It now passes the Date straight through. +- **Error-group merge is now scoped to a single project** (pre-release hardening of the merge/duplicate feature added in this release): error groups are unique per `(organization_id, project_id, fingerprint)`, so the same exception type/message/fingerprint legitimately exists in sibling projects of one org. `findDuplicateErrorGroups`, `mergeErrorGroups` and the exception-retag step scoped only by `organization_id`, so the duplicate list could surface another project's group, a merge could delete it, and the retag (an `UPDATE exceptions SET fingerprint=... WHERE organization_id=? AND fingerprint IN (...)`) could rewrite a sibling project's exceptions that shared the same fingerprint string, corrupting them. All three are now scoped to the target group's `project_id` (null-safe). The `merge_key` backfill in migration 055 also now prefers the source-mapped frame (`original_file`/`original_function`), matching `FingerprintService.topAppFrame()`, so backfilled keys line up with the keys the runtime computes for new occurrences. Cross-project isolation regression tests added. +- **A bad client-side telemetry key logged the user out on every page view** (UX audit): the global 401 interceptor treated a 401 from any `/api/v1/` request (other than `/auth/`) as an expired session and bounced the user to `/login`. But ingestion/telemetry endpoints (`/api/v1/ingest`, `/v1/otlp/*`, inbound `/api/v1/receivers/*`) authenticate with an API key or DSN, not the session, so a 401 there means a bad telemetry key, not an expired session. The client-side SDK (`hooks.client.ts`) captures a log on every navigation; if its DSN key is wrong or revoked, each page view returned 401 and the interceptor logged the user out in a loop. Those endpoints are now excluded from the session-expiry logout. +- **Silent PII rule toggles** (UX audit): toggling a PII masking rule on/off or changing its action (mask/redact/hash) updated the rule but showed no confirmation, unlike every other action in that page (create/update/delete all toast). Both now show a success toast, matching the rest of the app's consistent action feedback. +- **Inconsistent time-range selectors across pages** (UX audit): the preset time-range controls on Security, Usage and Metrics were each hand-rolled with different markup (button groups and a dropdown with different styling). They now share one `TimeRangeButtons` segmented component, so they look and behave identically. Logs and Traces keep the richer time-range popover (they support custom ranges); the preset-only pages are now visually consistent. +- **"Total Logs Today" and "Error Rate" collapsed to the last hour** (UX audit): the dashboard stats (project overview cards and custom-dashboard single-stat panels, both via `dashboardService.getStats`) read "today" from the hourly continuous aggregate for everything but the last hour, backfilling only the last hour from raw. When the aggregate is not kept warm (a refresh policy that is not running, seeded/backfilled data, or a fresh install), the older part of today is missing from it, so the total and the error rate silently collapsed to roughly the last hour. Verified live on the hosted demo: 24h = ~1,500 logs, last hour = ~67, yet "Total Logs Today" read ~60 and Error Rate 0.0%. Low-volume organizations now compute today/yesterday from an exact raw count (cheap over the bounded window and independent of aggregate health); high-volume instances keep the fast aggregate path, where live ingestion keeps the aggregate current. +- **The same error split into several error groups** (UX audit): Node.js runtime frames use the `node:` scheme (e.g. `node:internal/process/task_queues`), which was not in the exception detector's library-pattern list, so `isAppCode` classified them as application code. Async stack traces vary in which internal frames they carry, so those frames leaked into the stack fingerprint and the same logical error (observed live: one `TypeError: Cannot read properties of undefined (reading 'selectFrom')` at `products/routes.js:130` fanned out into 5+ groups of 60/48/46/1/1 occurrences). `node:` frames are now treated as library code: excluded from the fingerprint (so recurrences of one error collapse into a single group) and no longer shown under "Show App Code Only". Fingerprints for errors whose stacks included `node:` frames change, so existing groups re-consolidate going forward rather than retroactively. Regression tests added. +- **Sidebar flashed "Select organization" on a hard reload** (UX audit): the organization store only persisted the selected org's id and started with `currentOrganization: null`, so a full page load of any dashboard sub-route rendered "Select organization" in the sidebar until the org list finished fetching. The store now caches the whole selected organization object and hydrates its initial state from it, so the org name renders immediately and is reconciled with the authoritative list once it loads (falling back cleanly if the cached org no longer exists). `clear()` was also fixed to reset to an empty state instead of the module-load initial state, so logout no longer restores the cached org. +- **Project overview stat cards showed wrong trend units and a mislabeled window** (UX audit): the shared `StatsCard` hardcoded every trend as "N% from last period", so the "Active Services" card, whose trend is an absolute count delta (services today minus yesterday), rendered a plain count with a bogus "%" ("+1.00% from last period" for one new service). The card now takes an optional trend `unit` and `label`: Active Services shows "+1 vs yesterday", Error Rate shows its change in percentage points ("pp"), and Total Logs/Throughput keep percentages with accurate "from yesterday"/"from last hour" labels. Separately, the Error Rate card described itself as "last 24h" while the value it shows is today's rate (since local midnight); the description now reads "Error rate today" to match the number. +- **Error groups always showed "unknown" affected service on ClickHouse and MongoDB** (UX audit): the `error_groups` service attribution was computed by a Postgres trigger that resolved the service with `SELECT service FROM logs WHERE id = NEW.log_id`. On non-TimescaleDB reservoir backends the ingested logs never land in the Postgres `logs` table (there is deliberately no foreign key on `exceptions.log_id`, "no FK due to hypertable"), so the lookup returned NULL and every error group was attributed to `unknown`, regardless of the real emitting service. The ingestion path already knows the service (`log.service`), so it is now carried on the `exceptions` row (new nullable `service` column, migration 054) and the trigger prefers it (`NEW.service`), falling back to the logs lookup on TimescaleDB and only then to `unknown`. New occurrences also merge the real service into existing groups via the ON CONFLICT path, so groups previously stuck on `unknown` recover as they recur. Multi-engine regression tests added. +- **Monitoring page used unstyled native ``, which looked out of place next to the custom dropdowns used everywhere else in the app. All six now use the shared `Select` component, so styling, focus, and open/close behavior match the rest of LogTide. +- **Metrics Explorer showed duplicate project and time-range selectors** (UX audit): the Explorer tab's "Filters" card repeated the project and time-range selectors that already live in the always-visible page header (both bound to the same state), so the tab presented two of each control at once. The Filters card now keeps only the Explorer-specific metric selector; project and time range are driven solely by the header controls. +- **Projects page browser tab title said "Dashboard"** (UX audit): the `/dashboard/projects` list page hardcoded `Dashboard - LogTide`, so the browser tab and history entry read "Dashboard" instead of "Projects". Now titled "Projects - LogTide", matching every sibling page. +- **Members "Joined" column mixed absolute and relative dates** (UX audit): the settings members table used a local formatter that showed "Today"/"Yesterday"/"N days ago" for recent joins but flipped to an absolute `M/D/YYYY` date after a week, so the column mixed two formats side by side. It now shows a single consistent short date ("Apr 12, 2026") with the relative time in a hover tooltip, via a shared `formatDateShort` helper. The bespoke local formatter was removed. +- **Ingestion health counters no longer depend on the metering toggle** (#279). `METERING_ENABLED=false` silently suppressed every `ingestion.*` counter, including `ingestion.pii_rejected`, the safety counter for records dropped because PII masking failed. That toggle governs usage and resource metering, while the `ingestion.*` counters are operational health signals already excluded from tenant usage breakdowns by a prefix filter, so they now bypass it. The buffer hard cap still applies to them: that is a memory valve, not a feature switch. + +### Added +- **Custom time range now uses a calendar with explicit time inputs** (UX audit): the log/trace time-range picker's "Custom" option replaced the two native `datetime-local` inputs (an inconsistent, browser-styled widget) with a proper in-app range calendar (new shadcn-style `RangeCalendar` wrapping bits-ui, so it matches the rest of the UI) plus dedicated "From time" / "To time" inputs for choosing the hour. The stored value stays `YYYY-MM-DDTHH:mm`, so URL sync, the shared time-range store, and deep links are unchanged. Adds the `@internationalized/date` dependency (already used by bits-ui). +- **Auto-merge duplicate error groups at ingestion**: builds on the manual merge below so new occurrences stop creating duplicates in the first place. Alongside the exact stack fingerprint, each group now has a coarser `merge_key` (exception type + normalized message + the top application frame). When an incoming exception has no exact fingerprint match but shares a `merge_key` with an existing group in the same project, it folds into that group instead of spawning a duplicate; errors thrown from different sites, or with different messages once dynamic tokens (numbers, UUIDs, hex, emails, quoted strings) are masked, stay separate. The key is computed by a single Postgres function `logtide_merge_key` used by the trigger, the runtime fold lookup, and a one-time backfill of existing groups (migration 055), so there is no SQL/JS normalization to keep in sync; library-only exceptions (no app frame) get a null key and are never auto-merged. The manual "Merge N" banner remains for historical splits. DB-backed tests cover fold, throw-site separation, dynamic-message folding, and the library-only null case. +- **The time window is remembered across logs and traces** (UX audit): picking a range (or a custom window) on Logs now carries over to Traces and vice versa, via a shared `timeRange` store persisted in localStorage. Each page validates the stored preset against what it supports and falls back to its default otherwise; deep links with explicit from/to still win. Metrics/Security/Usage keep their own preset vocabularies. +- **The selected project is remembered across pages** (UX audit): Metrics, Traces and Monitoring each defaulted to "the first project" independently, so moving between them kept resetting which project you were looking at. A shared `currentProject` store now persists the last project you selected (localStorage) and those pages default to it when it is still valid, falling back to the first project otherwise. Deep links with an explicit project still win. +- **Recent searches in log search** (UX audit): the log search page remembers your recent query terms (in localStorage) and shows them as one-click chips under the search bar when the box is empty, with a Clear action. Capped at the last 8, deduplicated. +- **Dashboard auto-refresh** (UX audit): the custom dashboard header has an auto-refresh control (off / 30s / 1m / 5m) that periodically refreshes all panels of the active dashboard. The choice persists in localStorage and pauses automatically while editing the layout so a refresh cannot clobber unsaved edits. +- **Row density toggle in log search** (UX audit): a Cozy/Compact toggle in the table toolbar (styled as an outline button to match the neighbouring Columns control) tightens row padding for scanning more logs at once, persisted in localStorage. (Custom columns were already persisted per project.) +- **Error regression detection** (UX audit): when an error group that was marked resolved starts occurring again, it is now automatically reopened (status back to open, resolution cleared) and its notification is flagged as a regression, with distinct wording across the in-app notification ("Regression: ..."), the email subject ("[Regression] ..."), and the webhook envelope (high severity plus an `is_regression` field). Previously a resolved error could recur silently and stay hidden in the resolved bucket. Job + wording covered by tests. +- **Merge duplicate error groups** (UX audit): complements the fingerprint fix above for groups that already split before it landed. The error detail page detects other groups with the same exception type and message and offers a one-click "Merge N" that folds them into the current group: their exceptions are reassigned to the target fingerprint (so trend and logs stay complete), occurrence counts and affected services are summed/unioned, and first/last-seen are widened, all in one org-scoped transaction. New `GET /api/v1/error-groups/:id/duplicates` and `POST /api/v1/error-groups/:id/merge` (membership-checked, cross-org merges refused). Service + org-isolation tests added. +- **Shareable log search URLs** (UX audit): the log search page now writes its active filters back to the URL (query text, project, service, levels, trace/session id, and a custom time range), so a search can be bookmarked or sent to a teammate and reopened exactly as filtered. It reuses the existing (idempotent) deep-link reader, uses `replaceState` so it does not spam history, and skips the write when nothing changed. (The errors page already synced its filters; traces already accepts deep links.) +- **Command palette searches your data, not just navigation** (UX audit): typing in the palette now offers "Search logs for ...", "Search errors for ...", and, when the text looks like a 16/32-char hex trace id, "Open trace ...". The log search page learned a `q` query param and the traces page learned a `traceId` param so these deep links land pre-filled. +- **Duplicate monitors and alert rules** (UX audit): the monitoring list has a Duplicate action that opens the New Monitor form pre-filled from an existing monitor (name suffixed "(copy)"), and the alerts list has a Duplicate action that opens the builder pre-filled with the full rule (service, levels, threshold, time window, rate-of-change settings, metadata filters and notification channels). The alert builder's prefill was generalized into a single `AlertBuilderPrefill` object shared by duplicate and create-from-error. +- **"Create alert" from an error group** (UX audit): the error detail page has a Create alert button that opens the alert builder prefilled from the error, the name seeded from the exception type, levels set to error/critical, and the service filled in when the group is attributed to a real service. The builder accepts optional prefill props and the alerts page reads them from query params (then strips them so a refresh does not reopen the dialog). +- **Metrics empty state now shows copy-ready OTLP setup** (UX audit): the "No metrics found" state was a bare icon and one line. It now mirrors the traces empty state with a proper onboarding panel: Node.js/Python/Go OpenTelemetry metric-exporter snippets (pointing at `/v1/otlp/metrics`) with copy buttons, quick links to get an API key and the OTLP/OpenTelemetry docs, and a preview of what metrics unlock. +- **Theme "System" option** (UX audit): the theme control is now a Light / Dark / System menu instead of a two-state toggle. "System" follows the OS color scheme and tracks it live (a `prefers-color-scheme` change flips the app without a reload). The stored preference persists across sessions; existing consumers still read the resolved light/dark theme, and the previous `toggle()`/`set()` helpers keep working. +- **Search term highlighting and relative-time tooltips in log search** (UX audit): in full-text mode the matched query terms are now highlighted in the message column (rendered as plain text segments, never `{@html}`, so no markup injection), and each log's absolute timestamp carries a relative-time tooltip ("3 minutes ago") on hover. +- **Clock skew detection for spans and metrics** (follows #279). Extends the log-side detection to OTLP span and metric ingestion: a span whose `end_time`, or a metric data point whose `time`, sits more than 24 hours in the past or more than 5 minutes ahead of the server clock is counted and surfaced on the project overview, never rejected. Spans are observed on `end_time` rather than `start_time` on purpose: a span that STARTED 27 hours ago is routinely legitimate, since spans are exported at end and batch jobs, long transactions and long-poll connections all produce old start times. Clock skew shifts both ends together while a long span moves only its start, so `end_time` is what separates a broken clock from a slow operation. Unlike the log threshold, which is exactly the maximum alert `time_window` and therefore an exact statement, the span and metric thresholds are a judgement call: no alert rules run over them, and the harm is that the data is invisible in the time windows of trace views and dashboards. Counters are `ingestion.span_timestamp_skew` and `ingestion.metric_timestamp_skew`, kept under the `ingestion.` prefix because both the tenant usage-breakdown exclusion and the metering-toggle bypass key off it literally. The project ingestion health endpoint now reports skew per signal, and the banner carries per-signal copy: the log wording ("cannot trigger an alert", "omit the time field") is false for spans and metrics. +- **Ingestion clock skew detection** (#279). Logs whose client-supplied `time` is more than 24 hours in the past or more than 5 minutes ahead of the server clock are counted and surfaced, on the project overview page for tenants and in admin ingestion health for operators. Such logs are stored and searchable but can never be counted by a threshold alert rule, whose largest window is 24 hours, which previously made a misconfigured shipper look like an alerting bug: ingestion worked, rules were enabled, channels tested fine, and no alert ever fired. The 24 hour threshold is derived rather than chosen, being the maximum configurable alert `time_window`. The logs are never rejected, so backfilling historical data stays valid, and detection is always on with no configuration to get wrong. Detection runs inside the existing record-building pass in `ingestLogs`, the single choke point for every log path (batch, Fluent Bit, OTLP, receivers), at a cost of one subtraction per record. +- **Scheduled email digest reports, completed and enabled** (#154): the digest feature whose foundation landed in #209 (and shipped disabled since 1.0.0-beta) is now finished and active. The report grew from log volume only to five sections, each with a quiet empty state: log volume with period-over-period trend, top 5 services by error+critical count with delta vs the previous period (engine-agnostic via `reservoir.topValues`, so TimescaleDB/ClickHouse/MongoDB all work), new error groups first seen in the period (`error_groups.first_seen`), a security summary (windowed detection totals and top triggered Sigma rules read from raw `detection_events` to avoid continuous-aggregate staleness, plus a point-in-time open/investigating incident count), and monitor uptime (aggregated from `monitor_uptime_daily`, section omitted for orgs without monitors). Emails are now real HTML built on the shared `email-templates.ts` component library (inline CSS, escaped user-controlled strings, en-US number formatting) with the full report duplicated in the plaintext fallback, and the base template footer learned an `unsubscribeUrl` variant so digest recipients (who may have no account) get a one-click unsubscribe instead of the channel-settings link. Scheduling was redesigned: instead of one cron per organization registered at worker boot (which could never pick up config changes at runtime, because graphile-worker cron items are fixed once the runner starts), a single static hourly `digest-dispatch` cron sweeps enabled `digest_configs`, matches `delivery_hour`/`delivery_day_of_week` against the current UTC hour, and enqueues `digest-generation` jobs with hour-keyed dedup keys and a `last_sent_at` double-fire guard (migration 053), so config CRUD is live on both queue backends with no restart. New management surface: session-auth CRUD under `/api/v1/digests` (config upsert/delete, recipient add/remove/resubscribe with token rotation on resubscribe, membership for reads and owner/admin for writes, audit-logged as `digest.*`, recipients capped by a new `digests.max_recipients` capability with the canonical 4-case tests), a public `POST /api/v1/digests/unsubscribe` where the 32-byte random token is the credential (idempotent, returns a masked email), an "Email Digests" settings page (schedule card with UTC delivery time and recipient management) and a public `/unsubscribe` page consumed by the email footer link. The worker-side disable comment from the beta is gone: the dispatch cron registers at boot + ## [1.1.0] - 2026-07-08 ### Added diff --git a/README.md b/README.md index 286512d1..dc7b56d3 100644 --- a/README.md +++ b/README.md @@ -16,14 +16,14 @@ Coverage Docker Artifact Hub - Version + Version License Status
-> **🌊 LogTide 1.1.0 (public beta):** unified **Logs, Traces & Metrics** with a built-in **SIEM**, multi-engine storage (TimescaleDB / ClickHouse / MongoDB), uptime monitoring, parsing pipelines, and custom dashboards. +> **🌊 LogTide 1.2.0 (public beta):** unified **Logs, Traces & Metrics** with a built-in **SIEM**, multi-engine storage (TimescaleDB / ClickHouse / MongoDB), uptime monitoring, parsing pipelines, and custom dashboards. --- @@ -124,7 +124,7 @@ We host it for you. Perfect for testing. [**Sign up at logtide.dev**](https://lo --- -## ✨ Core Features (v1.1.0) +## ✨ Core Features (v1.2.0) ### Monitoring, Pipelines & Dashboards * 🩺 **Uptime Monitoring & Status Pages:** HTTP/TCP/heartbeat monitors with configurable thresholds, auto-created SIEM incidents on failure, scheduled maintenances, and public Uptime-Kuma-style status pages per project. diff --git a/docs/architecture/auth.md b/docs/architecture/auth.md new file mode 100644 index 00000000..f48da0a8 --- /dev/null +++ b/docs/architecture/auth.md @@ -0,0 +1,417 @@ +# Authentication Architecture + +Logtide's authentication system is built around an `AuthProvider` interface. Every authentication +method — local email/password, OIDC, LDAP — is an implementation of this interface. The core +login path, session layer, and user-management code are unaware of *how* a user authenticated; +they only care about the resulting `userId`. + +--- + +## Table of Contents + +1. [Core Concepts](#core-concepts) +2. [The `AuthProvider` Interface](#the-authprovider-interface) +3. [Provider Registry](#provider-registry) +4. [Database Schema](#database-schema) +5. [Session Model](#session-model) +6. [Built-in Providers](#built-in-providers) +7. [Writing a Custom Provider](#writing-a-custom-provider) +8. [Capability Gate](#capability-gate) +9. [What Is Not Abstracted](#what-is-not-abstracted) + +--- + +## Core Concepts + +| Concept | Location | Role | +|---------|----------|------| +| `AuthProvider` interface | `src/modules/auth/providers/types.ts` | Contract every provider must satisfy | +| `ProviderRegistry` | `src/modules/auth/providers/registry.ts` | Singleton that loads/caches provider instances | +| `AuthenticationService` | `src/modules/auth/authentication-service.ts` | Orchestrates provider delegation, user provisioning, session creation | +| `auth_providers` table | `migrations/010_auth_providers.sql` | Persistent provider configuration (type, slug, JSONB config) | +| `user_identities` table | `migrations/010_auth_providers.sql` | Maps each user to one or more provider identities | + +The flow for every login is: + +``` +HTTP route → AuthenticationService.authenticateWithProvider(slug, credentials) + └─ ProviderRegistry.getProvider(slug) + └─ provider.authenticate(credentials) + └─ findOrCreateUser(provider, result) ← user provisioning lives here + └─ createSession(userId) ← opaque token, provider-agnostic +``` + +--- + +## The `AuthProvider` Interface + +```typescript +// src/modules/auth/providers/types.ts + +export interface AuthProvider { + /** Provider type: 'local' | 'oidc' | 'ldap' (or your custom string) */ + readonly type: AuthProviderType; + + /** Full provider configuration row from auth_providers table */ + readonly config: AuthProviderConfig; + + /** + * Authenticate a user. + * - Local: receives { email, password } + * - LDAP: receives { username, password } + * - OIDC: not used (uses callback flow instead) + * - Custom: whatever shape you define + */ + authenticate(credentials: unknown): Promise; + + /** True for redirect-based flows (OIDC). False for credential-based flows. */ + supportsRedirect(): boolean; + + /** OIDC only: generate authorization URL + PKCE state. */ + getAuthorizationUrl?(redirectUri: string): Promise; + + /** OIDC only: handle provider callback and exchange code for tokens. */ + handleCallback?( + data: OidcCallbackData, + expectedNonce: string + ): Promise; + + /** Validate that config is structurally correct (called before saving). */ + validateConfig(): boolean; + + /** Optional: test the external connection (shown in admin UI). */ + testConnection?(): Promise<{ success: boolean; message: string }>; +} +``` + +`AuthenticationResult` carries the information `AuthenticationService` needs to look up or create +a user: + +```typescript +interface AuthenticationResult { + success: boolean; + providerUserId?: string; // stable external ID (OIDC 'sub', LDAP DN, email for local) + email?: string; + emailVerified?: boolean; // gates auto-linking; set to true only when provider asserts it + name?: string; + metadata?: Record; + error?: string; + errorCode?: AuthErrorCode; +} +``` + +--- + +## Provider Registry + +`ProviderRegistry` is a singleton (`providerRegistry`) that: + +1. Reads rows from `auth_providers` on first use (lazy init). +2. Creates provider instances via `createProvider(config)` — a factory switch on `config.type`. +3. Caches configs in Redis (5-minute TTL) and instances in memory. +4. Exposes `invalidateCache()` so the admin API can trigger a reload after config changes. + +```typescript +// src/modules/auth/providers/registry.ts (simplified) + +function createProvider(config: AuthProviderConfig): AuthProvider { + switch (config.type) { + case 'local': return new LocalProvider(config); + case 'oidc': return new OidcProvider(config); + case 'ldap': return new LdapProvider(config); + default: throw new Error(`Unknown provider type: ${config.type}`); + } +} +``` + +To add a new built-in provider type, add a `case` here and ship the class alongside it. Provider +configs are stored in the database, so no code change is needed to *configure* existing types at +runtime — only to add a new type. + +--- + +## Database Schema + +```sql +-- auth_providers: one row per configured provider instance +CREATE TABLE auth_providers ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + type VARCHAR(50) NOT NULL, -- 'local' | 'oidc' | 'ldap' + name VARCHAR(255) NOT NULL, -- display name + slug VARCHAR(255) NOT NULL UNIQUE, -- used in API routes: /auth/providers/:slug/... + enabled BOOLEAN NOT NULL DEFAULT true, + is_default BOOLEAN NOT NULL DEFAULT false, -- shown prominently on login page + display_order INT NOT NULL DEFAULT 0, + icon VARCHAR(100), + config JSONB NOT NULL DEFAULT '{}', -- provider-specific config (issuer URL, etc.) + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +-- user_identities: maps a user to an external identity +-- one user can have N identities (local + OIDC + LDAP simultaneously) +CREATE TABLE user_identities ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE, + provider_id UUID NOT NULL REFERENCES auth_providers(id) ON DELETE CASCADE, + provider_user_id VARCHAR(500) NOT NULL, -- OIDC 'sub', LDAP DN, email for local + metadata JSONB, -- cached claims / attributes + last_login_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE (provider_id, provider_user_id) +); +``` + +Migration `010_auth_providers.sql` seeds a `local` row in `auth_providers` and backfills +`user_identities` for every existing user (using `email` as `provider_user_id`). + +--- + +## Session Model + +Sessions are not changed by the provider abstraction. After any successful authentication: + +1. `AuthenticationService` calls `usersService.createSession(userId)`. +2. A row is inserted into `sessions` with an opaque 32-byte random token. +3. The token is returned to the client as a Bearer token. + +Sessions carry only `userId` and `expiresAt`. The provider that was used to authenticate is not +stored on the session. This means: + +- A user who logs in via OIDC gets exactly the same session as one who logs in locally. +- Provider-specific tokens (OIDC `access_token`, LDAP bind credentials) are **not** kept in the + session. They are consumed only during the authentication handshake. +- Revoking a provider (disabling its row in `auth_providers`) does not automatically expire + existing sessions. User management (disabling a user) invalidates sessions because session + validation checks `users.disabled`. + +--- + +## Built-in Providers + +| Type | Class | Credential shape | Redirect? | +|------|-------|-----------------|-----------| +| `local` | `LocalProvider` | `{ email, password }` | No | +| `oidc` | `OidcProvider` | N/A (callback flow) | Yes | +| `ldap` | `LdapProvider` | `{ username, password }` | No | + +Operators add provider instances via the admin UI (`/admin/auth/providers`) or the admin API +(`POST /api/v1/admin/auth/providers`). Multiple instances of the same type are supported (e.g., +two separate OIDC providers for different identity platforms). + +--- + +## Writing a Custom Provider + +Below is a complete, minimal example: a **magic-link** provider that authenticates users by +verifying a short-lived token passed as a credential. It demonstrates the minimum required to +satisfy the `AuthProvider` interface. + +### 1. Define the credential shape and config + +```typescript +// src/modules/auth/providers/magic-link-provider.ts + +import type { + AuthProvider, + AuthProviderConfig, + AuthenticationResult, +} from './types.js'; +import { AuthErrorCode } from './types.js'; +import { CacheManager } from '../../../utils/cache.js'; +import { db } from '../../../database/connection.js'; + +interface MagicLinkCredentials { + token: string; // the one-time token the user received by email +} + +interface MagicLinkConfig { + tokenTtlSeconds?: number; // stored in auth_providers.config +} + +export class MagicLinkProvider implements AuthProvider { + readonly type = 'magic_link' as const; + readonly config: AuthProviderConfig; + + private get providerConfig(): MagicLinkConfig { + return this.config.config as MagicLinkConfig; + } + + constructor(config: AuthProviderConfig) { + this.config = config; + } + + async authenticate(credentials: unknown): Promise { + const { token } = credentials as MagicLinkCredentials; + + if (!token) { + return { + success: false, + error: 'Token is required', + errorCode: AuthErrorCode.INVALID_CREDENTIALS, + }; + } + + // Tokens are stored in Redis as: magic_link: → email + const cacheKey = `magic_link:${token}`; + const email = await CacheManager.get(cacheKey); + + if (!email) { + return { + success: false, + error: 'Token is invalid or has expired', + errorCode: AuthErrorCode.INVALID_CREDENTIALS, + }; + } + + // One-time use: delete immediately after successful validation + await CacheManager.delete(cacheKey); + + // Look up the user (magic-link only works for existing accounts) + const user = await db + .selectFrom('users') + .select(['id', 'email', 'name', 'disabled']) + .where('email', '=', email.toLowerCase().trim()) + .executeTakeFirst(); + + if (!user) { + return { + success: false, + error: 'No account found for this email', + errorCode: AuthErrorCode.INVALID_CREDENTIALS, + }; + } + + if (user.disabled) { + return { + success: false, + error: 'This account has been disabled', + errorCode: AuthErrorCode.USER_DISABLED, + }; + } + + return { + success: true, + providerUserId: user.email, // stable external ID for this provider + email: user.email, + emailVerified: true, // magic-link implies ownership of the inbox + name: user.name, + }; + } + + supportsRedirect(): boolean { + return false; // credential-based, not redirect-based + } + + validateConfig(): boolean { + const ttl = this.providerConfig.tokenTtlSeconds; + if (ttl !== undefined && (typeof ttl !== 'number' || ttl < 60)) { + return false; + } + return true; + } + + async testConnection(): Promise<{ success: boolean; message: string }> { + return { success: true, message: 'Magic-link provider is always available' }; + } +} +``` + +### 2. Register the type in the factory + +```typescript +// src/modules/auth/providers/registry.ts — add one case + +function createProvider(config: AuthProviderConfig): AuthProvider { + switch (config.type) { + case 'local': return new LocalProvider(config); + case 'oidc': return new OidcProvider(config); + case 'ldap': return new LdapProvider(config); + case 'magic_link': return new MagicLinkProvider(config); // ← add this + default: throw new Error(`Unknown provider type: ${config.type}`); + } +} +``` + +### 3. Seed a provider row (migration or admin API) + +```sql +INSERT INTO auth_providers (type, name, slug, enabled, is_default, display_order, config) +VALUES ( + 'magic_link', + 'Magic Link', + 'magic-link', + true, + false, + 10, + '{"tokenTtlSeconds": 900}'::jsonb +); +``` + +Or via the admin API: + +```bash +curl -X POST /api/v1/admin/auth/providers \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ + "type": "magic_link", + "name": "Magic Link", + "slug": "magic-link", + "enabled": true, + "config": { "tokenTtlSeconds": 900 } + }' +``` + +### 4. Call the provider from a route + +The existing `POST /api/v1/auth/providers/:slug/login` route (for credential-based providers) +will automatically dispatch to your provider once the row exists in the database. No new route is +needed for credential-based flows. + +```bash +# A separate endpoint would issue the token to the user's inbox (not shown here), +# then the client presents it: +curl -X POST /api/v1/auth/providers/magic-link/login \ + -H 'Content-Type: application/json' \ + -d '{"token": "a3f9..."}' +``` + +`AuthenticationService.authenticateWithProvider('magic-link', { token })` resolves the rest: +it calls `MagicLinkProvider.authenticate()`, provisions the user if needed, and returns a session. + +--- + +## Capability Gate + +SSO providers (any type other than `local`) are gated behind the `auth.sso` capability: + +```typescript +// src/capabilities/registry.ts +'auth.sso': { + kind: 'boolean', + defaultEnabled: true, // enabled for all plans in OSS + description: 'Single sign-on / external auth provider selection', +} +``` + +The default is `true`, so all Logtide deployments can use external providers out of the box. +Cloud or enterprise distributions can override this per-organization via +`organization_entitlements`. + +--- + +## What Is Not Abstracted + +The following are intentionally **outside** the provider interface boundary: + +- **Session storage** — sessions remain server-side opaque tokens in the `sessions` table, + independent of how the user authenticated. +- **User management** — creating, disabling, and deleting users lives in `UsersService`, not in + providers. Providers only assert identity; they do not manage the user record. +- **SAML** — not yet implemented. The `AuthInput` union can be extended with a + `{ type: 'saml_assertion'; samlResponse: string }` branch when a SAML provider is added as a + separate module, without modifying this interface or the session layer. +- **Authorization** — what a user can do after login (roles, organization membership) is handled + by the capabilities and tenant isolation layers, not by auth providers. diff --git a/package.json b/package.json index b56f0384..29ab4ac1 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "logtide", - "version": "1.1.0", + "version": "1.2.0", "private": true, "description": "LogTide - Self-hosted log management platform", "author": "LogTide Team", @@ -17,7 +17,7 @@ "qs": ">=6.15.2", "devalue": ">=5.8.1", "picomatch": ">=4.0.4", - "brace-expansion": ">=5.0.6", + "brace-expansion": ">=5.0.7", "fast-xml-parser": ">=5.7.0", "fast-uri": ">=3.1.2", "minimatch": ">=10.2.3", @@ -25,7 +25,7 @@ "rollup": ">=4.59.0", "svelte": ">=5.55.7", "@sveltejs/kit": ">=2.60.1", - "protobufjs": ">=7.6.3 <8", + "protobufjs": ">=7.6.5 <8", "@protobufjs/utf8": ">=1.1.1", "kysely": ">=0.28.17 <0.29", "ws": ">=8.20.1", diff --git a/packages/backend/migrations/053_digest_last_sent.sql b/packages/backend/migrations/053_digest_last_sent.sql new file mode 100644 index 00000000..89c81cb2 --- /dev/null +++ b/packages/backend/migrations/053_digest_last_sent.sql @@ -0,0 +1,7 @@ +-- ============================================================================ +-- Migration 053: Digest dispatch bookkeeping +-- Adds last_sent_at to digest_configs. The hourly digest-dispatch cron uses it +-- as a double-fire guard and operators can see when a digest last went out. +-- ============================================================================ + +ALTER TABLE digest_configs ADD COLUMN IF NOT EXISTS last_sent_at TIMESTAMPTZ; diff --git a/packages/backend/migrations/054_exception_service_attribution.sql b/packages/backend/migrations/054_exception_service_attribution.sql new file mode 100644 index 00000000..d0467439 --- /dev/null +++ b/packages/backend/migrations/054_exception_service_attribution.sql @@ -0,0 +1,71 @@ +-- ============================================================================ +-- Migration 054: Engine-independent service attribution for error groups +-- The error-group trigger resolved a group's affected service with +-- `SELECT service FROM logs WHERE id = NEW.log_id`, but on ClickHouse and +-- MongoDB reservoir backends the ingested logs never land in the Postgres +-- `logs` table, so the lookup always returned NULL and every error group was +-- attributed to 'unknown'. The ingestion path already knows the service, so we +-- carry it on the exception row and have the trigger prefer it, falling back to +-- the logs lookup (TimescaleDB) and only then to 'unknown'. +-- ============================================================================ + +ALTER TABLE exceptions ADD COLUMN IF NOT EXISTS service TEXT; + +CREATE OR REPLACE FUNCTION update_error_group_on_exception() +RETURNS TRIGGER AS $$ +DECLARE + v_service TEXT; +BEGIN + -- Prefer the service carried on the exception (known at ingestion, works on + -- every storage engine); fall back to the Postgres logs table (TimescaleDB) + -- and finally to 'unknown'. + v_service := NEW.service; + + IF v_service IS NULL THEN + SELECT service INTO v_service + FROM logs + WHERE id = NEW.log_id + LIMIT 1; + END IF; + + v_service := COALESCE(v_service, 'unknown'); + + -- Insert or update error group + INSERT INTO error_groups ( + organization_id, + project_id, + fingerprint, + exception_type, + exception_message, + language, + occurrence_count, + first_seen, + last_seen, + affected_services, + sample_log_id + ) + VALUES ( + NEW.organization_id, + NEW.project_id, + NEW.fingerprint, + NEW.exception_type, + NEW.exception_message, + NEW.language, + 1, + NEW.created_at, + NEW.created_at, + ARRAY[v_service], + NEW.log_id + ) + ON CONFLICT (organization_id, COALESCE(project_id, '00000000-0000-0000-0000-000000000000'::UUID), fingerprint) + DO UPDATE SET + occurrence_count = error_groups.occurrence_count + 1, + last_seen = NEW.created_at, + affected_services = ( + SELECT ARRAY(SELECT DISTINCT unnest(array_cat(error_groups.affected_services, ARRAY[v_service]))) + ), + updated_at = NOW(); + + RETURN NEW; +END; +$$ LANGUAGE plpgsql; diff --git a/packages/backend/migrations/055_error_group_auto_merge.sql b/packages/backend/migrations/055_error_group_auto_merge.sql new file mode 100644 index 00000000..43e18c30 --- /dev/null +++ b/packages/backend/migrations/055_error_group_auto_merge.sql @@ -0,0 +1,127 @@ +-- ============================================================================ +-- Migration 055: Auto-merge error groups at ingestion +-- Stack fingerprints split one logical error into several groups when the deep +-- stack varies (source maps, async internal frames, different callers). We add a +-- coarser "merge key" (exception type + normalized message + the top application +-- frame) so a new occurrence folds into an existing group instead of creating a +-- duplicate. The key is computed by ONE Postgres function used by the backfill, +-- the trigger, and the runtime fold lookup, so there is no SQL/JS normalization +-- to keep in sync. See docs/superpowers/specs/2026-07-21-error-group-auto-merge-design.md +-- ============================================================================ + +-- Single source of truth for the merge key. IMMUTABLE so it can be used in a +-- WHERE clause against the merge_key index. Returns NULL when there is no app +-- frame, so library-only errors are never auto-merged. md5 (not a security +-- digest, just an internal grouping key) avoids a pgcrypto dependency. +CREATE OR REPLACE FUNCTION logtide_merge_key(p_type TEXT, p_message TEXT, p_top_frame TEXT) +RETURNS TEXT AS $$ +DECLARE + m TEXT; +BEGIN + IF p_top_frame IS NULL THEN + RETURN NULL; + END IF; + + m := COALESCE(p_message, ''); + -- UUIDs first (they contain hex runs the next step would otherwise match). + m := regexp_replace(m, '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', '', 'gi'); + -- Long hex runs (request ids, hashes, addresses). + m := regexp_replace(m, '\y[0-9a-f]{8,}\y', '', 'gi'); + m := regexp_replace(m, '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}', '', 'gi'); + m := regexp_replace(m, '''[^'']*''|"[^"]*"', '', 'g'); + m := regexp_replace(m, '\d+(\.\d+)?', '', 'g'); + m := btrim(regexp_replace(m, '\s+', ' ', 'g')); + + RETURN md5(p_type || E'\n' || m || E'\n' || p_top_frame); +END; +$$ LANGUAGE plpgsql IMMUTABLE; + +ALTER TABLE error_groups ADD COLUMN IF NOT EXISTS merge_key TEXT; +CREATE INDEX IF NOT EXISTS idx_error_groups_merge_key ON error_groups (organization_id, merge_key); + +-- Raw "file:function" of the first app frame, carried from the app (which has the +-- parsed frames) so the trigger can compute the group's merge key. +ALTER TABLE exceptions ADD COLUMN IF NOT EXISTS top_frame TEXT; + +CREATE OR REPLACE FUNCTION update_error_group_on_exception() +RETURNS TRIGGER AS $$ +DECLARE + v_service TEXT; + v_merge_key TEXT; +BEGIN + -- Service: prefer the value carried on the exception (works on every storage + -- engine), fall back to the Postgres logs table (TimescaleDB), then 'unknown'. + v_service := NEW.service; + IF v_service IS NULL THEN + SELECT service INTO v_service + FROM logs + WHERE id = NEW.log_id + LIMIT 1; + END IF; + v_service := COALESCE(v_service, 'unknown'); + + v_merge_key := logtide_merge_key(NEW.exception_type, NEW.exception_message, NEW.top_frame); + + INSERT INTO error_groups ( + organization_id, + project_id, + fingerprint, + exception_type, + exception_message, + language, + occurrence_count, + first_seen, + last_seen, + affected_services, + sample_log_id, + merge_key + ) + VALUES ( + NEW.organization_id, + NEW.project_id, + NEW.fingerprint, + NEW.exception_type, + NEW.exception_message, + NEW.language, + 1, + NEW.created_at, + NEW.created_at, + ARRAY[v_service], + NEW.log_id, + v_merge_key + ) + ON CONFLICT (organization_id, COALESCE(project_id, '00000000-0000-0000-0000-000000000000'::UUID), fingerprint) + DO UPDATE SET + occurrence_count = error_groups.occurrence_count + 1, + last_seen = NEW.created_at, + affected_services = ( + SELECT ARRAY(SELECT DISTINCT unnest(array_cat(error_groups.affected_services, ARRAY[v_service]))) + ), + merge_key = COALESCE(error_groups.merge_key, v_merge_key), + updated_at = NOW(); + + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +-- Backfill existing groups so historical duplicates can fold forward. +UPDATE error_groups g +SET merge_key = logtide_merge_key(g.exception_type, g.exception_message, tf.top_frame) +FROM ( + SELECT + eg.id AS group_id, + ( + -- Prefer the source-mapped frame, matching FingerprintService.topAppFrame() + -- so backfilled keys line up with keys the runtime computes for new + -- occurrences (which carry original_file/original_function when available). + SELECT COALESCE(sf.original_file, sf.file_path) || ':' || + COALESCE(sf.original_function, sf.function_name, '') + FROM exceptions e + JOIN stack_frames sf ON sf.exception_id = e.id + WHERE e.log_id = eg.sample_log_id AND sf.is_app_code = TRUE + ORDER BY sf.frame_index ASC + LIMIT 1 + ) AS top_frame + FROM error_groups eg +) tf +WHERE g.id = tf.group_id; diff --git a/packages/backend/package.json b/packages/backend/package.json index 5124cd30..d61d9942 100644 --- a/packages/backend/package.json +++ b/packages/backend/package.json @@ -1,6 +1,6 @@ { "name": "@logtide/backend", - "version": "1.1.0", + "version": "1.2.0", "private": true, "description": "LogTide Backend API", "type": "module", diff --git a/packages/backend/src/capabilities/registry.ts b/packages/backend/src/capabilities/registry.ts index 1d5ca1ca..ff671146 100644 --- a/packages/backend/src/capabilities/registry.ts +++ b/packages/backend/src/capabilities/registry.ts @@ -99,6 +99,11 @@ export const CAPABILITIES = { defaultLimit: null, description: 'Maximum inbound webhook receivers per organization', }, + 'digests.max_recipients': { + kind: 'limit', + defaultLimit: null, + description: 'Maximum digest email recipients per organization', + }, // Consumption quotas (OSS-permissive: null = unlimited). signal maps to #212 metering types. 'ingestion.max_bytes_monthly': { diff --git a/packages/backend/src/database/types.ts b/packages/backend/src/database/types.ts index 72c5e03a..ba9cbc95 100644 --- a/packages/backend/src/database/types.ts +++ b/packages/backend/src/database/types.ts @@ -641,6 +641,8 @@ export interface ExceptionsTable { fingerprint: string; raw_stack_trace: string; frame_count: number; + service: string | null; + top_frame: string | null; created_at: Generated; } @@ -689,6 +691,7 @@ export interface ErrorGroupsTable { resolved_by: string | null; affected_services: string[] | null; sample_log_id: string | null; + merge_key: string | null; last_notified_at: Timestamp | null; created_at: Generated; updated_at: Generated; @@ -1138,6 +1141,7 @@ export interface DigestConfigsTable { delivery_hour: number; delivery_day_of_week: number | null; enabled: Generated; + last_sent_at: Timestamp | null; created_at: Generated; updated_at: Generated; } diff --git a/packages/backend/src/lib/email-templates.ts b/packages/backend/src/lib/email-templates.ts index fce0953b..dcba239a 100644 --- a/packages/backend/src/lib/email-templates.ts +++ b/packages/backend/src/lib/email-templates.ts @@ -97,10 +97,13 @@ const colors = { interface BaseEmailOptions { preheader?: string; // Preview text shown in email clients + // When set, the footer swaps the channel-settings link for a one-click + // unsubscribe link (used by digest emails, where recipients may have no account) + unsubscribeUrl?: string; } function baseTemplate(content: string, options: BaseEmailOptions = {}): string { - const { preheader } = options; + const { preheader, unsubscribeUrl } = options; const frontendUrl = getFrontendUrl(); const logoUrl = getLogoUrl(); @@ -143,7 +146,9 @@ function baseTemplate(content: string, options: BaseEmailOptions = {}): string { Sent by LogTide

- Manage notification settings + ${unsubscribeUrl + ? `Unsubscribe from these reports` + : `Manage notification settings`}

@@ -833,3 +838,254 @@ Manage notifications: ${frontendUrl}/dashboard/settings/channels return { html, text }; } + +// ============================================================================ +// DIGEST REPORT EMAIL (#154) +// ============================================================================ + +export interface DigestEmailData { + organizationName: string; + frequency: 'daily' | 'weekly'; + periodLabel: string; + logVolume: { + current: number; + previous: number; + trend: string; + }; + topErrorServices: Array<{ + service: string; + errorCount: number; + previousCount: number; + delta: number; + }>; + newErrorGroups: Array<{ + exceptionType: string; + exceptionMessage: string; + occurrenceCount: number; + language: string; + }>; + security: { + totalDetections: number; + topRules: Array<{ ruleTitle: string; severity: string; count: number }>; + openIncidents: number; + }; + uptime: { + monitorCount: number; + overallUptimePct: number; + worstMonitors: Array<{ name: string; uptimePct: number }>; + } | null; + unsubscribeUrl: string; + dashboardUrl: string; +} + +function digestSectionTitle(title: string): string { + return ` + +

+ ${escapeHtml(title)} +

+ + `; +} + +function digestEmptyLine(message: string): string { + return ` + +

${escapeHtml(message)}

+ + `; +} + +function digestTable(headers: string[], rows: string[][]): string { + const headerCells = headers + .map( + (h, i) => + `${escapeHtml(h)}` + ) + .join(''); + const bodyRows = rows + .map( + (cells) => + `${cells + .map( + (c, i) => + `${escapeHtml(c)}` + ) + .join('')}` + ) + .join(''); + + return ` + + + ${headerCells} + ${bodyRows} +
+ + `; +} + +function formatDelta(delta: number): string { + if (delta > 0) return `+${delta.toLocaleString('en-US')}`; + return delta.toLocaleString('en-US'); +} + +export function generateDigestEmail(data: DigestEmailData): { html: string; text: string } { + const frequencyLabel = data.frequency === 'daily' ? 'Daily' : 'Weekly'; + const title = `${frequencyLabel} Digest`; + const quiet = data.logVolume.current === 0 && data.logVolume.previous === 0; + + const logVolumeSection = quiet + ? alertBox('No activity during this period. Your systems have been quiet.', 'info') + : infoBox([ + { label: 'Total Logs', value: data.logVolume.current.toLocaleString('en-US') }, + { label: 'Trend', value: data.logVolume.trend }, + { label: 'Previous Period', value: data.logVolume.previous.toLocaleString('en-US') }, + ]); + + const topServicesSection = + data.topErrorServices.length > 0 + ? digestTable( + ['Service', 'Errors', 'Previous', 'Delta'], + data.topErrorServices.map((s) => [ + s.service, + s.errorCount.toLocaleString('en-US'), + s.previousCount.toLocaleString('en-US'), + formatDelta(s.delta), + ]) + ) + : digestEmptyLine('No errors recorded in this period.'); + + const errorGroupsSection = + data.newErrorGroups.length > 0 + ? digestTable( + ['Error', 'Language', 'Occurrences'], + data.newErrorGroups.map((g) => [ + `${g.exceptionType}${g.exceptionMessage ? ': ' + truncate(g.exceptionMessage, 60) : ''}`, + g.language, + g.occurrenceCount.toLocaleString('en-US'), + ]) + ) + : digestEmptyLine('No new error groups appeared in this period.'); + + const securityRows = [ + { label: 'Detections', value: data.security.totalDetections.toLocaleString('en-US') }, + { label: 'Open Incidents', value: data.security.openIncidents.toLocaleString('en-US') }, + ]; + const securitySection = + data.security.topRules.length > 0 + ? infoBox(securityRows) + + digestTable( + ['Rule', 'Severity', 'Detections'], + data.security.topRules.map((r) => [r.ruleTitle, r.severity, r.count.toLocaleString('en-US')]) + ) + : infoBox(securityRows); + + const uptimeSection = data.uptime + ? digestSectionTitle('Uptime') + + infoBox([ + { label: 'Overall', value: `${data.uptime.overallUptimePct.toLocaleString('en-US')}%` }, + { label: 'Monitors', value: data.uptime.monitorCount.toLocaleString('en-US') }, + ]) + + (data.uptime.worstMonitors.length > 0 + ? digestTable( + ['Monitor', 'Uptime'], + data.uptime.worstMonitors.map((m) => [m.name, `${m.uptimePct.toLocaleString('en-US')}%`]) + ) + : '') + : ''; + + const html = baseTemplate( + card(` + ${header(title, { text: data.frequency, color: colors.info })} + ${divider()} + ${subtitle(`${data.organizationName} - ${data.periodLabel}`)} + ${timestamp()} + ${digestSectionTitle('Log Volume')} + ${logVolumeSection} + ${digestSectionTitle('Top Services by Errors')} + ${topServicesSection} + ${digestSectionTitle('New Error Groups')} + ${errorGroupsSection} + ${digestSectionTitle('Security')} + ${securitySection} + ${uptimeSection} + ${cta('View Dashboard', data.dashboardUrl)} + `), + { + preheader: quiet + ? `Quiet period for ${data.organizationName}` + : `${data.logVolume.current.toLocaleString('en-US')} logs, ${data.security.totalDetections.toLocaleString('en-US')} detections for ${data.organizationName}`, + unsubscribeUrl: data.unsubscribeUrl, + } + ); + + const lines: string[] = []; + lines.push(`LogTide ${frequencyLabel} Digest`); + lines.push(`Organization: ${data.organizationName}`); + lines.push(`Period: ${data.periodLabel}`); + lines.push(''); + lines.push('='.repeat(50)); + lines.push(''); + lines.push('LOG VOLUME'); + lines.push('-'.repeat(10)); + if (quiet) { + lines.push('No activity during this period.'); + lines.push('Your systems have been quiet.'); + } else { + lines.push(`Total logs: ${data.logVolume.current.toLocaleString('en-US')}`); + lines.push(`Trend: ${data.logVolume.trend}`); + lines.push(`Previous period: ${data.logVolume.previous.toLocaleString('en-US')}`); + } + lines.push(''); + lines.push('TOP SERVICES BY ERRORS'); + lines.push('-'.repeat(22)); + if (data.topErrorServices.length > 0) { + for (const s of data.topErrorServices) { + lines.push( + `${s.service}: ${s.errorCount.toLocaleString('en-US')} errors (previous: ${s.previousCount.toLocaleString('en-US')}, delta: ${formatDelta(s.delta)})` + ); + } + } else { + lines.push('No errors recorded in this period.'); + } + lines.push(''); + lines.push('NEW ERROR GROUPS'); + lines.push('-'.repeat(16)); + if (data.newErrorGroups.length > 0) { + for (const g of data.newErrorGroups) { + const message = g.exceptionMessage ? `: ${truncate(g.exceptionMessage, 80)}` : ''; + lines.push(`${g.exceptionType}${message} (${g.language}, ${g.occurrenceCount.toLocaleString('en-US')} occurrences)`); + } + } else { + lines.push('No new error groups appeared in this period.'); + } + lines.push(''); + lines.push('SECURITY'); + lines.push('-'.repeat(8)); + lines.push(`Detections: ${data.security.totalDetections.toLocaleString('en-US')}`); + lines.push(`Open incidents: ${data.security.openIncidents.toLocaleString('en-US')}`); + for (const r of data.security.topRules) { + lines.push(`${r.ruleTitle} [${r.severity}]: ${r.count.toLocaleString('en-US')} detections`); + } + if (data.uptime) { + lines.push(''); + lines.push('UPTIME'); + lines.push('-'.repeat(6)); + lines.push(`Overall: ${data.uptime.overallUptimePct.toLocaleString('en-US')}% across ${data.uptime.monitorCount.toLocaleString('en-US')} monitor(s)`); + for (const m of data.uptime.worstMonitors) { + lines.push(`${m.name}: ${m.uptimePct.toLocaleString('en-US')}%`); + } + } + lines.push(''); + lines.push(`View your dashboard: ${data.dashboardUrl}`); + lines.push(''); + lines.push('To unsubscribe from these reports, click:'); + lines.push(data.unsubscribeUrl); + lines.push(''); + lines.push('--'); + lines.push('Sent by LogTide'); + + return { html, text: lines.join('\n') }; +} + diff --git a/packages/backend/src/modules/admin/service.ts b/packages/backend/src/modules/admin/service.ts index d1a77899..356e0249 100644 --- a/packages/backend/src/modules/admin/service.ts +++ b/packages/backend/src/modules/admin/service.ts @@ -1234,6 +1234,9 @@ export class AdminService { 'ingestion.detection_enqueue_failed', 'ingestion.exception_enqueue_failed', 'ingestion.identifier_failed', + 'ingestion.timestamp_skew', + 'ingestion.span_timestamp_skew', + 'ingestion.metric_timestamp_skew', ]) .groupBy('type') .execute(); @@ -1249,6 +1252,9 @@ export class AdminService { detectionEnqueueFailed: byType['ingestion.detection_enqueue_failed'] ?? 0, exceptionEnqueueFailed: byType['ingestion.exception_enqueue_failed'] ?? 0, identifierFailed: byType['ingestion.identifier_failed'] ?? 0, + timestampSkew: byType['ingestion.timestamp_skew'] ?? 0, + spanTimestampSkew: byType['ingestion.span_timestamp_skew'] ?? 0, + metricTimestampSkew: byType['ingestion.metric_timestamp_skew'] ?? 0, }, enrichment: enrichmentService.getStatus(), }; diff --git a/packages/backend/src/modules/audit-log/actions.ts b/packages/backend/src/modules/audit-log/actions.ts index c6dca7fc..72e88839 100644 --- a/packages/backend/src/modules/audit-log/actions.ts +++ b/packages/backend/src/modules/audit-log/actions.ts @@ -65,6 +65,12 @@ export const AUDIT_ACTIONS = { 'channel.created': 'config_change', 'channel.updated': 'config_change', 'channel.deleted': 'config_change', + // email digest reports (#154) + 'digest.config_updated': 'config_change', + 'digest.config_deleted': 'config_change', + 'digest.recipient_added': 'config_change', + 'digest.recipient_removed': 'config_change', + 'digest.recipient_resubscribed': 'config_change', // webhooks 'webhook.delivery_replayed': 'config_change', // auth diff --git a/packages/backend/src/modules/auth/authentication-service.ts b/packages/backend/src/modules/auth/authentication-service.ts index be265513..53c550be 100644 --- a/packages/backend/src/modules/auth/authentication-service.ts +++ b/packages/backend/src/modules/auth/authentication-service.ts @@ -16,6 +16,8 @@ import type { UserProfile, SessionInfo } from '../users/service.js'; import { settingsService } from '../settings/service.js'; import { providerRegistry, + AuthError, + AuthErrorCode, type AuthProvider, type AuthenticationResult, type UserIdentity, @@ -51,14 +53,20 @@ export class AuthenticationService { // Get provider const provider = await providerRegistry.getProvider(providerSlug); if (!provider) { - throw new Error(`Authentication provider '${providerSlug}' not found or disabled`); + throw new AuthError( + `Authentication provider '${providerSlug}' not found or disabled`, + AuthErrorCode.PROVIDER_UNAVAILABLE + ); } // Authenticate with provider const result = await provider.authenticate(credentials); if (!result.success) { - throw new Error(result.error || 'Authentication failed'); + throw new AuthError( + result.error || 'Authentication failed', + result.errorCode ?? AuthErrorCode.PROVIDER_ERROR + ); } // Find or create user @@ -171,7 +179,7 @@ export class AuthenticationService { // Get provider const provider = await providerRegistry.getProviderById(stateData.providerId); if (!provider) { - throw new Error('Authentication provider not found or disabled'); + throw new AuthError('Authentication provider not found or disabled', AuthErrorCode.PROVIDER_UNAVAILABLE); } if (!provider.handleCallback) { @@ -210,7 +218,10 @@ export class AuthenticationService { await CacheManager.delete(`oidc:state:${state}`); if (!result.success) { - throw new Error(result.error || 'Authentication failed'); + throw new AuthError( + result.error || 'Authentication failed', + result.errorCode ?? AuthErrorCode.PROVIDER_ERROR + ); } // Find or create user @@ -257,7 +268,7 @@ export class AuthenticationService { if (existingIdentity) { // Check if user is disabled if (existingIdentity.disabled) { - throw new Error('This account has been disabled'); + throw new AuthError('This account has been disabled', AuthErrorCode.USER_DISABLED); } // Update user's last login @@ -291,7 +302,7 @@ export class AuthenticationService { if (existingUser) { // Check if user is disabled if (existingUser.disabled) { - throw new Error('This account has been disabled'); + throw new AuthError('This account has been disabled', AuthErrorCode.USER_DISABLED); } // Only auto-link to a pre-existing account when the provider asserts the diff --git a/packages/backend/src/modules/auth/providers/index.ts b/packages/backend/src/modules/auth/providers/index.ts index 5d6eebaa..4e843d7f 100644 --- a/packages/backend/src/modules/auth/providers/index.ts +++ b/packages/backend/src/modules/auth/providers/index.ts @@ -19,7 +19,7 @@ export type { UserIdentity, } from './types.js'; -export { type AuthProviderType, AuthErrorCode } from './types.js'; +export { type AuthProviderType, AuthErrorCode, AuthError } from './types.js'; // Providers export { LocalProvider } from './local-provider.js'; diff --git a/packages/backend/src/modules/auth/providers/local-provider.ts b/packages/backend/src/modules/auth/providers/local-provider.ts index 09d99289..dd362006 100644 --- a/packages/backend/src/modules/auth/providers/local-provider.ts +++ b/packages/backend/src/modules/auth/providers/local-provider.ts @@ -57,26 +57,28 @@ export class LocalProvider implements AuthProvider { return { success: false, error: 'Please log in using your organization SSO', - errorCode: AuthErrorCode.INVALID_CREDENTIALS, + errorCode: AuthErrorCode.SSO_REQUIRED, }; } - // Check if user is disabled - if (user.disabled) { + // Verify password before any other checks to prevent user enumeration: + // a wrong password must always return the same generic error regardless of + // whether the account is disabled or not. + const isValid = await bcrypt.compare(password, user.password_hash); + if (!isValid) { return { success: false, - error: 'This account has been disabled', - errorCode: AuthErrorCode.USER_DISABLED, + error: 'Invalid email or password', + errorCode: AuthErrorCode.INVALID_CREDENTIALS, }; } - // Verify password - const isValid = await bcrypt.compare(password, user.password_hash); - if (!isValid) { + // Check if user is disabled (only after the password is confirmed correct) + if (user.disabled) { return { success: false, - error: 'Invalid email or password', - errorCode: AuthErrorCode.INVALID_CREDENTIALS, + error: 'This account has been disabled', + errorCode: AuthErrorCode.USER_DISABLED, }; } @@ -84,6 +86,7 @@ export class LocalProvider implements AuthProvider { success: true, providerUserId: user.email, // For local, we use email as the provider user ID email: user.email, + emailVerified: true, // Password verification implies ownership of the account name: user.name, metadata: { userId: user.id, diff --git a/packages/backend/src/modules/auth/providers/types.ts b/packages/backend/src/modules/auth/providers/types.ts index 7466d751..6cb8c258 100644 --- a/packages/backend/src/modules/auth/providers/types.ts +++ b/packages/backend/src/modules/auth/providers/types.ts @@ -79,6 +79,7 @@ export interface AuthenticationResult { */ export enum AuthErrorCode { INVALID_CREDENTIALS = 'INVALID_CREDENTIALS', + SSO_REQUIRED = 'SSO_REQUIRED', USER_DISABLED = 'USER_DISABLED', PROVIDER_UNAVAILABLE = 'PROVIDER_UNAVAILABLE', PROVIDER_ERROR = 'PROVIDER_ERROR', @@ -89,6 +90,23 @@ export enum AuthErrorCode { ACCOUNT_LOCKED = 'ACCOUNT_LOCKED', } +/** + * Typed authentication error. + * + * Carries the provider's AuthErrorCode end to end so callers (HTTP routes, + * audit logging) can classify a failure by code instead of matching the + * user-facing message string, which would silently break on any reword. + */ +export class AuthError extends Error { + readonly code: AuthErrorCode; + + constructor(message: string, code: AuthErrorCode) { + super(message); + this.name = 'AuthError'; + this.code = code; + } +} + /** * Credentials for local (email/password) authentication */ diff --git a/packages/backend/src/modules/dashboard/service.ts b/packages/backend/src/modules/dashboard/service.ts index 938a28e2..9f64d67f 100644 --- a/packages/backend/src/modules/dashboard/service.ts +++ b/packages/backend/src/modules/dashboard/service.ts @@ -49,6 +49,20 @@ export interface TimelineEvent { detectionsBySeverity: { critical: number; high: number; medium: number; low: number }; } +/** + * Below this recent (last-hour) log volume, dashboard "today" stats are computed + * from raw logs instead of the hourly continuous aggregate. The aggregate only + * reflects data below its materialization watermark that its refresh policy has + * actually processed (start_offset = 3h); on instances where the policy is not + * keeping the aggregate warm, "today" collapses to roughly the last hour (the + * only part read from raw), which is what makes "Total Logs Today" and the error + * rate read far too low. Verified live on the hosted demo: 24h = ~1,500 logs, + * last hour = ~67, yet "Total Logs Today" showed ~60. A raw count over the + * bounded today/yesterday window is exact and cheap at low volume; higher-volume + * instances keep the fast aggregate path (their live ingestion keeps it warm). + */ +const RAW_STATS_MAX_RECENT_VOLUME = 5000; + class DashboardService { /** * Resolve project IDs from org or single project. @@ -163,6 +177,13 @@ class DashboardService { }); const recentVolume = recentTotalResult.count; + // Low-volume instances (fresh installs, backfilled/seeded data, or a stale + // continuous aggregate) are exactly where the aggregate undercounts "today", + // so count today/yesterday from raw for an exact, cheap result. + if (recentVolume <= RAW_STATS_MAX_RECENT_VOLUME) { + return this.getStatsFromRawLogs(projectIds, todayStart, yesterdayStart, lastHourStart, prevHourStart); + } + // 2. Query aggregate for historical data and reservoir for recent data in parallel const [todayAggregateStats, recentTotal, recentErrors, recentServices, yesterdayAggregateStats, prevHourCount] = await Promise.all([ // Today's historical stats from aggregate (today start to 1 hour ago) diff --git a/packages/backend/src/modules/digests/generator.ts b/packages/backend/src/modules/digests/generator.ts index 518a2781..67b35447 100644 --- a/packages/backend/src/modules/digests/generator.ts +++ b/packages/backend/src/modules/digests/generator.ts @@ -1,7 +1,9 @@ /** * Digest Generator Service * - * Generates and sends email digest reports summarizing log activity over a period. + * Generates and sends email digest reports summarizing organization activity + * over a period: log volume, top services by errors, new error groups, + * security detections and monitor uptime. */ import nodemailer from 'nodemailer'; @@ -9,12 +11,40 @@ import { db } from '../../database/connection.js'; import { reservoir } from '../../database/reservoir.js'; import { hub } from '@logtide/core'; import { config } from '../../config/index.js'; +import { generateDigestEmail } from '../../lib/email-templates.js'; import type { DigestJobPayload } from './scheduler.js'; -interface LogVolumeStats { - currentPeriodCount: number; - previousPeriodCount: number; - trend: string; +export interface DigestReportData { + organizationName: string; + frequency: 'daily' | 'weekly'; + periodLabel: string; + logVolume: { + current: number; + previous: number; + trend: string; + }; + topErrorServices: Array<{ + service: string; + errorCount: number; + previousCount: number; + delta: number; + }>; + newErrorGroups: Array<{ + exceptionType: string; + exceptionMessage: string; + occurrenceCount: number; + language: string; + }>; + security: { + totalDetections: number; + topRules: Array<{ ruleTitle: string; severity: string; count: number }>; + openIncidents: number; + }; + uptime: { + monitorCount: number; + overallUptimePct: number; + worstMonitors: Array<{ name: string; uptimePct: number }>; + } | null; } interface DigestRecipient { @@ -22,6 +52,14 @@ interface DigestRecipient { unsubscribe_token: string; } +interface Period { + from: Date; + to: Date; + previousFrom: Date; + previousTo: Date; +} + +const ERROR_LEVELS = ['error', 'critical'] as const; let emailTransporter: nodemailer.Transporter | null = null; @@ -53,14 +91,13 @@ function getEmailTransporter(): nodemailer.Transporter | null { } export class DigestGeneratorService { - + async generateAndSendDigest(payload: DigestJobPayload): Promise { const { organizationId, digestConfigId, frequency } = payload; hub.captureLog('info', `[DigestGenerator] Generating ${frequency} digest for org ${organizationId}`); try { - const organization = await db .selectFrom('organizations') .select(['name']) @@ -71,7 +108,6 @@ export class DigestGeneratorService { throw new Error(`Organization ${organizationId} not found`); } - const recipients = await this.fetchRecipients(organizationId, digestConfigId); if (recipients.length === 0) { @@ -79,16 +115,9 @@ export class DigestGeneratorService { return; } - - const stats = await this.calculateLogVolume(organizationId, frequency); + const report = await this.buildReportData(organizationId, organization.name, frequency); - - await this.sendDigestEmails( - recipients, - organization.name, - frequency, - stats - ); + await this.sendDigestEmails(recipients, report); hub.captureLog('info', `[DigestGenerator] Digest sent to ${recipients.length} recipient(s) for org ${organizationId}`); } catch (error: any) { @@ -97,92 +126,284 @@ export class DigestGeneratorService { } } - - private async fetchRecipients( + /** + * Compute all report sections for the period. Sections run sequentially: + * this is a background cron path where simplicity beats latency. + */ + async buildReportData( organizationId: string, - digestConfigId: string - ): Promise { - const recipients = await db - .selectFrom('digest_recipients') - .select(['email', 'unsubscribe_token']) + organizationName: string, + frequency: 'daily' | 'weekly' + ): Promise { + const period = this.buildPeriod(frequency); + + const projects = await db + .selectFrom('projects') + .select(['id']) .where('organization_id', '=', organizationId) - .where('digest_config_id', '=', digestConfigId) - .where('subscribed', '=', true) .execute(); + const projectIds = projects.map((p) => p.id); - return recipients; + const logVolume = await this.calculateLogVolume(projectIds, period); + const topErrorServices = await this.calculateTopErrorServices(projectIds, period); + const newErrorGroups = await this.calculateNewErrorGroups(organizationId, period); + const security = await this.calculateSecuritySummary(organizationId, period); + const uptime = await this.calculateUptimeSummary(organizationId, period); + + return { + organizationName, + frequency, + periodLabel: frequency === 'daily' ? 'last 24 hours' : 'last 7 days', + logVolume, + topErrorServices, + newErrorGroups, + security, + uptime, + }; + } + + private buildPeriod(frequency: 'daily' | 'weekly'): Period { + // Sliding window relative to execution time + const hoursInPeriod = frequency === 'daily' ? 24 : 168; + const now = new Date(); + const from = new Date(now.getTime() - hoursInPeriod * 60 * 60 * 1000); + const previousFrom = new Date(now.getTime() - hoursInPeriod * 2 * 60 * 60 * 1000); + + return { from, to: now, previousFrom, previousTo: from }; } private async calculateLogVolume( + projectIds: string[], + period: Period + ): Promise { + if (projectIds.length === 0) { + return { current: 0, previous: 0, trend: this.calculateTrend(0, 0) }; + } + + const currentResult = await reservoir.count({ + projectId: projectIds, + from: period.from, + to: period.to, + toExclusive: true, + }); + + const previousResult = await reservoir.count({ + projectId: projectIds, + from: period.previousFrom, + to: period.previousTo, + toExclusive: true, + }); + + return { + current: currentResult.count, + previous: previousResult.count, + trend: this.calculateTrend(currentResult.count, previousResult.count), + }; + } + + /** + * Top 5 services by error+critical log count, with the delta against the + * previous period. Engine-agnostic through reservoir.topValues. + */ + private async calculateTopErrorServices( + projectIds: string[], + period: Period + ): Promise { + if (projectIds.length === 0) { + return []; + } + + const current = await reservoir.topValues({ + field: 'service', + projectId: projectIds, + level: [...ERROR_LEVELS], + from: period.from, + to: period.to, + limit: 5, + }); + + if (current.values.length === 0) { + return []; + } + + // Wide limit so services that dropped out of the top 5 still resolve + const previous = await reservoir.topValues({ + field: 'service', + projectId: projectIds, + level: [...ERROR_LEVELS], + from: period.previousFrom, + to: period.previousTo, + limit: 100, + }); + + const previousByService = new Map(previous.values.map((v) => [v.value, v.count])); + + return current.values.map((v) => { + const previousCount = previousByService.get(v.value) ?? 0; + return { + service: v.value, + errorCount: v.count, + previousCount, + delta: v.count - previousCount, + }; + }); + } + + /** + * Error groups whose first occurrence falls inside the period. + */ + private async calculateNewErrorGroups( organizationId: string, - frequency: 'daily' | 'weekly' - ): Promise { - // Uses a 24-hour sliding window relative to execution time - const hoursInPeriod = frequency === 'daily' ? 24 : 168; // 7 days = 168 hours + period: Period + ): Promise { + const groups = await db + .selectFrom('error_groups') + .select(['exception_type', 'exception_message', 'occurrence_count', 'language']) + .where('organization_id', '=', organizationId) + .where('first_seen', '>=', period.from) + .where('first_seen', '<', period.to) + .orderBy('occurrence_count', 'desc') + .limit(10) + .execute(); - const now = new Date(); - const currentPeriodStart = new Date(now.getTime() - hoursInPeriod * 60 * 60 * 1000); - const previousPeriodStart = new Date(now.getTime() - hoursInPeriod * 2 * 60 * 60 * 1000); - const previousPeriodEnd = currentPeriodStart; + return groups.map((g) => ({ + exceptionType: g.exception_type, + exceptionMessage: g.exception_message ?? '', + occurrenceCount: g.occurrence_count, + language: g.language, + })); + } - let currentPeriodCount = 0; - let previousPeriodCount = 0; + /** + * Security summary: windowed detection totals and top triggered Sigma rules + * (raw detection_events, so the most recent hours are never stale like the + * continuous aggregates), plus a point-in-time open incident count. + */ + private async calculateSecuritySummary( + organizationId: string, + period: Period + ): Promise { + const totalRow = await db + .selectFrom('detection_events') + .select((eb) => eb.fn.countAll().as('count')) + .where('organization_id', '=', organizationId) + .where('time', '>=', period.from) + .where('time', '<', period.to) + .executeTakeFirst(); - try { - - // organization-level metrics we must count logs for all projects that - // belong to the organization and pass those project ids to the reservoir - const projects = await db - .selectFrom('projects') - .select(['id']) - .where('organization_id', '=', organizationId) - .execute(); - - const projectIds = projects.map((p) => p.id); - - // If the organization has no projects, return zero counts explicitly - if (projectIds.length === 0) { - const trend = this.calculateTrend(0, 0); - return { - currentPeriodCount: 0, - previousPeriodCount: 0, - trend, - }; - } else { - const currentPeriodResult = await reservoir.count({ - projectId: projectIds, - from: currentPeriodStart, - to: now, - toExclusive: true, - }); - currentPeriodCount = currentPeriodResult.count; + const topRules = await db + .selectFrom('detection_events') + .select((eb) => ['rule_title', 'severity', eb.fn.countAll().as('count')] as const) + .where('organization_id', '=', organizationId) + .where('time', '>=', period.from) + .where('time', '<', period.to) + .groupBy(['rule_title', 'severity']) + .orderBy('count', 'desc') + .limit(5) + .execute(); + + const openRow = await db + .selectFrom('incidents') + .select((eb) => eb.fn.countAll().as('count')) + .where('organization_id', '=', organizationId) + .where('status', 'in', ['open', 'investigating']) + .executeTakeFirst(); + + return { + totalDetections: Number(totalRow?.count ?? 0), + topRules: topRules.map((r) => ({ + ruleTitle: r.rule_title, + severity: r.severity, + count: Number(r.count), + })), + openIncidents: Number(openRow?.count ?? 0), + }; + } + + /** + * Uptime summary from monitor_uptime_daily. Returns null when the org has + * no enabled monitors so the email can skip the section entirely. + * Daily buckets only partially overlap a sliding 24h window; that + * approximation is acceptable for a digest. + */ + private async calculateUptimeSummary( + organizationId: string, + period: Period + ): Promise { + const monitors = await db + .selectFrom('monitors') + .select(['id', 'name']) + .where('organization_id', '=', organizationId) + .where('enabled', '=', true) + .execute(); + + if (monitors.length === 0) { + return null; + } - const previousPeriodResult = await reservoir.count({ - projectId: projectIds, - from: previousPeriodStart, - to: previousPeriodEnd, - toExclusive: true, + const bucketFrom = new Date(period.from); + bucketFrom.setUTCHours(0, 0, 0, 0); + + const rows = await db + .selectFrom('monitor_uptime_daily') + .select((eb) => [ + 'monitor_id', + eb.fn.sum('successful_checks').as('successful'), + eb.fn.sum('total_checks').as('total'), + ]) + .where('organization_id', '=', organizationId) + .where('bucket', '>=', bucketFrom) + .groupBy('monitor_id') + .execute(); + + const statsByMonitor = new Map(rows.map((r) => [r.monitor_id, r])); + + let overallSuccessful = 0; + let overallTotal = 0; + const perMonitor: Array<{ name: string; uptimePct: number }> = []; + + for (const monitor of monitors) { + const stats = statsByMonitor.get(monitor.id); + const successful = Number(stats?.successful ?? 0); + const total = Number(stats?.total ?? 0); + overallSuccessful += successful; + overallTotal += total; + + if (total > 0) { + perMonitor.push({ + name: monitor.name, + uptimePct: Math.round((successful / total) * 10000) / 100, }); - previousPeriodCount = previousPeriodResult.count; } - } catch (error: unknown) { - const message = error instanceof Error ? error.message : String(error); - hub.captureLog('error', `[DigestGenerator] Log volume count failed for org ${organizationId}: ${message}`, { error }); - throw error; } - const trend = this.calculateTrend(currentPeriodCount, previousPeriodCount); + const overallUptimePct = + overallTotal > 0 ? Math.round((overallSuccessful / overallTotal) * 10000) / 100 : 100; + + perMonitor.sort((a, b) => a.uptimePct - b.uptimePct); return { - currentPeriodCount, - previousPeriodCount, - trend, + monitorCount: monitors.length, + overallUptimePct, + worstMonitors: perMonitor.slice(0, 5), }; } - - //current and previous counts - + private async fetchRecipients( + organizationId: string, + digestConfigId: string + ): Promise { + const recipients = await db + .selectFrom('digest_recipients') + .select(['email', 'unsubscribe_token']) + .where('organization_id', '=', organizationId) + .where('digest_config_id', '=', digestConfigId) + .where('subscribed', '=', true) + .execute(); + + return recipients; + } + private calculateTrend(current: number, previous: number): string { if (previous === 0 && current === 0) { return 'no change'; @@ -204,12 +425,9 @@ export class DigestGeneratorService { } } - //mail private async sendDigestEmails( recipients: DigestRecipient[], - organizationName: string, - frequency: 'daily' | 'weekly', - stats: LogVolumeStats + report: DigestReportData ): Promise { const transporter = getEmailTransporter(); @@ -217,22 +435,22 @@ export class DigestGeneratorService { throw new Error('Email transporter not configured'); } - const subject = `[LogTide Digest] ${frequency === 'daily' ? 'Daily' : 'Weekly'} Report - ${organizationName}`; + const subject = `[LogTide Digest] ${report.frequency === 'daily' ? 'Daily' : 'Weekly'} Report - ${report.organizationName}`; + const frontendUrl = this.getFrontendUrl(); - const emailPromises = recipients.map(async (recipient) => { - const text = this.generatePlaintextEmail( - organizationName, - frequency, - stats, - recipient.unsubscribe_token - ); + const { html, text } = generateDigestEmail({ + ...report, + unsubscribeUrl: `${frontendUrl}/unsubscribe?token=${recipient.unsubscribe_token}`, + dashboardUrl: `${frontendUrl}/dashboard`, + }); await transporter.sendMail({ from: `"LogTide" <${config.SMTP_FROM || config.SMTP_USER}>`, to: recipient.email, subject, text, + html, }); hub.captureLog('info', `[DigestGenerator] Email sent to ${recipient.email}`); @@ -249,53 +467,6 @@ export class DigestGeneratorService { } } - //email template - private generatePlaintextEmail( - organizationName: string, - frequency: 'daily' | 'weekly', - stats: LogVolumeStats, - unsubscribeToken: string - ): string { - const period = frequency === 'daily' ? 'last 24 hours' : 'last 7 days'; - const frontendUrl = this.getFrontendUrl(); - const unsubscribeUrl = `${frontendUrl}/unsubscribe?token=${unsubscribeToken}`; - - let content = `LogTide ${frequency === 'daily' ? 'Daily' : 'Weekly'} Digest\n`; - content += `Organization: ${organizationName}\n`; - content += `Period: ${period}\n`; - content += `\n`; - content += `━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n`; - content += `\n`; - - if (stats.currentPeriodCount === 0 && stats.previousPeriodCount === 0) { - content += ` Log Volume\n`; - content += `\n`; - content += `No activity during this period.\n`; - content += `Your systems have been quiet.\n`; - } else { - content += ` Log Volume\n`; - content += `\n`; - // Fixed locale: digest content must not depend on the server's locale - content += `Total logs: ${stats.currentPeriodCount.toLocaleString('en-US')}\n`; - content += `Trend: ${stats.trend}\n`; - content += `Previous period: ${stats.previousPeriodCount.toLocaleString('en-US')}\n`; - } - - content += `\n`; - content += `━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n`; - content += `\n`; - content += `View your dashboard: ${frontendUrl}\n`; - content += `\n`; - content += `To unsubscribe from these reports, click:\n`; - content += `${unsubscribeUrl}\n`; - content += `\n`; - content += `—\n`; - content += `LogTide - observability for your infrastructure\n`; - - return content; - } - - private getFrontendUrl(): string { return config.FRONTEND_URL || 'http://localhost:3000'; } diff --git a/packages/backend/src/modules/digests/index.ts b/packages/backend/src/modules/digests/index.ts new file mode 100644 index 00000000..ec616a70 --- /dev/null +++ b/packages/backend/src/modules/digests/index.ts @@ -0,0 +1,5 @@ +export * from './service.js'; +export * from './routes.js'; +export * from './public-routes.js'; +export * from './scheduler.js'; +export * from './generator.js'; diff --git a/packages/backend/src/modules/digests/public-routes.ts b/packages/backend/src/modules/digests/public-routes.ts new file mode 100644 index 00000000..7a24c244 --- /dev/null +++ b/packages/backend/src/modules/digests/public-routes.ts @@ -0,0 +1,45 @@ +/** + * Public Digest Routes (#154) + * + * One-click unsubscribe consumed by the link in digest email footers. No + * authentication: the 32-byte random unsubscribe token is the credential + * (unique index on digest_recipients.unsubscribe_token, not enumerable). + */ + +import type { FastifyInstance } from 'fastify'; +import { z } from 'zod'; +import { digestsService } from './service.js'; + +const unsubscribeSchema = z.object({ + token: z.string().min(1).max(200), +}); + +/** + * Mask an email for the unsubscribe confirmation page: g***@example.com + */ +function maskEmail(email: string): string { + const [local, domain] = email.split('@'); + if (!domain) return '***'; + const visible = local.slice(0, 1); + return `${visible}***@${domain}`; +} + +export async function digestsPublicRoutes(fastify: FastifyInstance) { + /** + * POST /api/v1/digests/unsubscribe + * Marks the recipient behind the token as unsubscribed. Idempotent. + */ + fastify.post('/unsubscribe', async (request, reply) => { + const parsed = unsubscribeSchema.safeParse(request.body); + if (!parsed.success) { + return reply.status(400).send({ error: 'Invalid unsubscribe request' }); + } + + const result = await digestsService.unsubscribeByToken(parsed.data.token); + if (!result) { + return reply.status(404).send({ error: 'Invalid or expired unsubscribe link' }); + } + + return reply.send({ success: true, email: maskEmail(result.email) }); + }); +} diff --git a/packages/backend/src/modules/digests/routes.ts b/packages/backend/src/modules/digests/routes.ts new file mode 100644 index 00000000..d28a8794 --- /dev/null +++ b/packages/backend/src/modules/digests/routes.ts @@ -0,0 +1,320 @@ +/** + * Digest Configuration API Routes (#154) + * + * Session-authenticated CRUD for the per-organization digest config and its + * recipient list. Reads need membership; writes need owner/admin. + */ + +import type { FastifyInstance } from 'fastify'; +import { z } from 'zod'; +import { digestsService } from './service.js'; +import { authenticate } from '../auth/middleware.js'; +import { OrganizationsService } from '../organizations/service.js'; +import { context } from '@logtide/shared/context'; +import { assertWithinLimit, withLimitLock } from '../../capabilities/index.js'; +import { CapabilityError } from '../../capabilities/errors.js'; +import { auditLogService } from '../audit-log/service.js'; + +const organizationsService = new OrganizationsService(); + +// ============================================================================ +// VALIDATION SCHEMAS +// ============================================================================ + +const orgQuerySchema = z.object({ + organizationId: z.string().uuid('organizationId must be a valid uuid'), +}); + +const configBodySchema = z + .object({ + frequency: z.enum(['daily', 'weekly']), + deliveryHour: z.number().int().min(0).max(23), + deliveryDayOfWeek: z.number().int().min(0).max(6).nullish(), + enabled: z.boolean(), + }) + .refine((body) => body.frequency !== 'weekly' || body.deliveryDayOfWeek != null, { + message: 'deliveryDayOfWeek is required for weekly digests', + path: ['deliveryDayOfWeek'], + }); + +const addRecipientSchema = z.object({ + email: z.string().email('Must be a valid email address'), +}); + +const recipientParamsSchema = z.object({ + id: z.string().uuid(), +}); + +// ============================================================================ +// HELPERS +// ============================================================================ + +async function checkOrganizationMembership( + userId: string, + organizationId: string +): Promise { + const organizations = await organizationsService.getUserOrganizations(userId); + return organizations.some((org) => org.id === organizationId); +} + +async function checkAdminRole(userId: string, organizationId: string): Promise { + return organizationsService.isOwnerOrAdmin(organizationId, userId); +} + +// ============================================================================ +// ROUTES +// ============================================================================ + +export async function digestsRoutes(fastify: FastifyInstance) { + // All routes require authentication + fastify.addHook('onRequest', authenticate); + + /** + * GET /api/v1/digests/config + * Current digest config + recipients for an organization (members) + */ + fastify.get('/config', async (request: any, reply) => { + try { + const { organizationId } = orgQuerySchema.parse(request.query); + + const isMember = await checkOrganizationMembership(request.user.id, organizationId); + if (!isMember) { + return reply.status(403).send({ error: 'Not a member of this organization' }); + } + + const config = await digestsService.getConfig(organizationId); + const recipients = await digestsService.listRecipients(organizationId); + + return reply.send({ config: config ?? null, recipients }); + } catch (error) { + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Validation error', details: error.errors }); + } + console.error(error, 'Failed to get digest config'); + return reply.status(500).send({ error: 'Failed to get digest config' }); + } + }); + + /** + * PUT /api/v1/digests/config + * Create or update the organization digest config (admin only) + */ + fastify.put('/config', async (request: any, reply) => { + try { + const { organizationId } = orgQuerySchema.parse(request.query); + const body = configBodySchema.parse(request.body); + + const isMember = await checkOrganizationMembership(request.user.id, organizationId); + if (!isMember) { + return reply.status(403).send({ error: 'Not a member of this organization' }); + } + + const isAdmin = await checkAdminRole(request.user.id, organizationId); + if (!isAdmin) { + return reply.status(403).send({ error: 'Only admins can configure digests' }); + } + + const config = await digestsService.upsertConfig(organizationId, { + frequency: body.frequency, + deliveryHour: body.deliveryHour, + deliveryDayOfWeek: body.deliveryDayOfWeek, + enabled: body.enabled, + }); + + await auditLogService.record({ + action: 'digest.config_updated', + target: { type: 'digest_config', id: config.id }, + organizationId, + metadata: { + frequency: config.frequency, + deliveryHour: config.delivery_hour, + enabled: config.enabled, + }, + }); + + return reply.send({ config }); + } catch (error) { + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Validation error', details: error.errors }); + } + console.error(error, 'Failed to save digest config'); + return reply.status(500).send({ error: 'Failed to save digest config' }); + } + }); + + /** + * DELETE /api/v1/digests/config + * Remove the digest config (and its recipients via cascade) (admin only) + */ + fastify.delete('/config', async (request: any, reply) => { + try { + const { organizationId } = orgQuerySchema.parse(request.query); + + const isMember = await checkOrganizationMembership(request.user.id, organizationId); + if (!isMember) { + return reply.status(403).send({ error: 'Not a member of this organization' }); + } + + const isAdmin = await checkAdminRole(request.user.id, organizationId); + if (!isAdmin) { + return reply.status(403).send({ error: 'Only admins can configure digests' }); + } + + const deleted = await digestsService.deleteConfig(organizationId); + if (!deleted) { + return reply.status(404).send({ error: 'Digest config not found' }); + } + + await auditLogService.record({ + action: 'digest.config_deleted', + target: { type: 'digest_config', id: organizationId }, + organizationId, + }); + + return reply.status(204).send(); + } catch (error) { + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Validation error', details: error.errors }); + } + console.error(error, 'Failed to delete digest config'); + return reply.status(500).send({ error: 'Failed to delete digest config' }); + } + }); + + /** + * POST /api/v1/digests/recipients + * Add a recipient (admin only, capability-limited) + */ + fastify.post('/recipients', async (request: any, reply) => { + try { + const { organizationId } = orgQuerySchema.parse(request.query); + const body = addRecipientSchema.parse(request.body); + + const isMember = await checkOrganizationMembership(request.user.id, organizationId); + if (!isMember) { + return reply.status(403).send({ error: 'Not a member of this organization' }); + } + + const isAdmin = await checkAdminRole(request.user.id, organizationId); + if (!isAdmin) { + return reply.status(403).send({ error: 'Only admins can manage digest recipients' }); + } + + const recipient = await withLimitLock(organizationId, 'digests.max_recipients', async () => { + await context.runAsSystem('digests:recipient-limit-check', async () => { + await context.with({ organizationId }, async () => { + const count = await digestsService.countRecipients(organizationId); + await assertWithinLimit('digests.max_recipients', count); + }); + }); + + return digestsService.addRecipient(organizationId, body.email); + }); + + await auditLogService.record({ + action: 'digest.recipient_added', + target: { type: 'digest_recipient', id: recipient.id }, + organizationId, + metadata: { email: recipient.email }, + }); + + return reply.status(201).send({ recipient }); + } catch (error) { + if (error instanceof CapabilityError) { + throw error; + } + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Validation error', details: error.errors }); + } + if (error instanceof Error && error.message.includes('not configured')) { + return reply.status(409).send({ error: 'Configure the digest before adding recipients' }); + } + if (error instanceof Error && (error.message.includes('unique') || error.message.includes('duplicate'))) { + return reply.status(409).send({ error: 'This email is already a recipient' }); + } + console.error(error, 'Failed to add digest recipient'); + return reply.status(500).send({ error: 'Failed to add recipient' }); + } + }); + + /** + * DELETE /api/v1/digests/recipients/:id + * Remove a recipient (admin only) + */ + fastify.delete('/recipients/:id', async (request: any, reply) => { + try { + const { organizationId } = orgQuerySchema.parse(request.query); + const { id } = recipientParamsSchema.parse(request.params); + + const isMember = await checkOrganizationMembership(request.user.id, organizationId); + if (!isMember) { + return reply.status(403).send({ error: 'Not a member of this organization' }); + } + + const isAdmin = await checkAdminRole(request.user.id, organizationId); + if (!isAdmin) { + return reply.status(403).send({ error: 'Only admins can manage digest recipients' }); + } + + const removed = await digestsService.removeRecipient(organizationId, id); + if (!removed) { + return reply.status(404).send({ error: 'Recipient not found' }); + } + + await auditLogService.record({ + action: 'digest.recipient_removed', + target: { type: 'digest_recipient', id }, + organizationId, + }); + + return reply.status(204).send(); + } catch (error) { + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Validation error', details: error.errors }); + } + console.error(error, 'Failed to remove digest recipient'); + return reply.status(500).send({ error: 'Failed to remove recipient' }); + } + }); + + /** + * POST /api/v1/digests/recipients/:id/resubscribe + * Re-subscribe a recipient who opted out (admin only) + */ + fastify.post('/recipients/:id/resubscribe', async (request: any, reply) => { + try { + const { organizationId } = orgQuerySchema.parse(request.query); + const { id } = recipientParamsSchema.parse(request.params); + + const isMember = await checkOrganizationMembership(request.user.id, organizationId); + if (!isMember) { + return reply.status(403).send({ error: 'Not a member of this organization' }); + } + + const isAdmin = await checkAdminRole(request.user.id, organizationId); + if (!isAdmin) { + return reply.status(403).send({ error: 'Only admins can manage digest recipients' }); + } + + const recipient = await digestsService.resubscribeRecipient(organizationId, id); + if (!recipient) { + return reply.status(404).send({ error: 'Recipient not found' }); + } + + await auditLogService.record({ + action: 'digest.recipient_resubscribed', + target: { type: 'digest_recipient', id }, + organizationId, + metadata: { email: recipient.email }, + }); + + return reply.send({ recipient }); + } catch (error) { + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Validation error', details: error.errors }); + } + console.error(error, 'Failed to resubscribe digest recipient'); + return reply.status(500).send({ error: 'Failed to resubscribe recipient' }); + } + }); +} diff --git a/packages/backend/src/modules/digests/scheduler.ts b/packages/backend/src/modules/digests/scheduler.ts index 2dc163ac..540e3ba7 100644 --- a/packages/backend/src/modules/digests/scheduler.ts +++ b/packages/backend/src/modules/digests/scheduler.ts @@ -1,17 +1,21 @@ /** * Digest Scheduler * - * Reads all active digest configs from the database and registers them as - * repeating cron jobs via the queue abstraction. Called once at worker startup. + * Registers a single static hourly cron job ("digest-dispatch") at worker + * startup. The dispatch job (src/queue/jobs/digest-dispatch.ts) scans the + * active digest configs each hour and enqueues digest-generation jobs for the + * ones due at that UTC hour. * - * Both BullMQ (Redis) and graphile-worker (PostgreSQL) backends are supported - * through the ICronRegistry interface — this service never knows which is active. + * Earlier versions registered one cron job per organization at boot, but + * graphile-worker cron items cannot change while the runner is up, so config + * CRUD would have required a worker restart. A static dispatch cron plus a + * due-check keeps schedule changes live on both queue backends (BullMQ and + * graphile-worker) through the ICronRegistry interface - this service never + * knows which is active. */ -import { db } from '../../database/connection.js'; import { getCronRegistry } from '../../queue/queue-factory.js'; import { hub } from '@logtide/core'; -import type { CronJobDefinition } from '../../queue/abstractions/types.js'; export interface DigestJobPayload { organizationId: string; @@ -19,58 +23,24 @@ export interface DigestJobPayload { frequency: 'daily' | 'weekly'; } -export class DigestScheduler { - /** - * Register all active digest configs as repeating cron jobs. - */ - async registerAllDigests(): Promise { - const configs = await db - .selectFrom('digest_configs') - .select(['id', 'organization_id', 'frequency', 'delivery_hour', 'delivery_day_of_week']) - .where('enabled', '=', true) - .execute(); - - if (configs.length === 0) { - hub.captureLog('info', '[DigestScheduler] No active digest configs found'); - return; - } - - const items: CronJobDefinition[] = configs.map((config) => ({ - task: 'digest-generation', - cronExpression: this.buildCronExpression( - config.frequency as 'daily' | 'weekly', - config.delivery_hour, - config.delivery_day_of_week - ), - payload: { - organizationId: config.organization_id, - digestConfigId: config.id, - frequency: config.frequency, - } satisfies DigestJobPayload, - // Stable identifier per org — prevents duplicate schedules on restart - identifier: `digest:${config.organization_id}`, - })); - - await getCronRegistry('digest-generation').registerCronJobs(items); - hub.captureLog('info', `[DigestScheduler] Registered ${items.length} digest schedule(s)`); - } +export const DIGEST_DISPATCH_CRON = '0 * * * *'; // every hour on the hour +export class DigestScheduler { /** - * Build a standard 5-field cron expression from a digest config. - * - * Daily: "0 8 * * *" — every day at delivery_hour - * Weekly: "0 8 * * 1" — every week on delivery_day_of_week + * Register the hourly digest-dispatch cron job. Called once at worker boot. */ - private buildCronExpression( - frequency: 'daily' | 'weekly', - deliveryHour: number, - deliveryDayOfWeek: number | null - ): string { - if (frequency === 'daily') { - return `0 ${deliveryHour} * * *`; - } - // Weekly — delivery_day_of_week is guaranteed non-null by the DB constraint - return `0 ${deliveryHour} * * ${deliveryDayOfWeek}`; + async registerDispatchCron(): Promise { + await getCronRegistry('digest-dispatch').registerCronJobs([ + { + task: 'digest-dispatch', + cronExpression: DIGEST_DISPATCH_CRON, + payload: {}, + // Stable identifier - prevents duplicate schedules on restart + identifier: 'digest-dispatch', + }, + ]); + + hub.captureLog('info', '[DigestScheduler] Registered hourly digest dispatch cron'); } } diff --git a/packages/backend/src/modules/digests/service.ts b/packages/backend/src/modules/digests/service.ts new file mode 100644 index 00000000..30f99ca3 --- /dev/null +++ b/packages/backend/src/modules/digests/service.ts @@ -0,0 +1,164 @@ +/** + * Digests Service + * + * CRUD for digest_configs (one per organization) and digest_recipients, + * plus token-based unsubscribe used by the public endpoint. + */ + +import crypto from 'crypto'; +import { db } from '../../database/connection.js'; +import type { DigestFrequency } from '../../database/types.js'; + +export interface DigestConfigInput { + frequency: DigestFrequency; + deliveryHour: number; + deliveryDayOfWeek?: number | null; + enabled: boolean; +} + +export class DigestsService { + async getConfig(organizationId: string) { + return db + .selectFrom('digest_configs') + .select([ + 'id', + 'frequency', + 'delivery_hour', + 'delivery_day_of_week', + 'enabled', + 'last_sent_at', + 'created_at', + 'updated_at', + ]) + .where('organization_id', '=', organizationId) + .executeTakeFirst(); + } + + async upsertConfig(organizationId: string, input: DigestConfigInput) { + const values = { + organization_id: organizationId, + frequency: input.frequency, + delivery_hour: input.deliveryHour, + delivery_day_of_week: input.frequency === 'weekly' ? input.deliveryDayOfWeek ?? null : null, + enabled: input.enabled, + }; + + return db + .insertInto('digest_configs') + .values(values) + .onConflict((oc) => + oc.column('organization_id').doUpdateSet({ + frequency: values.frequency, + delivery_hour: values.delivery_hour, + delivery_day_of_week: values.delivery_day_of_week, + enabled: values.enabled, + updated_at: new Date(), + }) + ) + .returning([ + 'id', + 'frequency', + 'delivery_hour', + 'delivery_day_of_week', + 'enabled', + 'last_sent_at', + 'created_at', + 'updated_at', + ]) + .executeTakeFirstOrThrow(); + } + + async deleteConfig(organizationId: string): Promise { + const result = await db + .deleteFrom('digest_configs') + .where('organization_id', '=', organizationId) + .executeTakeFirst(); + return result.numDeletedRows > 0n; + } + + async listRecipients(organizationId: string) { + return db + .selectFrom('digest_recipients') + .select(['id', 'email', 'user_id', 'subscribed', 'created_at']) + .where('organization_id', '=', organizationId) + .orderBy('created_at', 'asc') + .execute(); + } + + async countRecipients(organizationId: string): Promise { + const row = await db + .selectFrom('digest_recipients') + .select((eb) => eb.fn.countAll().as('count')) + .where('organization_id', '=', organizationId) + .executeTakeFirst(); + return Number(row?.count ?? 0); + } + + /** + * Add a recipient to the organization's digest. The digest config must + * exist first; recipients hang off it. + */ + async addRecipient(organizationId: string, email: string, userId?: string | null) { + const config = await this.getConfig(organizationId); + if (!config) { + throw new Error('Digest is not configured for this organization'); + } + + return db + .insertInto('digest_recipients') + .values({ + organization_id: organizationId, + digest_config_id: config.id, + email, + user_id: userId ?? null, + unsubscribe_token: crypto.randomBytes(32).toString('base64url'), + }) + .returning(['id', 'email', 'user_id', 'subscribed', 'created_at']) + .executeTakeFirstOrThrow(); + } + + async removeRecipient(organizationId: string, recipientId: string): Promise { + const result = await db + .deleteFrom('digest_recipients') + .where('organization_id', '=', organizationId) + .where('id', '=', recipientId) + .executeTakeFirst(); + return result.numDeletedRows > 0n; + } + + /** + * Re-subscribe a recipient who unsubscribed. Rotates the unsubscribe token + * so links in previously delivered emails cannot flip the state back. + */ + async resubscribeRecipient(organizationId: string, recipientId: string) { + return db + .updateTable('digest_recipients') + .set({ + subscribed: true, + unsubscribe_token: crypto.randomBytes(32).toString('base64url'), + updated_at: new Date(), + }) + .where('organization_id', '=', organizationId) + .where('id', '=', recipientId) + .returning(['id', 'email', 'user_id', 'subscribed', 'created_at']) + .executeTakeFirst(); + } + + /** + * One-click unsubscribe from an email link. The opaque token IS the + * credential; no authentication involved. + */ + async unsubscribeByToken(token: string): Promise<{ email: string } | null> { + // tenant-scope-ok: public unsubscribe path, the unique random token is the credential + const row = await db + .updateTable('digest_recipients') + .set({ subscribed: false, updated_at: new Date() }) + .where('unsubscribe_token', '=', token) + .returning(['email']) + .executeTakeFirst(); + + return row ?? null; + } +} + +export const digestsService = new DigestsService(); diff --git a/packages/backend/src/modules/exceptions/detection.ts b/packages/backend/src/modules/exceptions/detection.ts index da2f3405..1f7b70c9 100644 --- a/packages/backend/src/modules/exceptions/detection.ts +++ b/packages/backend/src/modules/exceptions/detection.ts @@ -13,6 +13,11 @@ import type { ExceptionLanguage, StructuredException } from '@logtide/shared'; // Library patterns for detecting vendor/library code const LIBRARY_PATTERNS = [ /node_modules/, + // Node.js runtime frames (node:internal/..., node:events, ...). These are not + // application code, and because async stack traces vary in which internal + // frames they include, treating them as app code makes the same logical error + // fingerprint differently and split into several error groups. + /^node:/, /vendor\//, /site-packages/, /\.cargo/, diff --git a/packages/backend/src/modules/exceptions/fingerprint-service.ts b/packages/backend/src/modules/exceptions/fingerprint-service.ts index aafd6603..e60cf493 100644 --- a/packages/backend/src/modules/exceptions/fingerprint-service.ts +++ b/packages/backend/src/modules/exceptions/fingerprint-service.ts @@ -43,6 +43,20 @@ export class FingerprintService { return crypto.createHash('sha256').update(input).digest('hex'); } + /** + * The throw site: raw "file:function" of the first application frame, or null + * when there is none. Used as the coarse-grouping key's frame component (see + * migration 055's logtide_merge_key). Kept raw (not path-normalized) so the app + * and the SQL function agree without duplicating normalization. + */ + static topAppFrame(parsedException: ParsedException): string | null { + const frame = parsedException.frames.find((f) => f.isAppCode); + if (!frame) return null; + const file = frame.originalFile || frame.filePath; + const func = frame.originalFunction || frame.functionName || ''; + return `${file}:${func}`; + } + /** * Default normalization when parser is not available */ diff --git a/packages/backend/src/modules/exceptions/routes.ts b/packages/backend/src/modules/exceptions/routes.ts index a11a2fed..e7a78ff8 100644 --- a/packages/backend/src/modules/exceptions/routes.ts +++ b/packages/backend/src/modules/exceptions/routes.ts @@ -567,6 +567,85 @@ export async function exceptionsRoutes(fastify: FastifyInstance) { } ); + /** + * GET /api/v1/error-groups/:id/duplicates + * Other groups with the same exception type and message (merge candidates). + */ + fastify.get( + '/api/v1/error-groups/:id/duplicates', + async (request: any, reply) => { + try { + const { id } = z.object({ id: z.string().uuid() }).parse(request.params); + const { organizationId } = z + .object({ organizationId: z.string().uuid() }) + .parse(request.query); + + const isMember = await checkOrganizationMembership(request.user.id, organizationId); + if (!isMember) { + return reply.status(403).send({ error: 'You are not a member of this organization' }); + } + + const group = await exceptionService.getErrorGroupById(id); + if (!group || group.organizationId !== organizationId) { + return reply.status(404).send({ error: 'Error group not found' }); + } + + const duplicates = await exceptionService.findDuplicateErrorGroups(id, organizationId); + return reply.send({ duplicates }); + } catch (error: any) { + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Validation error', details: error.errors }); + } + console.error('Error finding duplicate error groups:', error); + return reply.status(500).send({ error: 'Failed to find duplicate error groups' }); + } + } + ); + + /** + * POST /api/v1/error-groups/:id/merge + * Merge the given source groups into this group. + */ + fastify.post( + '/api/v1/error-groups/:id/merge', + { + config: { rateLimit: { max: 20, timeWindow: '1 minute' } }, + }, + async (request: any, reply) => { + try { + const { id } = z.object({ id: z.string().uuid() }).parse(request.params); + const body = z + .object({ + organizationId: z.string().uuid(), + sourceIds: z.array(z.string().uuid()).min(1).max(100), + }) + .parse(request.body); + + const isMember = await checkOrganizationMembership(request.user.id, body.organizationId); + if (!isMember) { + return reply.status(403).send({ error: 'You are not a member of this organization' }); + } + + const group = await exceptionService.getErrorGroupById(id); + if (!group || group.organizationId !== body.organizationId) { + return reply.status(404).send({ error: 'Error group not found' }); + } + + const merged = await exceptionService.mergeErrorGroups(id, body.sourceIds, body.organizationId); + if (!merged) { + return reply.status(404).send({ error: 'Error group not found' }); + } + return reply.send(merged); + } catch (error: any) { + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Validation error', details: error.errors }); + } + console.error('Error merging error groups:', error); + return reply.status(500).send({ error: 'Failed to merge error groups' }); + } + } + ); + /** * GET /api/v1/error-groups/:id/trend * Get error group occurrence trend (time-series) diff --git a/packages/backend/src/modules/exceptions/service.ts b/packages/backend/src/modules/exceptions/service.ts index 8dd693c2..77333ac9 100644 --- a/packages/backend/src/modules/exceptions/service.ts +++ b/packages/backend/src/modules/exceptions/service.ts @@ -26,7 +26,7 @@ export class ExceptionService { * The trigger will automatically update/create the error group */ async createException(params: CreateExceptionParams): Promise { - const { organizationId, projectId, logId, parsedData, fingerprint } = params; + const { organizationId, projectId, logId, parsedData, fingerprint, service, topFrame } = params; return await this.db.transaction().execute(async (trx) => { const exception = await trx @@ -41,6 +41,8 @@ export class ExceptionService { fingerprint, raw_stack_trace: parsedData.rawStackTrace, frame_count: parsedData.frames.length, + service: service ?? null, + top_frame: topFrame ?? null, }) .returning('id') .executeTakeFirstOrThrow(); @@ -384,6 +386,154 @@ export class ExceptionService { return this.getErrorGroupById(groupId); } + /** + * Find other error groups in the same org that share this group's exception + * type and message (the shape a fingerprint split produces). Candidates to + * merge into this group. + */ + async findDuplicateErrorGroups( + groupId: string, + organizationId: string + ): Promise> { + const target = await this.db + .selectFrom('error_groups') + .select(['exception_type', 'exception_message', 'project_id']) + .where('id', '=', groupId) + .where('organization_id', '=', organizationId) + .executeTakeFirst(); + + if (!target) return []; + + let query = this.db + .selectFrom('error_groups') + .select(['id', 'occurrence_count', 'first_seen', 'last_seen']) + .where('organization_id', '=', organizationId) + .where('exception_type', '=', target.exception_type) + .where('id', '!=', groupId); + + // Error groups are unique per (org, project, fingerprint), so the same + // type+message legitimately exists in sibling projects; only surface merge + // candidates from the target's own project. + query = + target.project_id === null + ? query.where('project_id', 'is', null) + : query.where('project_id', '=', target.project_id); + + query = + target.exception_message === null + ? query.where('exception_message', 'is', null) + : query.where('exception_message', '=', target.exception_message); + + const rows = await query.execute(); + return rows.map((r) => ({ + id: r.id, + occurrenceCount: Number(r.occurrence_count), + firstSeen: r.first_seen as Date, + lastSeen: r.last_seen as Date, + })); + } + + /** + * Merge source error groups into a target group: reassign their exceptions to + * the target fingerprint, fold their counts / services / first-last seen into + * the target, then delete the now-empty source groups. All org-scoped and + * transactional. + */ + async mergeErrorGroups( + targetId: string, + sourceIds: string[], + organizationId: string + ): Promise { + await this.db.transaction().execute(async (trx) => { + const target = await trx + .selectFrom('error_groups') + .select([ + 'id', + 'fingerprint', + 'project_id', + 'occurrence_count', + 'affected_services', + 'first_seen', + 'last_seen', + ]) + .where('id', '=', targetId) + .where('organization_id', '=', organizationId) + .executeTakeFirst(); + + if (!target) return; + + // Sources must live in the target's own project: the same fingerprint can + // exist in sibling projects (unique per org+project+fingerprint), so an + // org-only filter could fold in, and delete, another project's group. + let sourcesQuery = trx + .selectFrom('error_groups') + .select(['id', 'fingerprint', 'occurrence_count', 'affected_services', 'first_seen', 'last_seen']) + .where('organization_id', '=', organizationId) + .where('id', 'in', sourceIds) + .where('id', '!=', targetId); + sourcesQuery = + target.project_id === null + ? sourcesQuery.where('project_id', 'is', null) + : sourcesQuery.where('project_id', '=', target.project_id); + const sources = await sourcesQuery.execute(); + + if (sources.length === 0) return; + + const sourceFingerprints = sources.map((s) => s.fingerprint); + + // Point the merged-in exceptions at the target's fingerprint so trend/logs + // queries for the target include them. Scope by project too: a sibling + // project's exceptions can share a fingerprint string and must not be + // retagged by this merge. + let retag = trx + .updateTable('exceptions') + .set({ fingerprint: target.fingerprint }) + .where('organization_id', '=', organizationId) + .where('fingerprint', 'in', sourceFingerprints); + retag = + target.project_id === null + ? retag.where('project_id', 'is', null) + : retag.where('project_id', '=', target.project_id); + await retag.execute(); + + const addOccurrences = sources.reduce((sum, s) => sum + Number(s.occurrence_count), 0); + const mergedServices = Array.from( + new Set([ + ...((target.affected_services as string[] | null) || []), + ...sources.flatMap((s) => (s.affected_services as string[] | null) || []), + ]) + ); + const times = [target, ...sources]; + const firstSeen = times + .map((g) => new Date(g.first_seen as unknown as string)) + .reduce((a, b) => (a < b ? a : b)); + const lastSeen = times + .map((g) => new Date(g.last_seen as unknown as string)) + .reduce((a, b) => (a > b ? a : b)); + + await trx + .updateTable('error_groups') + .set({ + occurrence_count: Number(target.occurrence_count) + addOccurrences, + affected_services: mergedServices, + first_seen: firstSeen, + last_seen: lastSeen, + updated_at: new Date(), + }) + .where('id', '=', targetId) + .where('organization_id', '=', organizationId) + .execute(); + + await trx + .deleteFrom('error_groups') + .where('organization_id', '=', organizationId) + .where('id', 'in', sources.map((s) => s.id)) + .execute(); + }); + + return this.getErrorGroupById(targetId); + } + /** * Get error group trend (time-series data) */ diff --git a/packages/backend/src/modules/exceptions/types.ts b/packages/backend/src/modules/exceptions/types.ts index 05569a9c..ffdb487b 100644 --- a/packages/backend/src/modules/exceptions/types.ts +++ b/packages/backend/src/modules/exceptions/types.ts @@ -47,4 +47,15 @@ export interface CreateExceptionParams { logId: string; parsedData: ParsedException; fingerprint: string; + /** + * Service that emitted the error log. Carried on the exception row so the + * error-group trigger can attribute the service on every storage engine, + * not just TimescaleDB (where logs live in Postgres). + */ + service?: string | null; + /** + * Raw "file:function" of the first app frame (throw site). Carried so the + * trigger can compute the group's coarse merge key for auto-merge. + */ + topFrame?: string | null; } diff --git a/packages/backend/src/modules/ingestion/service.ts b/packages/backend/src/modules/ingestion/service.ts index a3f814e7..b0dc425d 100644 --- a/packages/backend/src/modules/ingestion/service.ts +++ b/packages/backend/src/modules/ingestion/service.ts @@ -17,6 +17,7 @@ import { hooks, HookExecutionError } from '../../hooks/index.js'; import type { BeforeIngestContext } from '../../hooks/index.js'; import { hub } from '@logtide/core'; import { isInternalLoggingEnabled } from '../../utils/internal-logger.js'; +import { createSkewTracker } from './skew.js'; export interface IngestRejection { /** Index of the record in the submitted batch. */ @@ -156,10 +157,13 @@ export class IngestionService { // Convert logs to reservoir LogRecord format // Note: reservoir handles null byte sanitization internally + // Clock skew (#279): observed inside the existing map so a large batch costs + // one extra subtraction per record and one Date.now() per batch. + const skewTracker = createSkewTracker(Date.now()); let records: BeforeIngestContext['records'] = logs.map((log) => { // Extract hostname if not already set in metadata const hostname = log.metadata?.hostname || extractHostname(log); - + const metadata = { ...log.metadata, ...(hostname && { hostname }), @@ -167,8 +171,11 @@ export class IngestionService { const hasMetadata = Object.keys(metadata).length > 0; + const time = typeof log.time === 'string' ? new Date(log.time) : log.time; + skewTracker.observe(time); + return { - time: typeof log.time === 'string' ? new Date(log.time) : log.time, + time, projectId, service: sanitizeForPostgres(log.service), level: log.level as ReservoirLogLevel, @@ -180,6 +187,20 @@ export class IngestionService { }; }); + // Clock skew signal (#279). Fire-and-forget, never rejects: the records above + // are written unchanged. Guarded on organizationId like the other ingestion + // health counters, since anonymous ingestion has no org to attribute to. + const skew = skewTracker.summary(); + if (skew && organizationId) { + metering.record({ + type: 'ingestion.timestamp_skew', + quantity: skew.count, + organizationId, + projectId, + metadata: { maxPastMs: skew.maxPastMs, maxFutureMs: skew.maxFutureMs }, + }); + } + // Lifecycle hook (#216): last interception point before the reservoir // write. Hooks may reject (throw) or mutate/replace `records`. // hasHandlers guard keeps the OSS no-hooks path at zero overhead diff --git a/packages/backend/src/modules/ingestion/skew.ts b/packages/backend/src/modules/ingestion/skew.ts new file mode 100644 index 00000000..5a7922b1 --- /dev/null +++ b/packages/backend/src/modules/ingestion/skew.ts @@ -0,0 +1,73 @@ +/** + * Ingestion clock skew detection (#279). + * + * A log whose client-supplied `time` sits far from the server clock is stored + * and is visible in the UI, but threshold alert rules count logs in + * [now - time_window, now] and so can never see it. The largest configurable + * window is 24h (alerts/routes.ts), which is where the past threshold comes + * from: past that point, no threshold rule in the product can match the log. + * + * Observation only. Skewed records are written unchanged. + * + * Always on: these thresholds are load-bearing product invariants, not + * tunable knobs, so they are hardcoded rather than read from config. There is + * no "disable" state. + */ + +/** 24h: the largest alert `time_window` (alerts/routes.ts). Past this point, + * no threshold rule in the product can ever count the log. Derived, not guessed. */ +const PAST_THRESHOLD_MS = 86400000; + +/** 5m: room for NTP jitter. */ +const FUTURE_THRESHOLD_MS = 300000; + +export interface SkewSummary { + /** Number of skewed records in the batch. */ + count: number; + /** Worst past delta seen, in ms. 0 when nothing was skewed into the past. */ + maxPastMs: number; + /** Worst future delta seen, in ms. 0 when nothing was skewed into the future. */ + maxFutureMs: number; +} + +export interface SkewTracker { + observe(time: Date | undefined): void; + summary(): SkewSummary | null; +} + +/** + * Single-pass tracker. `now` is read once per batch by the caller so that a + * large batch is measured against one instant, not a drifting one. + */ +export function createSkewTracker(now: number): SkewTracker { + let count = 0; + let maxPastMs = 0; + let maxFutureMs = 0; + + return { + observe(time: Date | undefined): void { + if (!time) return; + + // NaN for an Invalid Date. Both comparisons below are false for NaN, so a + // malformed timestamp is never miscounted as skew: that is a separate + // problem and counting it here would make this signal dishonest. + const delta = now - time.getTime(); + + if (delta > PAST_THRESHOLD_MS) { + count++; + if (delta > maxPastMs) maxPastMs = delta; + return; + } + + if (delta < -FUTURE_THRESHOLD_MS) { + count++; + const ahead = -delta; + if (ahead > maxFutureMs) maxFutureMs = ahead; + } + }, + + summary(): SkewSummary | null { + return count > 0 ? { count, maxPastMs, maxFutureMs } : null; + }, + }; +} diff --git a/packages/backend/src/modules/metering/recorder.ts b/packages/backend/src/modules/metering/recorder.ts index 76b8feec..84485b04 100644 --- a/packages/backend/src/modules/metering/recorder.ts +++ b/packages/backend/src/modules/metering/recorder.ts @@ -29,7 +29,15 @@ export class MeteringRecorder { } record(event: MeteringEvent): void { - if (!config.METERING_ENABLED) return; + // METERING_ENABLED gates usage/resource metering (billing-shaped). The + // `ingestion.*` types are operational health counters, not usage: they are + // already excluded from tenant usage breakdowns by the `NOT LIKE + // 'ingestion.%'` filter in breakdown.ts. A billing toggle must not be able + // to silently blind operators to ingestion health (including the + // pii_rejected safety counter), so those types bypass this gate. The + // hardCap drop-guard below still applies: it is a memory safety valve, + // not a feature toggle. + if (!config.METERING_ENABLED && !event.type.startsWith('ingestion.')) return; if (this.buffer.length >= this.hardCap) { this.dropped++; return; diff --git a/packages/backend/src/modules/metering/types.ts b/packages/backend/src/modules/metering/types.ts index 34680053..c5a24f76 100644 --- a/packages/backend/src/modules/metering/types.ts +++ b/packages/backend/src/modules/metering/types.ts @@ -14,7 +14,13 @@ export type MeteringEventType = | 'ingestion.pii_rejected' | 'ingestion.detection_enqueue_failed' | 'ingestion.exception_enqueue_failed' - | 'ingestion.identifier_failed'; + | 'ingestion.identifier_failed' + // Clock skew at ingestion (#279): not billed, surfaced in admin stats and per project. + | 'ingestion.timestamp_skew' + // Clock skew for spans/metrics (span-metric-skew): same rationale as above, + // kept under the ingestion.* prefix (see breakdown.ts / recorder.ts). + | 'ingestion.span_timestamp_skew' + | 'ingestion.metric_timestamp_skew'; export interface MeteringEvent { type: MeteringEventType; diff --git a/packages/backend/src/modules/metrics/service.ts b/packages/backend/src/modules/metrics/service.ts index e75ae1f4..8986a826 100644 --- a/packages/backend/src/modules/metrics/service.ts +++ b/packages/backend/src/modules/metrics/service.ts @@ -1,5 +1,7 @@ import { reservoir } from '../../database/reservoir.js'; import { projectsService } from '../projects/service.js'; +import { metering } from '../metering/index.js'; +import { createSkewTracker } from '../ingestion/skew.js'; import type { MetricRecord, AggregationInterval, @@ -14,14 +16,34 @@ export class MetricsService { ): Promise { if (records.length === 0) return 0; - const enriched = records.map((r) => ({ - ...r, - projectId, - organizationId, - })); + // Clock skew (span-metric-skew): observed on the data point timestamp + // (r.time), inside the same map as the enrichment above. No PII masking + // exists on this path, so there is nothing to order against. 24h is a + // judgement call (see traces/service.ts ingestSpans for the full rationale), + // not a derivation: the harm here is a metric invisible in dashboard windows. + const metricSkewTracker = createSkewTracker(Date.now()); + const enriched = records.map((r) => { + metricSkewTracker.observe(r.time); + return { + ...r, + projectId, + organizationId, + }; + }); const result = await reservoir.ingestMetrics(enriched); + const metricSkew = metricSkewTracker.summary(); + if (metricSkew && organizationId) { + metering.record({ + type: 'ingestion.metric_timestamp_skew', + quantity: metricSkew.count, + organizationId, + projectId, + metadata: { maxPastMs: metricSkew.maxPastMs, maxFutureMs: metricSkew.maxFutureMs }, + }); + } + // Mark the project as having metrics (debounced, fire-and-forget) projectsService.markHasData(projectId, 'metrics').catch(() => {}); diff --git a/packages/backend/src/modules/projects/routes.ts b/packages/backend/src/modules/projects/routes.ts index a51672ce..de235571 100644 --- a/packages/backend/src/modules/projects/routes.ts +++ b/packages/backend/src/modules/projects/routes.ts @@ -158,6 +158,28 @@ export async function projectsRoutes(fastify: FastifyInstance) { } }); + // Get project ingestion health (#279: clock skew warning surface) + fastify.get('/:id/ingestion-health', async (request: any, reply) => { + try { + const { id } = projectIdSchema.parse(request.params); + + // Access control + org scoping: getProjectById joins organization_members + // on the caller, so a project outside the caller's orgs reads as 404. + const project = await projectsService.getProjectById(id, request.user.id); + if (!project) { + return reply.status(404).send({ error: 'Project not found' }); + } + + const health = await projectsService.getIngestionHealth(project.organizationId, id); + return reply.send(health); + } catch (error) { + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Invalid project ID' }); + } + throw error; + } + }); + // Restore a soft-deleted project fastify.post('/:id/restore', async (request: any, reply) => { try { diff --git a/packages/backend/src/modules/projects/service.ts b/packages/backend/src/modules/projects/service.ts index d625f001..026ed426 100644 --- a/packages/backend/src/modules/projects/service.ts +++ b/packages/backend/src/modules/projects/service.ts @@ -1,3 +1,4 @@ +import { sql } from 'kysely'; import { db } from '../../database/connection.js'; import type { Project, StatusPageVisibility } from '@logtide/shared'; import bcrypt from 'bcrypt'; @@ -38,6 +39,30 @@ export interface UpdateProjectInput { statusPagePassword?: string; } +export interface ProjectSkewHealth { + /** Total skewed records seen in the last 24h. */ + count24h: number; + /** Worst past delta across the window, in ms. */ + maxPastMs: number; + /** Worst future delta across the window, in ms. */ + maxFutureMs: number; + lastSeenAt: string; +} + +/** Per-signal skew health (span-metric-skew). Each entry is null when that + * signal had no skew in the 24h window. */ +export interface ProjectSkewHealthMap { + logs: ProjectSkewHealth | null; + spans: ProjectSkewHealth | null; + metrics: ProjectSkewHealth | null; +} + +const SKEW_TYPE_TO_SIGNAL: Record = { + 'ingestion.timestamp_skew': 'logs', + 'ingestion.span_timestamp_skew': 'spans', + 'ingestion.metric_timestamp_skew': 'metrics', +}; + // All columns returned on every Project fetch const PROJECT_COLUMNS = [ 'id', @@ -232,6 +257,69 @@ export class ProjectsService { return mapProject(project); } + /** + * Per-project ingestion health (#279, span-metric-skew). Reads the clock skew + * counters written by ingestLogs/ingestSpans/ingestMetrics, aggregated in SQL + * with one round trip per project (GROUP BY type keeps it a single query + * instead of three). Filters on (organization_id, type, time) plus project_id, + * so the planner favors idx_metering_org_type_time: type is highly selective + * while (org, project, 24h) alone matches every metering row for the project. + * metadata->>'x' yields text, so each field is cast to float8 before MAX to + * avoid a lexicographic (string) max. With GROUP BY, a signal with zero + * matching rows simply produces no row (rather than one row of NULLs), so + * each entry of the returned map starts null and is only filled in for types + * that actually matched. + */ + async getIngestionHealth( + organizationId: string, + projectId: string, + ): Promise<{ skew: ProjectSkewHealthMap }> { + const since = new Date(Date.now() - 24 * 60 * 60 * 1000); + + const query = sql<{ + type: string; + count24h: number | string | null; + max_past_ms: number | string | null; + max_future_ms: number | string | null; + last_seen_at: Date | string | null; + }>` + SELECT + type, + COALESCE(SUM(quantity), 0)::float8 AS count24h, + COALESCE(MAX((metadata->>'maxPastMs')::float8), 0) AS max_past_ms, + COALESCE(MAX((metadata->>'maxFutureMs')::float8), 0) AS max_future_ms, + MAX(time) AS last_seen_at + FROM metering_events + WHERE organization_id = ${organizationId} + AND project_id = ${projectId} + AND type IN ('ingestion.timestamp_skew', 'ingestion.span_timestamp_skew', 'ingestion.metric_timestamp_skew') + AND time >= ${since} + GROUP BY type + `; + const result = await query.execute(db); + + const toNumber = (value: number | string | null): number => + typeof value === 'number' ? value : parseFloat(value ?? '0'); + + const skew: ProjectSkewHealthMap = { logs: null, spans: null, metrics: null }; + + for (const row of result.rows) { + const signal = SKEW_TYPE_TO_SIGNAL[row.type]; + if (!signal) continue; + + const lastSeenAt = row.last_seen_at instanceof Date ? row.last_seen_at : new Date(row.last_seen_at as string); + + skew[signal] = { + count24h: toNumber(row.count24h), + maxPastMs: toNumber(row.max_past_ms), + maxFutureMs: toNumber(row.max_future_ms), + lastSeenAt: lastSeenAt.toISOString(), + }; + } + + return { skew }; + } + /** * Update a project (only active projects can be updated) */ diff --git a/packages/backend/src/modules/traces/service.ts b/packages/backend/src/modules/traces/service.ts index 243a40e4..3cacec52 100644 --- a/packages/backend/src/modules/traces/service.ts +++ b/packages/backend/src/modules/traces/service.ts @@ -2,8 +2,9 @@ import { db } from '../../database/index.js'; import { pool } from '../../database/connection.js'; import { reservoir } from '../../database/reservoir.js'; import { projectsService } from '../projects/service.js'; -import { recordSpanIngestion } from '../metering/index.js'; +import { recordSpanIngestion, metering } from '../metering/index.js'; import { piiMaskingService } from '../pii-masking/service.js'; +import { createSkewTracker } from '../ingestion/skew.js'; import type { TransformedSpan, AggregatedTrace } from '../otlp/trace-transformer.js'; import type { SpanRecord as ReservoirSpanRecord, @@ -142,8 +143,41 @@ export class TracesService { ); } + // Clock skew (span-metric-skew): observed on end_time, not start_time. A span + // that STARTED 27h ago is routinely legitimate (batch jobs, long transactions, + // long-poll), because spans are exported at end; clock skew shifts both ends + // together, so end_time is what separates the two cases. Counted over the + // post-masking survivors only, to match the log path's semantics of only + // counting records that actually get written (a second loop, not folded into + // the map above, since masking runs after it here). + // + // 24h is a judgement call here, not a derivation: for logs it is exactly the + // maximum alert time_window, so past it no rule can ever match. No alert rules + // run over spans, so there is no equivalent statement; the harm is weaker + // (invisible in trace view default time windows). Reused for consistency. + // + // Accepted limitation: a collector draining a multi-day persistent queue, or an + // offline edge agent, produces genuinely old end_time with a correct clock and + // will be flagged the same as real skew. Same accepted equivalence class as log + // backfill; we warn rather than reject because we cannot tell them apart. + const spanSkewTracker = createSkewTracker(Date.now()); + for (const span of spansToStore) { + spanSkewTracker.observe(span.endTime); + } + const spanSkew = spanSkewTracker.summary(); + const result = await reservoir.ingestSpans(spansToStore); + if (spanSkew && organizationId) { + metering.record({ + type: 'ingestion.span_timestamp_skew', + quantity: spanSkew.count, + organizationId, + projectId, + metadata: { maxPastMs: spanSkew.maxPastMs, maxFutureMs: spanSkew.maxFutureMs }, + }); + } + // Metering: record ingested span count (fire-and-forget; activates // the tracing.max_spans_monthly quota in the capability system). if (organizationId) { diff --git a/packages/backend/src/modules/users/routes.ts b/packages/backend/src/modules/users/routes.ts index fb6b0c8d..a5173ae2 100644 --- a/packages/backend/src/modules/users/routes.ts +++ b/packages/backend/src/modules/users/routes.ts @@ -1,6 +1,10 @@ import type { FastifyInstance } from 'fastify'; import { z } from 'zod'; +import { hub } from '@logtide/core'; +import { isInternalLoggingEnabled } from '../../utils/internal-logger.js'; +import { AuthError, AuthErrorCode } from '../auth/providers/types.js'; import { usersService } from './service.js'; +import { authenticationService } from '../auth/authentication-service.js'; import { config } from '../../config/index.js'; import { settingsService } from '../settings/service.js'; import { bootstrapService } from '../bootstrap/service.js'; @@ -52,11 +56,29 @@ export async function usersRoutes(fastify: FastifyInstance) { const user = await usersService.createUser(body); - // Automatically log in the new user - const session = await usersService.login({ - email: body.email, - password: body.password, - }); + // Automatically log in the new user through the provider registry. + // This also creates the user_identities link for the local provider. + // If this fails for a transient reason the user row is already committed, + // so we return 201 with no session instead of propagating a 500. The + // account is fully recoverable via a subsequent /login. + let session: { token: string; expiresAt: Date } | null = null; + try { + const result = await authenticationService.authenticateWithProvider('local', { + email: body.email, + password: body.password, + }); + session = result.session; + } catch (autoLoginError) { + // The user row is already committed, so this stays a 201 without a + // session (recoverable via /login), but the cause must be visible to + // operators: a persistent failure here silently degrades every signup. + if (isInternalLoggingEnabled()) { + hub.captureLog('error', '[Auth] Post-registration auto-login failed', { + userId: user.id, + error: autoLoginError instanceof Error ? autoLoginError.message : String(autoLoginError), + }); + } + } await auditLogService.record({ action: 'user.registered', @@ -72,10 +94,12 @@ export async function usersRoutes(fastify: FastifyInstance) { name: user.name, is_admin: user.is_admin, }, - session: { - token: session.token, - expiresAt: session.expiresAt, - }, + ...(session ? { + session: { + token: session.token, + expiresAt: session.expiresAt, + }, + } : {}), }); } catch (error) { if (error instanceof z.ZodError) { @@ -111,14 +135,7 @@ export async function usersRoutes(fastify: FastifyInstance) { try { parsedBody = loginSchema.parse(request.body); - const session = await usersService.login(parsedBody); - const user = await usersService.getUserById(session.userId); - - if (!user) { - return reply.status(500).send({ - error: 'Internal server error', - }); - } + const { user, session } = await authenticationService.authenticateWithProvider('local', parsedBody); await auditLogService.record({ action: 'auth.login_succeeded', @@ -147,17 +164,30 @@ export async function usersRoutes(fastify: FastifyInstance) { }); } - if (error instanceof Error) { - if (error.message.includes('Invalid')) { - // parsedBody is defined here (zod succeeded before the service threw) + if (error instanceof AuthError) { + // Classify by the provider's typed code, not the message string, so a + // reword of a user-facing message can never silently turn a login + // failure into a 500 or drop its audit row. + const mapping: Partial> = { + [AuthErrorCode.INVALID_CREDENTIALS]: { status: 401, reason: 'invalid_credentials' }, + [AuthErrorCode.SSO_REQUIRED]: { status: 401, reason: 'sso_required' }, + [AuthErrorCode.USER_DISABLED]: { status: 401, reason: 'account_disabled' }, + [AuthErrorCode.PROVIDER_UNAVAILABLE]: { status: 503, reason: 'provider_unavailable' }, + }; + const mapped = mapping[error.code]; + + if (mapped) { + // parsedBody is defined here (zod succeeded before the service threw). + // The reason lands only in the audit log, never in the HTTP response, + // so recording it does not weaken the anti-enumeration behavior. await auditLogService.record({ action: 'auth.login_failed', outcome: 'failure', organizationId: null, actor: { type: 'user', id: null, label: parsedBody?.email ?? null }, - metadata: { method: 'local' }, + metadata: { method: 'local', reason: mapped.reason }, }); - return reply.status(401).send({ + return reply.status(mapped.status).send({ error: error.message, }); } diff --git a/packages/backend/src/modules/users/service.ts b/packages/backend/src/modules/users/service.ts index 3dc1f36b..2c5819f1 100644 --- a/packages/backend/src/modules/users/service.ts +++ b/packages/backend/src/modules/users/service.ts @@ -5,7 +5,6 @@ import { db } from '../../database/connection.js'; import { CacheManager, CACHE_TTL } from '../../utils/cache.js'; const SALT_ROUNDS = 10; -const SESSION_DURATION_DAYS = 30; // Constant key for the Postgres transaction-scoped advisory lock that // serializes the first-admin promotion decision across concurrent @@ -18,11 +17,6 @@ export interface CreateUserInput { name: string; } -export interface LoginInput { - email: string; - password: string; -} - export interface SessionInfo { sessionId: string; userId: string; @@ -149,67 +143,6 @@ export class UsersService { }; } - /** - * Authenticate a user and create a session - */ - async login(input: LoginInput): Promise { - // Find user by email (case-insensitive, matching how emails are stored) - const user = await db - .selectFrom('users') - .select(['id', 'email', 'password_hash', 'disabled']) - .where('email', '=', input.email.toLowerCase().trim()) - .executeTakeFirst(); - - if (!user) { - throw new Error('Invalid email or password'); - } - - // Check if user has a local password (external auth users may not) - if (!user.password_hash) { - throw new Error('Please log in using your organization SSO'); - } - - // Verify password - const isValidPassword = await this.verifyPassword(input.password, user.password_hash); - if (!isValidPassword) { - throw new Error('Invalid email or password'); - } - - // Check if account is disabled - if (user.disabled) { - throw new Error('This account has been disabled'); - } - - // Update last login - await db - .updateTable('users') - .set({ last_login: new Date() }) - .where('id', '=', user.id) - .execute(); - - // Create session - const token = this.generateToken(); - const expiresAt = new Date(); - expiresAt.setDate(expiresAt.getDate() + SESSION_DURATION_DAYS); - - const session = await db - .insertInto('sessions') - .values({ - user_id: user.id, - token, - expires_at: expiresAt, - }) - .returning(['id', 'token', 'expires_at']) - .executeTakeFirstOrThrow(); - - return { - sessionId: session.id, - userId: user.id, - token: session.token, - expiresAt: new Date(session.expires_at), - }; - } - /** * Validate a session token and return user info * Cached for performance - session validation happens on every request diff --git a/packages/backend/src/queue/jobs/digest-dispatch.ts b/packages/backend/src/queue/jobs/digest-dispatch.ts new file mode 100644 index 00000000..3595b115 --- /dev/null +++ b/packages/backend/src/queue/jobs/digest-dispatch.ts @@ -0,0 +1,112 @@ +/** + * Digest Dispatch Job Processor + * + * Runs hourly (single static cron registered at worker boot). Scans the active + * digest configs, decides which ones are due at the current UTC hour and + * enqueues one digest-generation job per due config. + * + * This indirection exists because graphile-worker cron items are fixed at + * runner start: per-org cron jobs cannot be added or removed while the worker + * is running, so config CRUD would need a worker restart to take effect. A + * single static dispatch cron plus a due-check keeps schedule changes live on + * both queue backends. + */ + +import { db } from '../../database/connection.js'; +import { createQueue } from '../connection.js'; +import { hub } from '@logtide/core'; +import type { IJob } from '../abstractions/types.js'; +import type { DigestJobPayload } from '../../modules/digests/scheduler.js'; + +interface DispatchableConfig { + frequency: 'daily' | 'weekly'; + delivery_hour: number; + delivery_day_of_week: number | null; + last_sent_at: Date | null; +} + +// A digest fires at most once per delivery slot; anything sent less than two +// hours ago is a duplicate trigger (worker restart, overlapping cron tick). +const DOUBLE_FIRE_GUARD_MS = 2 * 60 * 60 * 1000; + +export function isConfigDue(config: DispatchableConfig, now: Date): boolean { + if (now.getUTCHours() !== config.delivery_hour) { + return false; + } + + if (config.frequency === 'weekly' && now.getUTCDay() !== config.delivery_day_of_week) { + return false; + } + + if ( + config.last_sent_at && + now.getTime() - new Date(config.last_sent_at).getTime() < DOUBLE_FIRE_GUARD_MS + ) { + return false; + } + + return true; +} + +export async function processDigestDispatch(_job: IJob>): Promise { + const now = new Date(); + + // tenant-scope-ok: system dispatch cron sweeps every org's digest config by design + const configs = await db + .selectFrom('digest_configs') + .select([ + 'id', + 'organization_id', + 'frequency', + 'delivery_hour', + 'delivery_day_of_week', + 'last_sent_at', + ]) + .where('enabled', '=', true) + .execute(); + + const due = configs.filter((config) => + isConfigDue( + { + frequency: config.frequency, + delivery_hour: config.delivery_hour, + delivery_day_of_week: config.delivery_day_of_week, + // Already a Date (Kysely Timestamp select type); no string round-trip. + last_sent_at: config.last_sent_at ?? null, + }, + now + ) + ); + + if (due.length === 0) { + return; + } + + hub.captureLog('info', `[DigestDispatch] ${due.length} digest(s) due at ${now.toISOString()}`); + + const queue = createQueue('digest-generation'); + const hourKey = now.toISOString().slice(0, 13); // YYYY-MM-DDTHH (UTC) + + for (const config of due) { + await queue.add( + 'digest-generation', + { + organizationId: config.organization_id, + digestConfigId: config.id, + frequency: config.frequency, + }, + { + // Hour-keyed so a duplicate dispatch within the same slot dedupes + jobKey: `digest:${config.id}:${hourKey}`, + removeOnComplete: true, + } + ); + + // tenant-scope-ok: id comes from the enabled-configs sweep above; system cron has no org context + await db + .updateTable('digest_configs') + .set({ last_sent_at: now }) + .where('id', '=', config.id) + .execute(); + } +} diff --git a/packages/backend/src/queue/jobs/error-notification.ts b/packages/backend/src/queue/jobs/error-notification.ts index abd10dd3..12336fd1 100644 --- a/packages/backend/src/queue/jobs/error-notification.ts +++ b/packages/backend/src/queue/jobs/error-notification.ts @@ -28,6 +28,8 @@ export interface ErrorNotificationJobData { language: ExceptionLanguage; service: string; isNewErrorGroup: boolean; + /** True when a previously resolved error group recurred (a regression). */ + isRegression?: boolean; } // Create the queue @@ -72,9 +74,9 @@ async function sendErrorWebhook( organizationId: data.organizationId, projectId: data.projectId ?? null, data: { - title: `${data.isNewErrorGroup ? 'New Error' : 'Error'}: ${data.exceptionType}`, + title: `${data.isRegression ? 'Regression' : data.isNewErrorGroup ? 'New Error' : 'Error'}: ${data.exceptionType}`, message: data.exceptionMessage || `An error occurred in ${data.service}`, - severity: data.isNewErrorGroup ? 'high' : 'medium', + severity: data.isRegression || data.isNewErrorGroup ? 'high' : 'medium', organization: { id: data.organizationId, name: orgName, @@ -88,6 +90,7 @@ async function sendErrorWebhook( language: data.language, service: data.service, is_new: data.isNewErrorGroup, + is_regression: data.isRegression ?? false, link: `${frontendUrl}/dashboard/errors/${errorGroupId}`, }, }); @@ -220,7 +223,9 @@ export async function processErrorNotification(job: IJob } const fingerprint = FingerprintService.generate(parsed); + const topFrame = FingerprintService.topAppFrame(parsed); // Check if this is a new error group (first occurrence with this fingerprint) - const existingGroup = await db + let existingGroup = await db .selectFrom('error_groups') .select(['id', 'occurrence_count', 'status']) .where('fingerprint', '=', fingerprint) .where('organization_id', '=', organizationId) .executeTakeFirst(); + // No exact fingerprint match: auto-merge into an existing group that shares + // the coarse key (same type + normalized message + top app frame), so a + // stack that differs only in its deep frames does not spawn a duplicate + // group. The key is computed by the same Postgres function on both sides. + let effectiveFingerprint = fingerprint; + if (!existingGroup && topFrame) { + const mergeMatch = await db + .selectFrom('error_groups') + .select(['id', 'fingerprint', 'occurrence_count', 'status']) + .where('organization_id', '=', organizationId) + .where('project_id', '=', projectId) + .where( + 'merge_key', + '=', + sql`logtide_merge_key(${parsed.exceptionType}, ${parsed.exceptionMessage}, ${topFrame})` + ) + .orderBy('first_seen', 'asc') + .executeTakeFirst(); + + if (mergeMatch) { + effectiveFingerprint = mergeMatch.fingerprint; + existingGroup = { + id: mergeMatch.id, + occurrence_count: mergeMatch.occurrence_count, + status: mergeMatch.status, + }; + } + } + const isNewErrorGroup = !existingGroup; + // A previously resolved error that recurs is a regression. + const isRegression = existingGroup?.status === 'resolved'; const exceptionId = await exceptionService.createException({ organizationId, projectId, logId: log.id, parsedData: parsed, - fingerprint, + fingerprint: effectiveFingerprint, + service: log.service, + topFrame, }); + // Reopen a resolved group so the regression is visible and not silently + // folded into a "resolved" bucket. + if (isRegression && existingGroup) { + await db + .updateTable('error_groups') + .set({ status: 'open', resolved_at: null, resolved_by: null, updated_at: new Date() }) + .where('id', '=', existingGroup.id) + .where('organization_id', '=', organizationId) + .execute(); + } + stats.parsed++; console.log( @@ -104,12 +150,13 @@ export async function processExceptionParsing(job: IJob exceptionId, organizationId, projectId, - fingerprint, + fingerprint: effectiveFingerprint, exceptionType: parsed.exceptionType, exceptionMessage: parsed.exceptionMessage, language: parsed.language, service: log.service, isNewErrorGroup, + isRegression, }; await errorNotificationQueue.add('error-notification', notificationData, { diff --git a/packages/backend/src/server.ts b/packages/backend/src/server.ts index 1733622a..ccc9f346 100644 --- a/packages/backend/src/server.ts +++ b/packages/backend/src/server.ts @@ -50,6 +50,8 @@ import { auditLogRoutes, auditLogService } from './modules/audit-log/index.js'; import { bootstrapService } from './modules/bootstrap/index.js'; import { runDataAvailabilityBackfill } from './modules/projects/data-availability-backfill.js'; import { notificationChannelsRoutes } from './modules/notification-channels/index.js'; +import { digestsRoutes } from './modules/digests/routes.js'; +import { digestsPublicRoutes } from './modules/digests/public-routes.js'; import { webhookDeliveriesRoutes } from './modules/webhooks/routes.js'; import internalLoggingPlugin from './plugins/internal-logging-plugin.js'; import { initializeInternalLogging, shutdownInternalLogging } from './utils/internal-logger.js'; @@ -182,6 +184,9 @@ export async function build(opts = {}) { await fastify.register(projectsRoutes, { prefix: '/api/v1/projects' }); await fastify.register(notificationsRoutes, { prefix: '/api/v1/notifications' }); await fastify.register(notificationChannelsRoutes, { prefix: '/api/v1/notification-channels' }); + await fastify.register(digestsRoutes, { prefix: '/api/v1/digests' }); + // Public one-click unsubscribe: the URL token is the credential, no auth (#154) + await fastify.register(digestsPublicRoutes, { prefix: '/api/v1/digests' }); await fastify.register(webhookDeliveriesRoutes, { prefix: '/api/v1/webhooks' }); await fastify.register(onboardingRoutes, { prefix: '/api/v1/onboarding' }); await fastify.register(alertsRoutes, { prefix: '/api/v1/alerts' }); diff --git a/packages/backend/src/tests/helpers/factories.ts b/packages/backend/src/tests/helpers/factories.ts index 55032d20..954fb890 100644 --- a/packages/backend/src/tests/helpers/factories.ts +++ b/packages/backend/src/tests/helpers/factories.ts @@ -7,14 +7,16 @@ import crypto from 'crypto'; */ export async function createTestUser(overrides: { email?: string; - password?: string; + /** null creates an SSO-only user with no local password */ + password?: string | null; name?: string; + disabled?: boolean; } = {}) { const email = overrides.email || `test-${Date.now()}@example.com`; - const password = overrides.password || 'password123'; + const password = overrides.password === undefined ? 'password123' : overrides.password; const name = overrides.name || 'Test User'; - const hashedPassword = await bcrypt.hash(password, 10); + const hashedPassword = password === null ? null : await bcrypt.hash(password, 10); const user = await db .insertInto('users') @@ -22,6 +24,7 @@ export async function createTestUser(overrides: { email, password_hash: hashedPassword, name, + disabled: overrides.disabled ?? false, }) .returningAll() .executeTakeFirstOrThrow(); diff --git a/packages/backend/src/tests/lib/email-templates.test.ts b/packages/backend/src/tests/lib/email-templates.test.ts index 0cb9692e..ff9b25df 100644 --- a/packages/backend/src/tests/lib/email-templates.test.ts +++ b/packages/backend/src/tests/lib/email-templates.test.ts @@ -18,6 +18,8 @@ import { generateIncidentEmail, generateSigmaDetectionEmail, generateInvitationEmail, + generateDigestEmail, + type DigestEmailData, } from '../../lib/email-templates.js'; describe('Email Templates - Helpers', () => { @@ -752,3 +754,120 @@ describe('Email Templates - Common Elements', () => { expect(result.html).toContain(''); }); }); + +describe('Email Templates - Digest', () => { + const baseData: DigestEmailData = { + organizationName: 'Acme Corp', + frequency: 'daily', + periodLabel: 'last 24 hours', + logVolume: { current: 15000, previous: 12000, trend: '+3000 (+25.0%)' }, + topErrorServices: [ + { service: 'api-', errorCount: 50, previousCount: 20, delta: 30 }, + { service: 'web', errorCount: 10, previousCount: 15, delta: -5 }, + ], + newErrorGroups: [ + { + exceptionType: 'TypeError', + exceptionMessage: 'Cannot read properties of ', + occurrenceCount: 42, + language: 'nodejs', + }, + ], + security: { + totalDetections: 11, + topRules: [{ ruleTitle: 'Suspicious ', severity: 'high', count: 7 }], + openIncidents: 2, + }, + uptime: { + monitorCount: 3, + overallUptimePct: 99.5, + worstMonitors: [{ name: 'API Health', uptimePct: 98.5 }], + }, + unsubscribeUrl: 'https://app.logtide.dev/unsubscribe?token=tok_123', + dashboardUrl: 'https://app.logtide.dev/dashboard', + }; + + it('renders every section in the HTML body', () => { + const result = generateDigestEmail(baseData); + + expect(result.html).toContain(''); + expect(result.html).toContain('Acme Corp'); + expect(result.html).toContain('15,000'); + expect(result.html).toContain('+3000 (+25.0%)'); + expect(result.html).toContain('TypeError'); + expect(result.html).toContain('42'); + expect(result.html).toContain('Suspicious'); + expect(result.html).toContain('99.5'); + expect(result.html).toContain('98.5'); + expect(result.html).toContain('last 24 hours'); + }); + + it('escapes user-controlled values in HTML', () => { + const result = generateDigestEmail(baseData); + + expect(result.html).not.toContain('api-'); + expect(result.html).toContain('api-<gateway>'); + expect(result.html).not.toContain('Suspicious '); + expect(result.html).toContain('Suspicious <Login>'); + expect(result.html).not.toContain('Cannot read properties of '); + }); + + it('uses the unsubscribe link in the footer instead of channel settings', () => { + const result = generateDigestEmail(baseData); + + expect(result.html).toContain('https://app.logtide.dev/unsubscribe?token=tok_123'); + expect(result.html).toContain('Unsubscribe'); + expect(result.html).not.toContain('/dashboard/settings/channels'); + }); + + it('carries all sections in the plaintext fallback', () => { + const result = generateDigestEmail(baseData); + + expect(result.text).toContain('Acme Corp'); + expect(result.text).toContain('Total logs: 15,000'); + expect(result.text).toContain('Trend: +3000 (+25.0%)'); + expect(result.text).toContain('Previous period: 12,000'); + expect(result.text).toContain('api-'); + expect(result.text).toContain('TypeError'); + expect(result.text).toContain('Suspicious '); + expect(result.text).toContain('Open incidents: 2'); + expect(result.text).toContain('99.5'); + expect(result.text).toContain('https://app.logtide.dev/unsubscribe?token=tok_123'); + }); + + it('shows the quiet-period message when there was no log activity', () => { + const quiet: DigestEmailData = { + ...baseData, + logVolume: { current: 0, previous: 0, trend: 'no change' }, + topErrorServices: [], + newErrorGroups: [], + security: { totalDetections: 0, topRules: [], openIncidents: 0 }, + uptime: null, + }; + + const result = generateDigestEmail(quiet); + + expect(result.html).toContain('No activity during this period'); + expect(result.text).toContain('No activity during this period'); + expect(result.text).toContain('quiet'); + }); + + it('omits the uptime section when uptime is null', () => { + const noUptime: DigestEmailData = { ...baseData, uptime: null }; + + const result = generateDigestEmail(noUptime); + + expect(result.html).not.toContain('Uptime'); + expect(result.text).not.toContain('Uptime'); + }); + + it('labels weekly digests as weekly', () => { + const weekly: DigestEmailData = { ...baseData, frequency: 'weekly', periodLabel: 'last 7 days' }; + + const result = generateDigestEmail(weekly); + + expect(result.html).toContain('Weekly Digest'); + expect(result.text).toContain('Weekly Digest'); + expect(result.text).toContain('last 7 days'); + }); +}); diff --git a/packages/backend/src/tests/modules/admin/routes.test.ts b/packages/backend/src/tests/modules/admin/routes.test.ts index 27fd29d2..f29b5217 100644 --- a/packages/backend/src/tests/modules/admin/routes.test.ts +++ b/packages/backend/src/tests/modules/admin/routes.test.ts @@ -844,6 +844,61 @@ describe('Admin Routes', () => { expect(body.counters24h.identifierFailed).toBe(0); expect(body.enrichment).toBeDefined(); }); + + it('reports the timestamp skew counter', async () => { + await db + .insertInto('metering_events') + .values({ + organization_id: testOrg.id, + project_id: testProject.id, + type: 'ingestion.timestamp_skew', + quantity: 7, + metadata: { maxPastMs: 97200000, maxFutureMs: 0 }, + }) + .execute(); + + const response = await app.inject({ + method: 'GET', + url: '/api/v1/admin/stats/ingestion-health', + headers: { authorization: `Bearer ${adminToken}` }, + }); + + expect(response.statusCode).toBe(200); + expect(response.json().counters24h.timestampSkew).toBe(7); + }); + + it('reports the span and metric timestamp skew counters', async () => { + await db + .insertInto('metering_events') + .values([ + { + organization_id: testOrg.id, + project_id: testProject.id, + type: 'ingestion.span_timestamp_skew', + quantity: 5, + metadata: { maxPastMs: 88000000, maxFutureMs: 0 }, + }, + { + organization_id: testOrg.id, + project_id: testProject.id, + type: 'ingestion.metric_timestamp_skew', + quantity: 2, + metadata: { maxPastMs: 0, maxFutureMs: 500000 }, + }, + ]) + .execute(); + + const response = await app.inject({ + method: 'GET', + url: '/api/v1/admin/stats/ingestion-health', + headers: { authorization: `Bearer ${adminToken}` }, + }); + + expect(response.statusCode).toBe(200); + const { counters24h } = response.json(); + expect(counters24h.spanTimestampSkew).toBe(5); + expect(counters24h.metricTimestampSkew).toBe(2); + }); }); describe('PATCH /api/v1/admin/users/:id/role', () => { diff --git a/packages/backend/src/tests/modules/audit-log/record-integration.test.ts b/packages/backend/src/tests/modules/audit-log/record-integration.test.ts index 1173f326..5c3bd0e7 100644 --- a/packages/backend/src/tests/modules/audit-log/record-integration.test.ts +++ b/packages/backend/src/tests/modules/audit-log/record-integration.test.ts @@ -5,6 +5,7 @@ import { build } from '../../../server.js'; import { truncateAllTables } from '../../helpers/index.js'; import { createTestUser, createTestOrganization, createTestProject } from '../../helpers/factories.js'; import { createTestSession } from '../../helpers/auth.js'; +import { providerRegistry } from '../../../modules/auth/providers/registry.js'; import { db } from '../../../database/index.js'; describe('auth audit records', () => { @@ -71,6 +72,53 @@ describe('auth audit records', () => { expect(row.actor_id).toBeNull(); expect(row.user_email).toBe('fail-audit@example.com'); expect((row.metadata as any)?.method).toBe('local'); + expect((row.metadata as any)?.reason).toBe('invalid_credentials'); + }); + + it('disabled-account login with correct password produces auth.login_failed row', async () => { + await createTestUser({ email: 'disabled-audit@example.com', password: 'password123', disabled: true }); + + await request(app.server) + .post('/api/v1/auth/login') + .send({ email: 'disabled-audit@example.com', password: 'password123' }) + .expect(401); + + const rows = await db + .selectFrom('audit_log') + .selectAll() + .where('action', '=', 'auth.login_failed') + .execute(); + + expect(rows).toHaveLength(1); + const row = rows[0]; + expect(row.outcome).toBe('failure'); + expect(row.actor_id).toBeNull(); + expect(row.user_email).toBe('disabled-audit@example.com'); + expect((row.metadata as any)?.method).toBe('local'); + expect((row.metadata as any)?.reason).toBe('account_disabled'); + }); + + it('local login against an SSO-only account produces auth.login_failed row', async () => { + await createTestUser({ email: 'sso-audit@example.com', password: null }); + + await request(app.server) + .post('/api/v1/auth/login') + .send({ email: 'sso-audit@example.com', password: 'anypassword' }) + .expect(401); + + const rows = await db + .selectFrom('audit_log') + .selectAll() + .where('action', '=', 'auth.login_failed') + .execute(); + + expect(rows).toHaveLength(1); + const row = rows[0]; + expect(row.outcome).toBe('failure'); + expect(row.actor_id).toBeNull(); + expect(row.user_email).toBe('sso-audit@example.com'); + expect((row.metadata as any)?.method).toBe('local'); + expect((row.metadata as any)?.reason).toBe('sso_required'); }); it('zod-invalid login payload produces NO auth.login_failed row', async () => { @@ -88,6 +136,34 @@ describe('auth audit records', () => { expect(rows).toHaveLength(0); }); + + it('login with the local provider unavailable produces 503 and a provider_unavailable row', async () => { + await createTestUser({ email: 'provider-down@example.com', password: 'password123' }); + + // Simulate the local provider being disabled/removed + const getProviderSpy = vi.spyOn(providerRegistry, 'getProvider').mockResolvedValueOnce(null); + + await request(app.server) + .post('/api/v1/auth/login') + .send({ email: 'provider-down@example.com', password: 'password123' }) + .expect(503); + + const rows = await db + .selectFrom('audit_log') + .selectAll() + .where('action', '=', 'auth.login_failed') + .execute(); + + expect(rows).toHaveLength(1); + const row = rows[0]; + expect(row.outcome).toBe('failure'); + expect(row.actor_id).toBeNull(); + expect(row.user_email).toBe('provider-down@example.com'); + expect((row.metadata as any)?.method).toBe('local'); + expect((row.metadata as any)?.reason).toBe('provider_unavailable'); + + getProviderSpy.mockRestore(); + }); }); vi.mock('../../../config/index.js', async (importOriginal) => { diff --git a/packages/backend/src/tests/modules/auth/authentication-service.test.ts b/packages/backend/src/tests/modules/auth/authentication-service.test.ts index 20d4cdf4..ca904563 100644 --- a/packages/backend/src/tests/modules/auth/authentication-service.test.ts +++ b/packages/backend/src/tests/modules/auth/authentication-service.test.ts @@ -5,6 +5,7 @@ import { AuthenticationService } from '../../../modules/auth/authentication-serv import { CacheManager } from '../../../utils/cache.js'; import { settingsService } from '../../../modules/settings/service.js'; import * as providerRegistryModule from '../../../modules/auth/providers/registry.js'; +import { AuthErrorCode } from '../../../modules/auth/providers/types.js'; import crypto from 'crypto'; describe('AuthenticationService', () => { @@ -28,7 +29,10 @@ describe('AuthenticationService', () => { await db.deleteFrom('projects').execute(); await db.deleteFrom('organizations').execute(); await db.deleteFrom('users').execute(); - await db.deleteFrom('auth_providers').execute(); + // Keep the migration-seeded 'local' provider: POST /auth/login and + // /auth/register resolve it through the provider registry, so wiping + // it breaks every login-based test file that runs after this one. + await db.deleteFrom('auth_providers').where('slug', '!=', 'local').execute(); }); afterAll(async () => { @@ -41,22 +45,30 @@ describe('AuthenticationService', () => { await db.deleteFrom('projects').execute(); await db.deleteFrom('organizations').execute(); await db.deleteFrom('users').execute(); - await db.deleteFrom('auth_providers').execute(); + await db.deleteFrom('auth_providers').where('slug', '!=', 'local').execute(); }); describe('authenticateWithProvider', () => { - it('should throw error when provider not found', async () => { + it('should throw AuthError with PROVIDER_UNAVAILABLE when provider not found', async () => { await expect( authService.authenticateWithProvider('nonexistent', { email: 'test@test.com', password: 'test' }) ).rejects.toThrow("Authentication provider 'nonexistent' not found or disabled"); + + await expect( + authService.authenticateWithProvider('nonexistent', { email: 'test@test.com', password: 'test' }) + ).rejects.toMatchObject({ name: 'AuthError', code: AuthErrorCode.PROVIDER_UNAVAILABLE }); }); - it('should throw error when authentication fails', async () => { + it('should throw AuthError carrying the provider errorCode when authentication fails', async () => { const mockProviderId = crypto.randomUUID(); // Create a mock provider const mockProvider = { config: { id: mockProviderId, type: 'local', name: 'Local', slug: 'local', enabled: true, config: {} }, - authenticate: vi.fn().mockResolvedValue({ success: false, error: 'Invalid credentials' }), + authenticate: vi.fn().mockResolvedValue({ + success: false, + error: 'Invalid credentials', + errorCode: AuthErrorCode.INVALID_CREDENTIALS, + }), supportsRedirect: () => false, }; @@ -65,7 +77,7 @@ describe('AuthenticationService', () => { await expect( authService.authenticateWithProvider('local', { email: 'test@test.com', password: 'wrong' }) - ).rejects.toThrow('Invalid credentials'); + ).rejects.toMatchObject({ name: 'AuthError', code: AuthErrorCode.INVALID_CREDENTIALS }); getProviderSpy.mockRestore(); }); @@ -520,18 +532,19 @@ describe('AuthenticationService', () => { it('should throw error when identity not found', async () => { const user = await createTestUser(); + const providerAId = crypto.randomUUID(); + const providerBId = crypto.randomUUID(); // Create two identities so we can try to unlink + // (slugs must not collide with the migration-seeded 'local' provider) await db.insertInto('auth_providers').values([ - { id: crypto.randomUUID(), type: 'local', name: 'Local', slug: 'local', enabled: true, config: {} }, - { id: crypto.randomUUID(), type: 'oidc', name: 'OIDC', slug: 'oidc', enabled: true, config: {} }, + { id: providerAId, type: 'local', name: 'Local', slug: 'local-notfound', enabled: true, config: {} }, + { id: providerBId, type: 'oidc', name: 'OIDC', slug: 'oidc-notfound', enabled: true, config: {} }, ]).execute(); - const providers = await db.selectFrom('auth_providers').selectAll().execute(); - await db.insertInto('user_identities').values([ - { user_id: user.id, provider_id: providers[0].id, provider_user_id: user.id, metadata: {} }, - { user_id: user.id, provider_id: providers[1].id, provider_user_id: 'ext-id', metadata: {} }, + { user_id: user.id, provider_id: providerAId, provider_user_id: user.id, metadata: {} }, + { user_id: user.id, provider_id: providerBId, provider_user_id: 'ext-id', metadata: {} }, ]).execute(); // Use a valid UUID format that doesn't exist diff --git a/packages/backend/src/tests/modules/auth/external-routes.test.ts b/packages/backend/src/tests/modules/auth/external-routes.test.ts index 931eb4f1..ee697199 100644 --- a/packages/backend/src/tests/modules/auth/external-routes.test.ts +++ b/packages/backend/src/tests/modules/auth/external-routes.test.ts @@ -58,7 +58,10 @@ describe('External Auth Routes', () => { await db.deleteFrom('projects').execute(); await db.deleteFrom('organizations').execute(); await db.deleteFrom('users').execute(); - await db.deleteFrom('auth_providers').execute(); + // Keep the migration-seeded 'local' provider: POST /auth/login and + // /auth/register resolve it through the provider registry, so wiping + // it breaks every login-based test file that runs after this one. + await db.deleteFrom('auth_providers').where('slug', '!=', 'local').execute(); }); afterAll(async () => { diff --git a/packages/backend/src/tests/modules/auth/local-provider.test.ts b/packages/backend/src/tests/modules/auth/local-provider.test.ts index ad2d99c9..010fc73a 100644 --- a/packages/backend/src/tests/modules/auth/local-provider.test.ts +++ b/packages/backend/src/tests/modules/auth/local-provider.test.ts @@ -2,7 +2,6 @@ import { describe, it, expect, beforeEach, afterAll } from 'vitest'; import bcrypt from 'bcrypt'; import { db } from '../../../database/index.js'; import { LocalProvider } from '../../../modules/auth/providers/local-provider.js'; -import { createTestUser } from '../../helpers/factories.js'; import { AuthErrorCode } from '../../../modules/auth/providers/types.js'; describe('LocalProvider', () => { @@ -108,10 +107,10 @@ describe('LocalProvider', () => { expect(result.success).toBe(false); expect(result.error).toBe('Please log in using your organization SSO'); - expect(result.errorCode).toBe(AuthErrorCode.INVALID_CREDENTIALS); + expect(result.errorCode).toBe(AuthErrorCode.SSO_REQUIRED); }); - it('should return error when user is disabled', async () => { + it('should return error when user is disabled (correct password)', async () => { const passwordHash = await bcrypt.hash('password123', 10); await db.insertInto('users').values({ email: 'disabled@example.com', @@ -131,6 +130,26 @@ describe('LocalProvider', () => { expect(result.errorCode).toBe(AuthErrorCode.USER_DISABLED); }); + it('should return generic error when user is disabled and password is wrong (no enumeration)', async () => { + const passwordHash = await bcrypt.hash('password123', 10); + await db.insertInto('users').values({ + email: 'disabled-wrong-pw@example.com', + name: 'Disabled User', + password_hash: passwordHash, + is_admin: false, + disabled: true, + }).execute(); + + const result = await localProvider.authenticate({ + email: 'disabled-wrong-pw@example.com', + password: 'wrongpassword', + }); + + expect(result.success).toBe(false); + expect(result.error).toBe('Invalid email or password'); + expect(result.errorCode).toBe(AuthErrorCode.INVALID_CREDENTIALS); + }); + it('should return error when password is incorrect', async () => { const passwordHash = await bcrypt.hash('correctpassword', 10); await db.insertInto('users').values({ @@ -171,6 +190,7 @@ describe('LocalProvider', () => { expect(result.success).toBe(true); expect(result.providerUserId).toBe('valid@example.com'); expect(result.email).toBe('valid@example.com'); + expect(result.emailVerified).toBe(true); expect(result.name).toBe('Valid User'); expect(result.metadata?.userId).toBe(user.id); }); @@ -179,13 +199,13 @@ describe('LocalProvider', () => { const password = 'password123'; const passwordHash = await bcrypt.hash(password, 10); - const user = await db.insertInto('users').values({ + await db.insertInto('users').values({ email: 'uppercase@example.com', name: 'Test User', password_hash: passwordHash, is_admin: false, disabled: false, - }).returningAll().executeTakeFirstOrThrow(); + }).execute(); const result = await localProvider.authenticate({ email: 'UPPERCASE@EXAMPLE.COM', diff --git a/packages/backend/src/tests/modules/capabilities/digest-recipients-limit.test.ts b/packages/backend/src/tests/modules/capabilities/digest-recipients-limit.test.ts new file mode 100644 index 00000000..de8ee9ab --- /dev/null +++ b/packages/backend/src/tests/modules/capabilities/digest-recipients-limit.test.ts @@ -0,0 +1,154 @@ +import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest'; +import Fastify, { FastifyInstance } from 'fastify'; +import crypto from 'crypto'; +import { db } from '../../../database/index.js'; +import { digestsRoutes } from '../../../modules/digests/routes.js'; +import { contextPlugin } from '../../../context/index.js'; +import { capabilities } from '../../../capabilities/index.js'; +import { createTestContext } from '../../helpers/factories.js'; + +async function createTestSession(userId: string) { + const token = crypto.randomBytes(32).toString('hex'); + await db + .insertInto('sessions') + .values({ user_id: userId, token, expires_at: new Date(Date.now() + 24 * 60 * 60 * 1000) }) + .execute(); + return token; +} + +async function insertConfig(organizationId: string) { + return db + .insertInto('digest_configs') + .values({ organization_id: organizationId, frequency: 'daily', delivery_hour: 8 }) + .returningAll() + .executeTakeFirstOrThrow(); +} + +async function insertRecipient(organizationId: string, configId: string, email: string) { + return db + .insertInto('digest_recipients') + .values({ + organization_id: organizationId, + digest_config_id: configId, + email, + unsubscribe_token: crypto.randomBytes(32).toString('base64url'), + }) + .returningAll() + .executeTakeFirstOrThrow(); +} + +describe('digests.max_recipients enforcement', () => { + let app: FastifyInstance; + let orgId: string; + let userId: string; + let token: string; + let configId: string; + + beforeAll(async () => { + app = Fastify(); + await app.register(contextPlugin); + await app.register(digestsRoutes, { prefix: '/api/v1/digests' }); + await app.ready(); + }); + + afterAll(async () => { + await app.close(); + }); + + beforeEach(async () => { + await db.deleteFrom('organization_entitlements').execute(); + await db.deleteFrom('digest_recipients').execute(); + await db.deleteFrom('digest_configs').execute(); + await db.deleteFrom('sessions').execute(); + await db.deleteFrom('organization_members').execute(); + await db.deleteFrom('projects').execute(); + await db.deleteFrom('organizations').execute(); + await db.deleteFrom('users').execute(); + + const ctx = await createTestContext(); + orgId = ctx.organization.id; + userId = ctx.user.id; + token = await createTestSession(userId); + const config = await insertConfig(orgId); + configId = config.id; + capabilities.invalidate(orgId); + }); + + // Case 1: at limit => blocked + it('blocks adding a recipient when the org limit is reached', async () => { + await db + .insertInto('organization_entitlements') + .values({ organization_id: orgId, capability: 'digests.max_recipients', enabled: null, limit_value: 1 }) + .execute(); + capabilities.invalidate(orgId); + + await insertRecipient(orgId, configId, 'first@example.com'); + + const res = await app.inject({ + method: 'POST', + url: `/api/v1/digests/recipients?organizationId=${orgId}`, + headers: { Authorization: `Bearer ${token}` }, + payload: { email: 'second@example.com' }, + }); + + expect(res.statusCode).toBe(403); + const body = JSON.parse(res.body); + expect(body.code).toBe('capability.digests.max_recipients.limit_reached'); + }); + + // Case 2: under limit => passes + it('allows adding a recipient when under the limit', async () => { + await db + .insertInto('organization_entitlements') + .values({ organization_id: orgId, capability: 'digests.max_recipients', enabled: null, limit_value: 2 }) + .execute(); + capabilities.invalidate(orgId); + + await insertRecipient(orgId, configId, 'first@example.com'); + + const res = await app.inject({ + method: 'POST', + url: `/api/v1/digests/recipients?organizationId=${orgId}`, + headers: { Authorization: `Bearer ${token}` }, + payload: { email: 'second@example.com' }, + }); + + expect(res.statusCode).toBe(201); + }); + + // Case 3: unlimited default (no entitlement row) => passes + it('allows adding a recipient when no limit is configured (unlimited default)', async () => { + const res = await app.inject({ + method: 'POST', + url: `/api/v1/digests/recipients?organizationId=${orgId}`, + headers: { Authorization: `Bearer ${token}` }, + payload: { email: 'anyone@example.com' }, + }); + + expect(res.statusCode).toBe(201); + }); + + // Case 4: org isolation => org A at limit does not block org B + it('does not block org B when org A is at the limit', async () => { + await db + .insertInto('organization_entitlements') + .values({ organization_id: orgId, capability: 'digests.max_recipients', enabled: null, limit_value: 1 }) + .execute(); + capabilities.invalidate(orgId); + await insertRecipient(orgId, configId, 'first@example.com'); + + const ctxB = await createTestContext(); + const tokenB = await createTestSession(ctxB.user.id); + await insertConfig(ctxB.organization.id); + capabilities.invalidate(ctxB.organization.id); + + const res = await app.inject({ + method: 'POST', + url: `/api/v1/digests/recipients?organizationId=${ctxB.organization.id}`, + headers: { Authorization: `Bearer ${tokenB}` }, + payload: { email: 'orgb@example.com' }, + }); + + expect(res.statusCode).toBe(201); + }); +}); diff --git a/packages/backend/src/tests/modules/dashboard/dashboard-service.test.ts b/packages/backend/src/tests/modules/dashboard/dashboard-service.test.ts index 8ae9b969..cbe829f4 100644 --- a/packages/backend/src/tests/modules/dashboard/dashboard-service.test.ts +++ b/packages/backend/src/tests/modules/dashboard/dashboard-service.test.ts @@ -76,6 +76,31 @@ describe('DashboardService', () => { expect(stats.errorRate.value).toBe(20); // 2/10 = 20% }); + it('counts logs from earlier today, not just the last hour', async () => { + // Low-volume orgs take the raw-count path, which must count the whole + // day and not collapse to the last hour the way a stale continuous + // aggregate does (see RAW_STATS_MAX_RECENT_VOLUME). The logs below sit + // earlier today (older than the last hour) yet must all be counted. + const { organization, project } = await createTestContext(); + const now = new Date(); + const todayStart = new Date(now.getFullYear(), now.getMonth(), now.getDate()); + const earlierToday = new Date( + Math.max(todayStart.getTime() + 60_000, now.getTime() - 3 * 60 * 60 * 1000) + ); + + for (let i = 0; i < 6; i++) { + await createTestLog({ projectId: project.id, service: 'api', level: 'info', time: earlierToday }); + } + for (let i = 0; i < 2; i++) { + await createTestLog({ projectId: project.id, service: 'api', level: 'error', time: earlierToday }); + } + + const stats = await dashboardService.getStats(organization.id); + + expect(stats.totalLogsToday.value).toBe(8); + expect(stats.errorRate.value).toBeCloseTo(25, 1); // 2 errors / 8 logs + }); + it('should count distinct active services', async () => { const { organization, project } = await createTestContext(); diff --git a/packages/backend/src/tests/modules/digests/generator-integration.test.ts b/packages/backend/src/tests/modules/digests/generator-integration.test.ts index 3e0e3b40..0e5af4cd 100644 --- a/packages/backend/src/tests/modules/digests/generator-integration.test.ts +++ b/packages/backend/src/tests/modules/digests/generator-integration.test.ts @@ -132,7 +132,178 @@ describe('DigestGeneratorService Integration', () => { expect(trendMatch, 'Email should contain "Trend" metric').toBeTruthy(); expect(trendMatch![1]).toBe(`+${expectedDelta}`); - + expect(trendMatch![2]).toBe(`+${expectedPercentChange}`); }); + + it('should include error groups, security and uptime sections in the digest', async () => { + const user = await db + .insertInto('users') + .values({ + email: 'sections@example.com', + password_hash: 'test_hash', + name: 'Sections User', + }) + .returning('id') + .executeTakeFirstOrThrow(); + + const org = await db + .insertInto('organizations') + .values({ + name: 'Sections Org', + slug: 'sections-org', + owner_id: user.id, + }) + .returningAll() + .executeTakeFirstOrThrow(); + + const project = await db + .insertInto('projects') + .values({ + organization_id: org.id, + name: 'Sections Project', + slug: 'sections-project', + user_id: user.id, + }) + .returning('id') + .executeTakeFirstOrThrow(); + + const config = await db + .insertInto('digest_configs') + .values({ + organization_id: org.id, + frequency: 'daily', + delivery_hour: 8, + }) + .returning('id') + .executeTakeFirstOrThrow(); + + await db + .insertInto('digest_recipients') + .values({ + organization_id: org.id, + digest_config_id: config.id, + email: 'sections-digest@example.com', + unsubscribe_token: 'sections_token', + }) + .execute(); + + const now = new Date(); + const twoHoursAgo = new Date(now.getTime() - 2 * 60 * 60 * 1000); + + // New error group first seen inside the period + await db + .insertInto('error_groups') + .values({ + organization_id: org.id, + project_id: project.id, + fingerprint: 'fp_sections_1', + exception_type: 'TypeError', + exception_message: 'Cannot read properties of undefined', + language: 'nodejs', + occurrence_count: 42, + first_seen: twoHoursAgo, + last_seen: now, + }) + .execute(); + + // Sigma rule + detections inside the period + const rule = await db + .insertInto('sigma_rules') + .values({ + organization_id: org.id, + title: 'Suspicious Login Burst', + level: 'high', + status: 'stable', + logsource: JSON.stringify({ product: 'test' }), + detection: JSON.stringify({ condition: 'selection' }), + }) + .returning('id') + .executeTakeFirstOrThrow(); + + await db + .insertInto('detection_events') + .values([1, 2, 3].map(() => ({ + time: twoHoursAgo, + organization_id: org.id, + project_id: project.id, + sigma_rule_id: rule.id, + log_id: crypto.randomUUID(), + severity: 'high', + rule_title: 'Suspicious Login Burst', + service: 'auth-service', + log_level: 'warn', + log_message: 'failed login', + }))) + .execute(); + + // Open incident + await db + .insertInto('incidents') + .values({ + organization_id: org.id, + project_id: project.id, + title: 'Brute force attempt', + severity: 'high', + status: 'open', + }) + .execute(); + + // Monitor with mixed up/down results in the period + const monitor = await db + .insertInto('monitors') + .values({ + organization_id: org.id, + project_id: project.id, + name: 'API Health', + type: 'http', + target: 'https://example.com/health', + }) + .returning('id') + .executeTakeFirstOrThrow(); + + const results = []; + for (let i = 0; i < 10; i++) { + results.push({ + time: new Date(twoHoursAgo.getTime() + i * 60 * 1000), + monitor_id: monitor.id, + organization_id: org.id, + project_id: project.id, + status: i === 0 ? ('down' as const) : ('up' as const), + }); + } + await db.insertInto('monitor_results').values(results).execute(); + + await generator.generateAndSendDigest({ + organizationId: org.id, + digestConfigId: config.id, + frequency: 'daily', + }); + + expect(mockSendMail).toHaveBeenCalledTimes(1); + const emailCall = mockSendMail.mock.calls[0][0]; + const text = emailCall.text as string; + const html = emailCall.html as string; + + // New error group section + expect(text).toContain('TypeError'); + expect(text).toContain('Cannot read properties of undefined'); + expect(text).toContain('42'); + + // Security section (3 detections, top rule, 1 open incident) + expect(text).toContain('Detections: 3'); + expect(text).toContain('Suspicious Login Burst'); + expect(text).toContain('Open incidents: 1'); + + // Uptime section (9 up / 10 checks = 90%) + expect(text).toContain('UPTIME'); + expect(text).toContain('API Health'); + expect(text).toContain('90%'); + + // HTML version carries the same content and the unsubscribe link + expect(html).toContain(''); + expect(html).toContain('TypeError'); + expect(html).toContain('Suspicious Login Burst'); + expect(html).toContain('unsubscribe?token=sections_token'); + }); }); diff --git a/packages/backend/src/tests/modules/digests/generator.test.ts b/packages/backend/src/tests/modules/digests/generator.test.ts index f11a94ba..b0a7619a 100644 --- a/packages/backend/src/tests/modules/digests/generator.test.ts +++ b/packages/backend/src/tests/modules/digests/generator.test.ts @@ -5,6 +5,7 @@ import type { DigestJobPayload } from '../../../modules/digests/scheduler.js'; vi.mock('../../../database/reservoir.js', () => ({ reservoir: { count: vi.fn(), + topValues: vi.fn(), }, })); @@ -19,6 +20,9 @@ vi.mock('../../../database/connection.js', () => ({ selectFrom: vi.fn().mockReturnThis(), select: vi.fn().mockReturnThis(), where: vi.fn().mockReturnThis(), + groupBy: vi.fn().mockReturnThis(), + orderBy: vi.fn().mockReturnThis(), + limit: vi.fn().mockReturnThis(), execute: vi.fn(), executeTakeFirst: vi.fn(), } @@ -60,9 +64,11 @@ describe('DigestGeneratorService', () => { const { db } = await import('../../../database/connection.js'); mockDb = db; - mockDb.execute.mockReset(); - mockDb.executeTakeFirst.mockReset(); - mockReservoir.count.mockReset(); + // Defaults keep the section queries harmless; tests override with *Once. + mockDb.execute.mockReset().mockResolvedValue([]); + mockDb.executeTakeFirst.mockReset().mockResolvedValue({ count: 0 }); + mockReservoir.count.mockReset().mockResolvedValue({ count: 0 }); + mockReservoir.topValues.mockReset().mockResolvedValue({ values: [] }); mockSendMail.mockReset().mockResolvedValue({ messageId: 'test-message-id' }); generator = new DigestGeneratorService(); @@ -73,6 +79,11 @@ describe('DigestGeneratorService', () => { }); describe('generateAndSendDigest', () => { + /** + * Call order with the shared fluent mock: + * executeTakeFirst: org lookup, then per-section counts (default {count: 0}) + * execute: recipients, projects, then per-section lists (default []) + */ it('should generate and send daily digest with log volume', async () => { const payload: DigestJobPayload = { organizationId: 'org_1', @@ -80,7 +91,6 @@ describe('DigestGeneratorService', () => { frequency: 'daily', }; - // organization lookup mockDb.executeTakeFirst.mockResolvedValueOnce({ name: 'Test Organization', }); @@ -97,25 +107,16 @@ describe('DigestGeneratorService', () => { }, ]); - // Mock projects query + // projects query mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); - const { reservoir } = await import('../../../database/reservoir.js'); - - (reservoir.count as any).mockResolvedValueOnce({ - count: 15000, - }); - - (reservoir.count as any).mockResolvedValueOnce({ - count: 12000, - }); + mockReservoir.count.mockResolvedValueOnce({ count: 15000 }); + mockReservoir.count.mockResolvedValueOnce({ count: 12000 }); await generator.generateAndSendDigest(payload); - // Verify emails were sent expect(mockSendMail).toHaveBeenCalledTimes(2); - // Check first email const firstCall = mockSendMail.mock.calls[0][0]; expect(firstCall.to).toBe('user1@test.com'); expect(firstCall.subject).toContain('Daily Report'); @@ -124,7 +125,6 @@ describe('DigestGeneratorService', () => { expect(firstCall.text).toContain('+3000 (+25.0%)'); expect(firstCall.text).toContain('token_1'); - // Check second email const secondCall = mockSendMail.mock.calls[1][0]; expect(secondCall.to).toBe('user2@test.com'); expect(secondCall.text).toContain('token_2'); @@ -148,18 +148,10 @@ describe('DigestGeneratorService', () => { }, ]); - // Mock projects query mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); - const { reservoir } = await import('../../../database/reservoir.js'); - - (reservoir.count as any).mockResolvedValueOnce({ - count: 100000, - }); - - (reservoir.count as any).mockResolvedValueOnce({ - count: 95000, - }); + mockReservoir.count.mockResolvedValueOnce({ count: 100000 }); + mockReservoir.count.mockResolvedValueOnce({ count: 95000 }); await generator.generateAndSendDigest(payload); @@ -188,13 +180,8 @@ describe('DigestGeneratorService', () => { }, ]); - // Mock projects query mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); - const { reservoir } = await import('../../../database/reservoir.js'); - (reservoir.count as any).mockResolvedValueOnce({ count: 0 }); - (reservoir.count as any).mockResolvedValueOnce({ count: 0 }); - await generator.generateAndSendDigest(payload); expect(mockSendMail).toHaveBeenCalledTimes(1); @@ -221,12 +208,10 @@ describe('DigestGeneratorService', () => { }, ]); - // Mock projects query mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); - const { reservoir } = await import('../../../database/reservoir.js'); - (reservoir.count as any).mockResolvedValueOnce({ count: 8000 }); - (reservoir.count as any).mockResolvedValueOnce({ count: 10000 }); + mockReservoir.count.mockResolvedValueOnce({ count: 8000 }); + mockReservoir.count.mockResolvedValueOnce({ count: 10000 }); await generator.generateAndSendDigest(payload); @@ -254,12 +239,10 @@ describe('DigestGeneratorService', () => { }, ]); - // Mock projects query mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); - const { reservoir } = await import('../../../database/reservoir.js'); - (reservoir.count as any).mockResolvedValueOnce({ count: 5000 }); - (reservoir.count as any).mockResolvedValueOnce({ count: 0 }); + mockReservoir.count.mockResolvedValueOnce({ count: 5000 }); + mockReservoir.count.mockResolvedValueOnce({ count: 0 }); await generator.generateAndSendDigest(payload); @@ -285,7 +268,6 @@ describe('DigestGeneratorService', () => { await generator.generateAndSendDigest(payload); - // Should not send any emails expect(mockSendMail).not.toHaveBeenCalled(); }); @@ -313,6 +295,9 @@ describe('DigestGeneratorService', () => { expect(mockSendMail).toHaveBeenCalledTimes(1); const emailCall = mockSendMail.mock.calls[0][0]; expect(emailCall.text).toContain('No activity during this period'); + // No projects: the reservoir must never be queried + expect(mockReservoir.count).not.toHaveBeenCalled(); + expect(mockReservoir.topValues).not.toHaveBeenCalled(); }); it('should throw error if organization not found', async () => { @@ -322,7 +307,6 @@ describe('DigestGeneratorService', () => { frequency: 'daily', }; - // Organization not found mockDb.executeTakeFirst.mockResolvedValueOnce(undefined); await expect(generator.generateAndSendDigest(payload)).rejects.toThrow( @@ -351,9 +335,8 @@ describe('DigestGeneratorService', () => { ]); mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); - const { reservoir } = await import('../../../database/reservoir.js'); - (reservoir.count as any).mockResolvedValueOnce({ count: 1000 }); - (reservoir.count as any).mockResolvedValueOnce({ count: 900 }); + mockReservoir.count.mockResolvedValueOnce({ count: 1000 }); + mockReservoir.count.mockResolvedValueOnce({ count: 900 }); await generator.generateAndSendDigest(payload); @@ -364,4 +347,151 @@ describe('DigestGeneratorService', () => { expect(emailCall.text).toContain('https://app.logtide.com/unsubscribe?token=secure_token_abc123'); }); }); + + describe('buildReportData', () => { + /** + * Call order inside buildReportData with the shared fluent mock: + * execute: projects, error_groups, top rules, monitors, uptime rows + * executeTakeFirst: detection total, open incidents + * reservoir: count x2, topValues x2 (previous period skipped when current is empty) + */ + it('computes top error services with delta vs previous period', async () => { + mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); // projects + + mockReservoir.topValues.mockResolvedValueOnce({ + values: [ + { value: 'api', count: 50 }, + { value: 'web', count: 10 }, + ], + }); + mockReservoir.topValues.mockResolvedValueOnce({ + values: [{ value: 'api', count: 20 }], + }); + + const report = await generator.buildReportData('org_1', 'Test Org', 'daily'); + + expect(report.topErrorServices).toEqual([ + { service: 'api', errorCount: 50, previousCount: 20, delta: 30 }, + { service: 'web', errorCount: 10, previousCount: 0, delta: 10 }, + ]); + + // error+critical levels, top 5 for current period + const firstCall = mockReservoir.topValues.mock.calls[0][0]; + expect(firstCall.field).toBe('service'); + expect(firstCall.level).toEqual(['error', 'critical']); + expect(firstCall.limit).toBe(5); + }); + + it('skips the previous-period query when there are no current errors', async () => { + mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); + + const report = await generator.buildReportData('org_1', 'Test Org', 'daily'); + + expect(report.topErrorServices).toEqual([]); + expect(mockReservoir.topValues).toHaveBeenCalledTimes(1); + }); + + it('collects new error groups first seen in the period', async () => { + mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); // projects + mockDb.execute.mockResolvedValueOnce([ + { + exception_type: 'TypeError', + exception_message: 'Cannot read properties of undefined', + occurrence_count: 42, + language: 'nodejs', + }, + { + exception_type: 'ValueError', + exception_message: null, + occurrence_count: 3, + language: 'python', + }, + ]); // error_groups + + const report = await generator.buildReportData('org_1', 'Test Org', 'daily'); + + expect(report.newErrorGroups).toEqual([ + { + exceptionType: 'TypeError', + exceptionMessage: 'Cannot read properties of undefined', + occurrenceCount: 42, + language: 'nodejs', + }, + { + exceptionType: 'ValueError', + exceptionMessage: '', + occurrenceCount: 3, + language: 'python', + }, + ]); + }); + + it('computes the security summary', async () => { + mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); // projects + mockDb.execute.mockResolvedValueOnce([]); // error_groups + mockDb.execute.mockResolvedValueOnce([ + { rule_title: 'Suspicious Login', severity: 'high', count: 7 }, + { rule_title: 'Port Scan', severity: 'medium', count: 4 }, + ]); // top rules + + mockDb.executeTakeFirst.mockResolvedValueOnce({ count: 11 }); // detections total + mockDb.executeTakeFirst.mockResolvedValueOnce({ count: 2 }); // open incidents + + const report = await generator.buildReportData('org_1', 'Test Org', 'weekly'); + + expect(report.security).toEqual({ + totalDetections: 11, + topRules: [ + { ruleTitle: 'Suspicious Login', severity: 'high', count: 7 }, + { ruleTitle: 'Port Scan', severity: 'medium', count: 4 }, + ], + openIncidents: 2, + }); + }); + + it('computes uptime summary from monitor daily buckets', async () => { + mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); // projects + mockDb.execute.mockResolvedValueOnce([]); // error_groups + mockDb.execute.mockResolvedValueOnce([]); // top rules + mockDb.execute.mockResolvedValueOnce([ + { id: 'mon_1', name: 'API Health' }, + { id: 'mon_2', name: 'Landing Page' }, + ]); // monitors + mockDb.execute.mockResolvedValueOnce([ + { monitor_id: 'mon_1', successful: 95, total: 100 }, + { monitor_id: 'mon_2', successful: 100, total: 100 }, + ]); // uptime rows + + const report = await generator.buildReportData('org_1', 'Test Org', 'daily'); + + expect(report.uptime).toEqual({ + monitorCount: 2, + overallUptimePct: 97.5, + worstMonitors: [ + { name: 'API Health', uptimePct: 95 }, + { name: 'Landing Page', uptimePct: 100 }, + ], + }); + }); + + it('returns null uptime when the org has no enabled monitors', async () => { + mockDb.execute.mockResolvedValueOnce([{ id: 'project_1' }]); // projects + // error_groups, top rules, monitors all fall through to the [] default + + const report = await generator.buildReportData('org_1', 'Test Org', 'daily'); + + expect(report.uptime).toBeNull(); + }); + + it('labels the period by frequency', async () => { + mockDb.execute.mockResolvedValueOnce([]); // projects (none) + + const daily = await generator.buildReportData('org_1', 'Test Org', 'daily'); + expect(daily.periodLabel).toBe('last 24 hours'); + + mockDb.execute.mockResolvedValueOnce([]); + const weekly = await generator.buildReportData('org_1', 'Test Org', 'weekly'); + expect(weekly.periodLabel).toBe('last 7 days'); + }); + }); }); diff --git a/packages/backend/src/tests/modules/digests/routes.test.ts b/packages/backend/src/tests/modules/digests/routes.test.ts new file mode 100644 index 00000000..7729c74f --- /dev/null +++ b/packages/backend/src/tests/modules/digests/routes.test.ts @@ -0,0 +1,447 @@ +import { describe, it, expect, beforeEach, afterAll, beforeAll } from 'vitest'; +import Fastify, { FastifyInstance } from 'fastify'; +import crypto from 'crypto'; +import { db } from '../../../database/index.js'; +import { digestsRoutes } from '../../../modules/digests/routes.js'; +import { digestsPublicRoutes } from '../../../modules/digests/public-routes.js'; +import { createTestContext, createTestUser } from '../../helpers/factories.js'; + +async function createTestSession(userId: string) { + const token = crypto.randomBytes(32).toString('hex'); + await db + .insertInto('sessions') + .values({ user_id: userId, token, expires_at: new Date(Date.now() + 24 * 60 * 60 * 1000) }) + .execute(); + return token; +} + +describe('Digests Routes', () => { + let app: FastifyInstance; + let authToken: string; + let testUser: any; + let testOrganization: any; + + beforeAll(async () => { + app = Fastify(); + await app.register(digestsRoutes, { prefix: '/api/v1/digests' }); + await app.register(digestsPublicRoutes, { prefix: '/api/v1/digests' }); + await app.ready(); + }); + + afterAll(async () => { + await app.close(); + }); + + beforeEach(async () => { + await db.deleteFrom('digest_recipients').execute(); + await db.deleteFrom('digest_configs').execute(); + await db.deleteFrom('organization_members').execute(); + await db.deleteFrom('projects').execute(); + await db.deleteFrom('organizations').execute(); + await db.deleteFrom('sessions').execute(); + await db.deleteFrom('users').execute(); + + const context = await createTestContext(); + testUser = context.user; + testOrganization = context.organization; + + authToken = await createTestSession(testUser.id); + }); + + async function insertConfig(organizationId: string, overrides: Record = {}) { + return db + .insertInto('digest_configs') + .values({ + organization_id: organizationId, + frequency: 'daily', + delivery_hour: 8, + ...overrides, + }) + .returningAll() + .executeTakeFirstOrThrow(); + } + + async function insertRecipient( + organizationId: string, + configId: string, + email = 'someone@example.com' + ) { + return db + .insertInto('digest_recipients') + .values({ + organization_id: organizationId, + digest_config_id: configId, + email, + unsubscribe_token: crypto.randomBytes(32).toString('base64url'), + }) + .returningAll() + .executeTakeFirstOrThrow(); + } + + describe('GET /api/v1/digests/config', () => { + it('returns null config when the org has none', async () => { + const res = await app.inject({ + method: 'GET', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(res.statusCode).toBe(200); + const body = JSON.parse(res.body); + expect(body.config).toBeNull(); + expect(body.recipients).toEqual([]); + }); + + it('returns the config with its recipients', async () => { + const config = await insertConfig(testOrganization.id); + await insertRecipient(testOrganization.id, config.id); + + const res = await app.inject({ + method: 'GET', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(res.statusCode).toBe(200); + const body = JSON.parse(res.body); + expect(body.config.frequency).toBe('daily'); + expect(body.config.delivery_hour).toBe(8); + expect(body.recipients).toHaveLength(1); + expect(body.recipients[0].email).toBe('someone@example.com'); + // The unsubscribe token must never leak through the management API + expect(body.recipients[0].unsubscribe_token).toBeUndefined(); + }); + + it('rejects non-members', async () => { + const outsider = await createTestUser({ email: 'outsider@example.com' }); + const outsiderToken = await createTestSession(outsider.id); + + const res = await app.inject({ + method: 'GET', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${outsiderToken}` }, + }); + + expect(res.statusCode).toBe(403); + }); + + it('rejects unauthenticated requests', async () => { + const res = await app.inject({ + method: 'GET', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + }); + + expect(res.statusCode).toBe(401); + }); + }); + + describe('PUT /api/v1/digests/config', () => { + it('creates the config on first save', async () => { + const res = await app.inject({ + method: 'PUT', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { frequency: 'daily', deliveryHour: 9, enabled: true }, + }); + + expect(res.statusCode).toBe(200); + const body = JSON.parse(res.body); + expect(body.config.frequency).toBe('daily'); + expect(body.config.delivery_hour).toBe(9); + expect(body.config.enabled).toBe(true); + }); + + it('updates the existing config (upsert)', async () => { + await insertConfig(testOrganization.id, { delivery_hour: 8 }); + + const res = await app.inject({ + method: 'PUT', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { frequency: 'weekly', deliveryHour: 18, deliveryDayOfWeek: 1, enabled: false }, + }); + + expect(res.statusCode).toBe(200); + const body = JSON.parse(res.body); + expect(body.config.frequency).toBe('weekly'); + expect(body.config.delivery_hour).toBe(18); + expect(body.config.delivery_day_of_week).toBe(1); + expect(body.config.enabled).toBe(false); + + const rows = await db.selectFrom('digest_configs').selectAll().execute(); + expect(rows).toHaveLength(1); + }); + + it('rejects weekly frequency without a day of week', async () => { + const res = await app.inject({ + method: 'PUT', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { frequency: 'weekly', deliveryHour: 8, enabled: true }, + }); + + expect(res.statusCode).toBe(400); + }); + + it('rejects an out-of-range delivery hour', async () => { + const res = await app.inject({ + method: 'PUT', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { frequency: 'daily', deliveryHour: 24, enabled: true }, + }); + + expect(res.statusCode).toBe(400); + }); + + it('rejects members without admin role', async () => { + const member = await createTestUser({ email: 'member@example.com' }); + await db + .insertInto('organization_members') + .values({ organization_id: testOrganization.id, user_id: member.id, role: 'member' }) + .execute(); + const memberToken = await createTestSession(member.id); + + const res = await app.inject({ + method: 'PUT', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${memberToken}` }, + payload: { frequency: 'daily', deliveryHour: 8, enabled: true }, + }); + + expect(res.statusCode).toBe(403); + }); + }); + + describe('DELETE /api/v1/digests/config', () => { + it('deletes the config and cascades to recipients', async () => { + const config = await insertConfig(testOrganization.id); + await insertRecipient(testOrganization.id, config.id); + + const res = await app.inject({ + method: 'DELETE', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(res.statusCode).toBe(204); + const recipients = await db.selectFrom('digest_recipients').selectAll().execute(); + expect(recipients).toHaveLength(0); + }); + + it('returns 404 when there is no config', async () => { + const res = await app.inject({ + method: 'DELETE', + url: `/api/v1/digests/config?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(res.statusCode).toBe(404); + }); + }); + + describe('POST /api/v1/digests/recipients', () => { + it('adds a recipient with a generated unsubscribe token', async () => { + await insertConfig(testOrganization.id); + + const res = await app.inject({ + method: 'POST', + url: `/api/v1/digests/recipients?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { email: 'new@example.com' }, + }); + + expect(res.statusCode).toBe(201); + const body = JSON.parse(res.body); + expect(body.recipient.email).toBe('new@example.com'); + expect(body.recipient.subscribed).toBe(true); + expect(body.recipient.unsubscribe_token).toBeUndefined(); + + const row = await db + .selectFrom('digest_recipients') + .selectAll() + .where('email', '=', 'new@example.com') + .executeTakeFirstOrThrow(); + expect(row.unsubscribe_token.length).toBeGreaterThanOrEqual(40); + }); + + it('rejects adding a recipient before the digest is configured', async () => { + const res = await app.inject({ + method: 'POST', + url: `/api/v1/digests/recipients?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { email: 'new@example.com' }, + }); + + expect(res.statusCode).toBe(409); + }); + + it('rejects duplicate emails', async () => { + const config = await insertConfig(testOrganization.id); + await insertRecipient(testOrganization.id, config.id, 'dup@example.com'); + + const res = await app.inject({ + method: 'POST', + url: `/api/v1/digests/recipients?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { email: 'dup@example.com' }, + }); + + expect(res.statusCode).toBe(409); + }); + + it('rejects invalid emails', async () => { + await insertConfig(testOrganization.id); + + const res = await app.inject({ + method: 'POST', + url: `/api/v1/digests/recipients?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { email: 'not-an-email' }, + }); + + expect(res.statusCode).toBe(400); + }); + }); + + describe('DELETE /api/v1/digests/recipients/:id', () => { + it('removes a recipient', async () => { + const config = await insertConfig(testOrganization.id); + const recipient = await insertRecipient(testOrganization.id, config.id); + + const res = await app.inject({ + method: 'DELETE', + url: `/api/v1/digests/recipients/${recipient.id}?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(res.statusCode).toBe(204); + }); + + it('cannot remove a recipient of another organization', async () => { + // org B with its own digest + recipient + const ctxB = await createTestContext(); + const configB = await insertConfig(ctxB.organization.id); + const recipientB = await insertRecipient(ctxB.organization.id, configB.id, 'orgb@example.com'); + + // org A admin tries to delete org B's recipient through their own org scope + const res = await app.inject({ + method: 'DELETE', + url: `/api/v1/digests/recipients/${recipientB.id}?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(res.statusCode).toBe(404); + const stillThere = await db + .selectFrom('digest_recipients') + .selectAll() + .where('id', '=', recipientB.id) + .executeTakeFirst(); + expect(stillThere).toBeDefined(); + }); + }); + + describe('POST /api/v1/digests/recipients/:id/resubscribe', () => { + it('resubscribes and rotates the unsubscribe token', async () => { + const config = await insertConfig(testOrganization.id); + const recipient = await insertRecipient(testOrganization.id, config.id); + await db + .updateTable('digest_recipients') + .set({ subscribed: false }) + .where('id', '=', recipient.id) + .execute(); + + const res = await app.inject({ + method: 'POST', + url: `/api/v1/digests/recipients/${recipient.id}/resubscribe?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(res.statusCode).toBe(200); + const body = JSON.parse(res.body); + expect(body.recipient.subscribed).toBe(true); + + const row = await db + .selectFrom('digest_recipients') + .selectAll() + .where('id', '=', recipient.id) + .executeTakeFirstOrThrow(); + expect(row.subscribed).toBe(true); + expect(row.unsubscribe_token).not.toBe(recipient.unsubscribe_token); + }); + + it('returns 404 for unknown recipients', async () => { + const res = await app.inject({ + method: 'POST', + url: `/api/v1/digests/recipients/${crypto.randomUUID()}/resubscribe?organizationId=${testOrganization.id}`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(res.statusCode).toBe(404); + }); + }); + + describe('POST /api/v1/digests/unsubscribe (public)', () => { + it('unsubscribes by token without authentication', async () => { + const config = await insertConfig(testOrganization.id); + const recipient = await insertRecipient(testOrganization.id, config.id, 'giuseppe@example.com'); + + const res = await app.inject({ + method: 'POST', + url: '/api/v1/digests/unsubscribe', + payload: { token: recipient.unsubscribe_token }, + }); + + expect(res.statusCode).toBe(200); + const body = JSON.parse(res.body); + expect(body.success).toBe(true); + expect(body.email).toBe('g***@example.com'); + + const row = await db + .selectFrom('digest_recipients') + .selectAll() + .where('id', '=', recipient.id) + .executeTakeFirstOrThrow(); + expect(row.subscribed).toBe(false); + }); + + it('is idempotent', async () => { + const config = await insertConfig(testOrganization.id); + const recipient = await insertRecipient(testOrganization.id, config.id); + + const first = await app.inject({ + method: 'POST', + url: '/api/v1/digests/unsubscribe', + payload: { token: recipient.unsubscribe_token }, + }); + const second = await app.inject({ + method: 'POST', + url: '/api/v1/digests/unsubscribe', + payload: { token: recipient.unsubscribe_token }, + }); + + expect(first.statusCode).toBe(200); + expect(second.statusCode).toBe(200); + }); + + it('returns 404 for unknown tokens', async () => { + const res = await app.inject({ + method: 'POST', + url: '/api/v1/digests/unsubscribe', + payload: { token: 'nope' }, + }); + + expect(res.statusCode).toBe(404); + }); + + it('rejects requests without a token', async () => { + const res = await app.inject({ + method: 'POST', + url: '/api/v1/digests/unsubscribe', + payload: {}, + }); + + expect(res.statusCode).toBe(400); + }); + }); +}); diff --git a/packages/backend/src/tests/modules/digests/scheduler.test.ts b/packages/backend/src/tests/modules/digests/scheduler.test.ts index e237dbaf..86891e09 100644 --- a/packages/backend/src/tests/modules/digests/scheduler.test.ts +++ b/packages/backend/src/tests/modules/digests/scheduler.test.ts @@ -1,6 +1,5 @@ import { describe, it, expect, vi, beforeEach } from 'vitest'; -import { DigestScheduler, digestScheduler } from '../../../modules/digests/scheduler.js'; -import { db } from '../../../database/connection.js'; +import { DigestScheduler, DIGEST_DISPATCH_CRON } from '../../../modules/digests/scheduler.js'; import { getCronRegistry } from '../../../queue/queue-factory.js'; vi.mock('@logtide/core', () => ({ @@ -9,23 +8,11 @@ vi.mock('@logtide/core', () => ({ }, })); - -vi.mock('../../../database/connection.js', () => { - return { - db: { - selectFrom: vi.fn().mockReturnThis(), - select: vi.fn().mockReturnThis(), - where: vi.fn().mockReturnThis(), - execute: vi.fn(), - }, - }; -}); - - +const mockRegisterCronJobs = vi.fn().mockResolvedValue(undefined); vi.mock('../../../queue/queue-factory.js', () => { return { getCronRegistry: vi.fn().mockReturnValue({ - registerCronJobs: vi.fn().mockResolvedValue(undefined), + registerCronJobs: (...args: unknown[]) => mockRegisterCronJobs(...args), }), }; }); @@ -35,73 +22,26 @@ describe('DigestScheduler', () => { vi.clearAllMocks(); }); - describe('registerAllDigests', () => { - it('should log and return early if no active configs are found', async () => { - const { hub } = await import('@logtide/core'); - ((db as any).execute as ReturnType).mockResolvedValue([]); - - const scheduler = new DigestScheduler(); - await scheduler.registerAllDigests(); - - expect(hub.captureLog).toHaveBeenCalledWith('info', '[DigestScheduler] No active digest configs found'); - expect(getCronRegistry('digest-generation').registerCronJobs).not.toHaveBeenCalled(); - }); - - it('should correctly register a daily and a weekly cron job', async () => { - const mockConfigs = [ - { - id: 'conf_1', - organization_id: 'org_1', - frequency: 'daily', - delivery_hour: 8, - delivery_day_of_week: null - }, - { - id: 'conf_2', - organization_id: 'org_2', - frequency: 'weekly', - delivery_hour: 14, - delivery_day_of_week: 1 - } - ]; - - ((db as any).execute as ReturnType).mockResolvedValue(mockConfigs); - const { hub } = await import('@logtide/core'); - + describe('registerDispatchCron', () => { + it('registers a single hourly dispatch cron job', async () => { const scheduler = new DigestScheduler(); - await scheduler.registerAllDigests(); - - const expectedItems = [ - { - task: 'digest-generation', - cronExpression: '0 8 * * *', - payload: { - organizationId: 'org_1', - digestConfigId: 'conf_1', - frequency: 'daily' - }, - identifier: 'digest:org_1' - }, - { - task: 'digest-generation', - cronExpression: '0 14 * * 1', - payload: { - organizationId: 'org_2', - digestConfigId: 'conf_2', - frequency: 'weekly' - }, - identifier: 'digest:org_2' - } - ]; - - expect(getCronRegistry('digest-generation').registerCronJobs).toHaveBeenCalledWith(expectedItems); - expect(hub.captureLog).toHaveBeenCalledWith('info', '[DigestScheduler] Registered 2 digest schedule(s)'); + await scheduler.registerDispatchCron(); + + expect(getCronRegistry).toHaveBeenCalledWith('digest-dispatch'); + expect(mockRegisterCronJobs).toHaveBeenCalledTimes(1); + + const items = mockRegisterCronJobs.mock.calls[0][0]; + expect(items).toHaveLength(1); + expect(items[0]).toEqual({ + task: 'digest-dispatch', + cronExpression: DIGEST_DISPATCH_CRON, + payload: {}, + identifier: 'digest-dispatch', + }); }); - }); - describe('export instance', () => { - it('should export a singleton instance', () => { - expect(digestScheduler).toBeInstanceOf(DigestScheduler); + it('uses an hourly cron expression', () => { + expect(DIGEST_DISPATCH_CRON).toBe('0 * * * *'); }); }); }); diff --git a/packages/backend/src/tests/modules/exceptions/detection.test.ts b/packages/backend/src/tests/modules/exceptions/detection.test.ts index 5149af5a..2649e2d0 100644 --- a/packages/backend/src/tests/modules/exceptions/detection.test.ts +++ b/packages/backend/src/tests/modules/exceptions/detection.test.ts @@ -1,5 +1,6 @@ import { describe, it, expect } from 'vitest'; import { ExceptionDetectionService } from '../../../modules/exceptions/detection.js'; +import { FingerprintService } from '../../../modules/exceptions/fingerprint-service.js'; describe('ExceptionDetectionService', () => { describe('detectException', () => { @@ -531,6 +532,52 @@ describe('ExceptionDetectionService', () => { expect(result?.frames[1].isAppCode).toBe(true); expect(result?.frames[2].isAppCode).toBe(true); }); + + it('should detect node: runtime frames as library code', () => { + const metadata = { + exception: { + type: 'Error', + message: 'Test', + stacktrace: [ + { file: 'node:internal/process/task_queues', function: 'process.processTicksAndRejections', line: 95 }, + { file: 'node:events', function: 'EventEmitter.emit', line: 1 }, + ], + }, + }; + + const result = ExceptionDetectionService.detectException('Error', metadata); + + expect(result?.frames[0].isAppCode).toBe(false); + expect(result?.frames[1].isAppCode).toBe(false); + }); + + it('should not split the same error across groups when only node: internal frames differ', () => { + // Regression: async Node stacks include varying internal frames + // (node:internal/...). When those were treated as app code they + // entered the fingerprint, so the same logical error fingerprinted + // differently and split into several error groups. + const appFrame = { file: '/app/apps/api/dist/modules/products/routes.js', function: 'Object.', line: 130, column: 14 }; + + const occurrenceA = ExceptionDetectionService.detectException('TypeError', { + exception: { + type: 'TypeError', + message: "Cannot read properties of undefined (reading 'selectFrom')", + stacktrace: [appFrame, { file: 'node:internal/process/task_queues', function: 'process.processTicksAndRejections', line: 95 }], + }, + }); + + const occurrenceB = ExceptionDetectionService.detectException('TypeError', { + exception: { + type: 'TypeError', + message: "Cannot read properties of undefined (reading 'selectFrom')", + stacktrace: [appFrame, { file: 'node:internal/async_hooks', function: 'AsyncResource.runInAsyncScope', line: 206 }], + }, + }); + + expect(occurrenceA).not.toBeNull(); + expect(occurrenceB).not.toBeNull(); + expect(FingerprintService.generate(occurrenceA!)).toBe(FingerprintService.generate(occurrenceB!)); + }); }); describe('raw trace reconstruction', () => { diff --git a/packages/backend/src/tests/modules/exceptions/routes.test.ts b/packages/backend/src/tests/modules/exceptions/routes.test.ts index c46ecaaf..f24ddb2e 100644 --- a/packages/backend/src/tests/modules/exceptions/routes.test.ts +++ b/packages/backend/src/tests/modules/exceptions/routes.test.ts @@ -842,4 +842,133 @@ describe('Exceptions Routes', () => { expect(response.statusCode).toBe(200); }); }); + + describe('error-group duplicates and merge', () => { + it('GET /:id/duplicates returns same type+message groups', async () => { + const target = await createTestErrorGroup({ + organizationId: testOrganization.id, + projectId: testProject.id, + fingerprint: `t-${Date.now()}`, + exceptionType: 'TypeError', + exceptionMessage: 'boom', + }); + const dup = await createTestErrorGroup({ + organizationId: testOrganization.id, + projectId: testProject.id, + fingerprint: `d-${Date.now()}`, + exceptionType: 'TypeError', + exceptionMessage: 'boom', + }); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/error-groups/${target.id}/duplicates`, + headers: { Authorization: `Bearer ${authToken}` }, + query: { organizationId: testOrganization.id }, + }); + + expect(response.statusCode).toBe(200); + expect(response.json().duplicates.map((d: any) => d.id)).toContain(dup.id); + }); + + it('GET /:id/duplicates returns 403 for a non-member organization', async () => { + const other = await createTestContext(); + const group = await createTestErrorGroup({ + organizationId: other.organization.id, + projectId: other.project.id, + }); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/error-groups/${group.id}/duplicates`, + headers: { Authorization: `Bearer ${authToken}` }, + query: { organizationId: other.organization.id }, + }); + + expect(response.statusCode).toBe(403); + }); + + it('GET /:id/duplicates returns 404 for a missing group', async () => { + const response = await app.inject({ + method: 'GET', + url: `/api/v1/error-groups/00000000-0000-0000-0000-000000000000/duplicates`, + headers: { Authorization: `Bearer ${authToken}` }, + query: { organizationId: testOrganization.id }, + }); + + expect(response.statusCode).toBe(404); + }); + + it('POST /:id/merge folds source groups into the target', async () => { + const target = await createTestErrorGroup({ + organizationId: testOrganization.id, + projectId: testProject.id, + fingerprint: `mt-${Date.now()}`, + exceptionType: 'TypeError', + exceptionMessage: 'boom', + occurrenceCount: 1, + }); + const source = await createTestErrorGroup({ + organizationId: testOrganization.id, + projectId: testProject.id, + fingerprint: `ms-${Date.now()}`, + exceptionType: 'TypeError', + exceptionMessage: 'boom', + occurrenceCount: 3, + }); + + const response = await app.inject({ + method: 'POST', + url: `/api/v1/error-groups/${target.id}/merge`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { organizationId: testOrganization.id, sourceIds: [source.id] }, + }); + + expect(response.statusCode).toBe(200); + expect(response.json().occurrenceCount).toBe(4); + + const gone = await db + .selectFrom('error_groups') + .select('id') + .where('id', '=', source.id) + .executeTakeFirst(); + expect(gone).toBeUndefined(); + }); + + it('POST /:id/merge returns 400 for empty sourceIds', async () => { + const target = await createTestErrorGroup({ + organizationId: testOrganization.id, + projectId: testProject.id, + }); + + const response = await app.inject({ + method: 'POST', + url: `/api/v1/error-groups/${target.id}/merge`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { organizationId: testOrganization.id, sourceIds: [] }, + }); + + expect(response.statusCode).toBe(400); + }); + + it('POST /:id/merge returns 403 for a non-member organization', async () => { + const other = await createTestContext(); + const target = await createTestErrorGroup({ + organizationId: other.organization.id, + projectId: other.project.id, + }); + + const response = await app.inject({ + method: 'POST', + url: `/api/v1/error-groups/${target.id}/merge`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { + organizationId: other.organization.id, + sourceIds: ['00000000-0000-0000-0000-000000000000'], + }, + }); + + expect(response.statusCode).toBe(403); + }); + }); }); diff --git a/packages/backend/src/tests/modules/exceptions/service.test.ts b/packages/backend/src/tests/modules/exceptions/service.test.ts index 229010e6..8710bce1 100644 --- a/packages/backend/src/tests/modules/exceptions/service.test.ts +++ b/packages/backend/src/tests/modules/exceptions/service.test.ts @@ -1,4 +1,5 @@ import { describe, it, expect, beforeEach } from 'vitest'; +import { randomUUID } from 'node:crypto'; import { ExceptionService } from '../../../modules/exceptions/service.js'; import { db } from '../../../database/index.js'; import { @@ -93,6 +94,318 @@ describe('ExceptionService', () => { }); }); + describe('error group service attribution', () => { + const parsedData = { + exceptionType: 'TypeError', + exceptionMessage: "Cannot read properties of undefined (reading 'x')", + language: 'nodejs' as const, + rawStackTrace: 'TypeError: x\n at h (/app/h.js:1:1)', + frames: [], + }; + + async function affectedServices(orgId: string, fingerprint: string): Promise { + const group = await db + .selectFrom('error_groups') + .select('affected_services') + .where('organization_id', '=', orgId) + .where('fingerprint', '=', fingerprint) + .executeTakeFirst(); + return (group?.affected_services as string[] | undefined) ?? []; + } + + it('attributes the service carried on the exception, not the Postgres logs lookup', async () => { + // Simulate a non-TimescaleDB reservoir (ClickHouse / MongoDB): the log is + // NOT in the Postgres logs table, so the trigger's logs lookup finds + // nothing. The service must still come through from the exception row. + const ctx = await createTestContext(); + const fingerprint = `svc-carry-${randomUUID().slice(0, 8)}`; + + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), // no matching row in `logs` + fingerprint, + service: 'checkout-api', + parsedData, + }); + + expect(await affectedServices(ctx.organization.id, fingerprint)).toEqual(['checkout-api']); + }); + + it('falls back to the logs table when no service is carried (TimescaleDB path)', async () => { + const ctx = await createTestContext(); + const log = await createTestLog({ + projectId: ctx.project.id, + level: 'error', + service: 'billing-worker', + }); + const fingerprint = `svc-fallback-${randomUUID().slice(0, 8)}`; + + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: log.id, + fingerprint, + parsedData, + }); + + expect(await affectedServices(ctx.organization.id, fingerprint)).toEqual(['billing-worker']); + }); + + it('never leaves a group as unknown when the service is known at ingestion', async () => { + const ctx = await createTestContext(); + const fingerprint = `svc-known-${randomUUID().slice(0, 8)}`; + + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint, + service: 'api', + parsedData, + }); + + expect(await affectedServices(ctx.organization.id, fingerprint)).not.toContain('unknown'); + }); + }); + + describe('mergeErrorGroups', () => { + const parsed = { + exceptionType: 'TypeError', + exceptionMessage: "Cannot read properties of undefined (reading 'x')", + language: 'nodejs' as const, + rawStackTrace: 'TypeError: x\n at h (/app/h.js:1:1)', + frames: [], + }; + + it('folds duplicate groups into one and reassigns their exceptions', async () => { + const ctx = await createTestContext(); + const fpTarget = `merge-target-${randomUUID().slice(0, 8)}`; + const fpSource = `merge-source-${randomUUID().slice(0, 8)}`; + + // Target: 1 occurrence on service "api" + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint: fpTarget, + service: 'api', + parsedData: parsed, + }); + // Source: 2 occurrences on service "worker", same type+message, different fingerprint + for (let i = 0; i < 2; i++) { + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint: fpSource, + service: 'worker', + parsedData: parsed, + }); + } + + const groups = await db + .selectFrom('error_groups') + .select(['id', 'fingerprint']) + .where('organization_id', '=', ctx.organization.id) + .execute(); + const target = groups.find((g) => g.fingerprint === fpTarget)!; + const source = groups.find((g) => g.fingerprint === fpSource)!; + + // Same type+message groups surface as duplicates of each other. + const dups = await service.findDuplicateErrorGroups(target.id, ctx.organization.id); + expect(dups.map((d) => d.id)).toContain(source.id); + + const merged = await service.mergeErrorGroups(target.id, [source.id], ctx.organization.id); + expect(merged).not.toBeNull(); + expect(merged!.occurrenceCount).toBe(3); // 1 + 2 + expect([...merged!.affectedServices].sort()).toEqual(['api', 'worker']); + + // Source group is gone + const remaining = await db + .selectFrom('error_groups') + .select('id') + .where('id', '=', source.id) + .executeTakeFirst(); + expect(remaining).toBeUndefined(); + + // Its exceptions now point at the target fingerprint + const stillSource = await db + .selectFrom('exceptions') + .select('id') + .where('organization_id', '=', ctx.organization.id) + .where('fingerprint', '=', fpSource) + .execute(); + expect(stillSource.length).toBe(0); + }); + + it('does not merge groups from another organization', async () => { + const ctx = await createTestContext(); + const other = await createTestContext(); + const fpTarget = `merge-iso-t-${randomUUID().slice(0, 8)}`; + const fpOther = `merge-iso-o-${randomUUID().slice(0, 8)}`; + + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint: fpTarget, + service: 'api', + parsedData: parsed, + }); + await service.createException({ + organizationId: other.organization.id, + projectId: other.project.id, + logId: randomUUID(), + fingerprint: fpOther, + service: 'api', + parsedData: parsed, + }); + + const target = await db + .selectFrom('error_groups') + .select(['id']) + .where('organization_id', '=', ctx.organization.id) + .where('fingerprint', '=', fpTarget) + .executeTakeFirstOrThrow(); + const otherGroup = await db + .selectFrom('error_groups') + .select(['id']) + .where('organization_id', '=', other.organization.id) + .where('fingerprint', '=', fpOther) + .executeTakeFirstOrThrow(); + + // Attempting to merge another org's group is a no-op; it stays put. + const merged = await service.mergeErrorGroups(target.id, [otherGroup.id], ctx.organization.id); + expect(merged!.occurrenceCount).toBe(1); + const survives = await db + .selectFrom('error_groups') + .select('id') + .where('id', '=', otherGroup.id) + .executeTakeFirst(); + expect(survives).not.toBeUndefined(); + }); + + it('does not surface duplicates from another project in the same org', async () => { + const ctx = await createTestContext(); + const project2 = await createTestProject({ organizationId: ctx.organization.id }); + const fpTarget = `dup-proj-t-${randomUUID().slice(0, 8)}`; + const fpOther = `dup-proj-o-${randomUUID().slice(0, 8)}`; + + // Same type+message in two projects of the same org. + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint: fpTarget, + service: 'api', + parsedData: parsed, + }); + await service.createException({ + organizationId: ctx.organization.id, + projectId: project2.id, + logId: randomUUID(), + fingerprint: fpOther, + service: 'api', + parsedData: parsed, + }); + + const target = await db + .selectFrom('error_groups') + .select(['id']) + .where('organization_id', '=', ctx.organization.id) + .where('fingerprint', '=', fpTarget) + .executeTakeFirstOrThrow(); + const otherGroup = await db + .selectFrom('error_groups') + .select(['id']) + .where('organization_id', '=', ctx.organization.id) + .where('fingerprint', '=', fpOther) + .executeTakeFirstOrThrow(); + + const dups = await service.findDuplicateErrorGroups(target.id, ctx.organization.id); + expect(dups.map((d) => d.id)).not.toContain(otherGroup.id); + }); + + it('does not merge or retag a sibling project sharing the same fingerprint', async () => { + const ctx = await createTestContext(); + const project2 = await createTestProject({ organizationId: ctx.organization.id }); + const fpTarget = `merge-proj-t-${randomUUID().slice(0, 8)}`; + // Deliberately identical fingerprint string across two projects. + const fpShared = `merge-proj-shared-${randomUUID().slice(0, 8)}`; + + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint: fpTarget, + service: 'api', + parsedData: parsed, + }); + // Source in the target's project (legitimate merge candidate). + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint: fpShared, + service: 'api', + parsedData: parsed, + }); + // Same fingerprint string, different project: must be untouched. + await service.createException({ + organizationId: ctx.organization.id, + projectId: project2.id, + logId: randomUUID(), + fingerprint: fpShared, + service: 'api', + parsedData: parsed, + }); + + const target = await db + .selectFrom('error_groups') + .select(['id']) + .where('organization_id', '=', ctx.organization.id) + .where('project_id', '=', ctx.project.id) + .where('fingerprint', '=', fpTarget) + .executeTakeFirstOrThrow(); + const sameProjectSource = await db + .selectFrom('error_groups') + .select(['id']) + .where('organization_id', '=', ctx.organization.id) + .where('project_id', '=', ctx.project.id) + .where('fingerprint', '=', fpShared) + .executeTakeFirstOrThrow(); + const siblingGroup = await db + .selectFrom('error_groups') + .select(['id']) + .where('organization_id', '=', ctx.organization.id) + .where('project_id', '=', project2.id) + .where('fingerprint', '=', fpShared) + .executeTakeFirstOrThrow(); + + // Merging the same-project source folds it in; the sibling project's group + // with the identical fingerprint stays put and its exceptions keep their + // fingerprint. + await service.mergeErrorGroups(target.id, [sameProjectSource.id], ctx.organization.id); + + const siblingSurvives = await db + .selectFrom('error_groups') + .select('id') + .where('id', '=', siblingGroup.id) + .executeTakeFirst(); + expect(siblingSurvives).not.toBeUndefined(); + + const siblingExceptions = await db + .selectFrom('exceptions') + .select('id') + .where('organization_id', '=', ctx.organization.id) + .where('project_id', '=', project2.id) + .where('fingerprint', '=', fpShared) + .execute(); + expect(siblingExceptions.length).toBe(1); + }); + }); + describe('getExceptionByLogId', () => { it('should return exception with frames for valid log', async () => { const ctx = await createTestContext(); diff --git a/packages/backend/src/tests/modules/ingestion/ingestion-service.test.ts b/packages/backend/src/tests/modules/ingestion/ingestion-service.test.ts index 771d3495..f3b3db1b 100644 --- a/packages/backend/src/tests/modules/ingestion/ingestion-service.test.ts +++ b/packages/backend/src/tests/modules/ingestion/ingestion-service.test.ts @@ -473,4 +473,68 @@ describe('IngestionService', () => { expect(result.rejected).toEqual([]); }); }); + + describe('clock skew detection (#279)', () => { + it('records a skew counter and still stores the log', async () => { + const recordSpy = vi.spyOn(metering, 'record'); + + const skewed = new Date(Date.now() - 27 * 60 * 60 * 1000).toISOString(); + const result = await ingestionService.ingestLogs( + [{ time: skewed, service: 'ubuntu', level: 'critical', message: 'skewed log' }], + projectId, + ); + + // The invariant that matters: skew never rejects. + expect(result.received).toBe(1); + expect(result.rejected).toEqual([]); + + const skewCall = recordSpy.mock.calls.find( + ([e]) => e.type === 'ingestion.timestamp_skew', + ); + expect(skewCall).toBeDefined(); + expect(skewCall![0].quantity).toBe(1); + expect(skewCall![0].projectId).toBe(projectId); + expect(skewCall![0].metadata).toMatchObject({ maxFutureMs: 0 }); + expect((skewCall![0].metadata as { maxPastMs: number }).maxPastMs).toBeGreaterThan( + 24 * 60 * 60 * 1000, + ); + + recordSpy.mockRestore(); + }); + + it('records no skew counter for a fresh log', async () => { + const recordSpy = vi.spyOn(metering, 'record'); + + await ingestionService.ingestLogs( + [{ time: new Date().toISOString(), service: 'ubuntu', level: 'info', message: 'fresh' }], + projectId, + ); + + expect( + recordSpy.mock.calls.find(([e]) => e.type === 'ingestion.timestamp_skew'), + ).toBeUndefined(); + + recordSpy.mockRestore(); + }); + + it('counts only the skewed records in a mixed batch', async () => { + const recordSpy = vi.spyOn(metering, 'record'); + + await ingestionService.ingestLogs( + [ + { time: new Date(Date.now() - 27 * 60 * 60 * 1000).toISOString(), service: 'a', level: 'info', message: 'old' }, + { time: new Date().toISOString(), service: 'a', level: 'info', message: 'fresh' }, + { time: new Date(Date.now() - 30 * 60 * 60 * 1000).toISOString(), service: 'a', level: 'info', message: 'older' }, + ], + projectId, + ); + + const skewCall = recordSpy.mock.calls.find( + ([e]) => e.type === 'ingestion.timestamp_skew', + ); + expect(skewCall![0].quantity).toBe(2); + + recordSpy.mockRestore(); + }); + }); }); diff --git a/packages/backend/src/tests/modules/ingestion/skew.test.ts b/packages/backend/src/tests/modules/ingestion/skew.test.ts new file mode 100644 index 00000000..0b3e31e9 --- /dev/null +++ b/packages/backend/src/tests/modules/ingestion/skew.test.ts @@ -0,0 +1,48 @@ +import { describe, it, expect } from 'vitest'; +import { createSkewTracker } from '../../../modules/ingestion/skew.js'; + +const NOW = new Date('2026-07-17T12:00:00.000Z').getTime(); +const ago = (ms: number) => new Date(NOW - ms); + +describe('createSkewTracker', () => { + it('returns null when nothing is skewed', () => { + const t = createSkewTracker(NOW); + t.observe(ago(0)); + t.observe(ago(60_000)); + t.observe(ago(86_399_000)); + expect(t.summary()).toBeNull(); + }); + + it('counts a log older than the past threshold and records the worst delta', () => { + const t = createSkewTracker(NOW); + t.observe(ago(97_200_000)); // 27h in the past, the #279 case + t.observe(ago(90_000_000)); // 25h, skewed but less extreme + expect(t.summary()).toEqual({ count: 2, maxPastMs: 97_200_000, maxFutureMs: 0 }); + }); + + it('counts a log ahead of the future threshold and records the worst delta', () => { + const t = createSkewTracker(NOW); + t.observe(ago(-600_000)); // 10m in the future + expect(t.summary()).toEqual({ count: 1, maxPastMs: 0, maxFutureMs: 600_000 }); + }); + + it('tracks both directions in one batch', () => { + const t = createSkewTracker(NOW); + t.observe(ago(97_200_000)); + t.observe(ago(-600_000)); + t.observe(ago(1000)); + expect(t.summary()).toEqual({ count: 2, maxPastMs: 97_200_000, maxFutureMs: 600_000 }); + }); + + it('does not count an Invalid Date as skew', () => { + const t = createSkewTracker(NOW); + t.observe(new Date('garbage')); + expect(t.summary()).toBeNull(); + }); + + it('does not count a missing time', () => { + const t = createSkewTracker(NOW); + t.observe(undefined); + expect(t.summary()).toBeNull(); + }); +}); diff --git a/packages/backend/src/tests/modules/metering/recorder.test.ts b/packages/backend/src/tests/modules/metering/recorder.test.ts index e1055632..8b08388a 100644 --- a/packages/backend/src/tests/modules/metering/recorder.test.ts +++ b/packages/backend/src/tests/modules/metering/recorder.test.ts @@ -1,5 +1,6 @@ -import { describe, it, expect, beforeEach } from 'vitest'; +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import { db } from '../../../database/index.js'; +import { config } from '../../../config/index.js'; import { MeteringRecorder } from '../../../modules/metering/recorder.js'; import { createTestContext } from '../../helpers/factories.js'; @@ -62,4 +63,34 @@ describe('MeteringRecorder', () => { expect(rows).toHaveLength(1); expect(Number(rows[0].quantity)).toBe(99); }); + + describe('METERING_ENABLED gate', () => { + const original = config.METERING_ENABLED; + + afterEach(() => { + config.METERING_ENABLED = original; + }); + + it('still buffers an ingestion.* health counter when metering is disabled', () => { + config.METERING_ENABLED = false; + const rec = new MeteringRecorder({ maxBuffer: 1000 }); + rec.record({ type: 'ingestion.timestamp_skew', quantity: 1, organizationId: orgId, projectId }); + expect(rec.bufferSize).toBe(1); + }); + + it('does not buffer a usage event when metering is disabled', () => { + config.METERING_ENABLED = false; + const rec = new MeteringRecorder({ maxBuffer: 1000 }); + rec.record({ type: 'logs.ingested.events', quantity: 1, organizationId: orgId, projectId }); + expect(rec.bufferSize).toBe(0); + }); + + it('buffers both ingestion.* and usage events when metering is enabled', () => { + config.METERING_ENABLED = true; + const rec = new MeteringRecorder({ maxBuffer: 1000 }); + rec.record({ type: 'ingestion.timestamp_skew', quantity: 1, organizationId: orgId, projectId }); + rec.record({ type: 'logs.ingested.events', quantity: 1, organizationId: orgId, projectId }); + expect(rec.bufferSize).toBe(2); + }); + }); }); diff --git a/packages/backend/src/tests/modules/metrics/service.test.ts b/packages/backend/src/tests/modules/metrics/service.test.ts index 4c1d9e85..6ded7545 100644 --- a/packages/backend/src/tests/modules/metrics/service.test.ts +++ b/packages/backend/src/tests/modules/metrics/service.test.ts @@ -20,6 +20,14 @@ vi.mock('../../../database/reservoir.js', () => ({ }, })); +const mockMeteringRecord = vi.fn(); + +vi.mock('../../../modules/metering/index.js', () => ({ + metering: { + record: (...args: unknown[]) => mockMeteringRecord(...args), + }, +})); + import { MetricsService } from '../../../modules/metrics/service.js'; describe('MetricsService', () => { @@ -100,6 +108,90 @@ describe('MetricsService', () => { expect(result).toBe(5); }); + + describe('clock skew detection (span-metric-skew)', () => { + it('counts a metric with a past-skewed time', async () => { + mockIngestMetrics.mockResolvedValueOnce({ ingested: 1 }); + + const skewedTime = new Date(Date.now() - 27 * 60 * 60 * 1000); + const records = [ + { + time: skewedTime, + metricName: 'old_metric', + metricType: 'gauge' as const, + value: 1, + serviceName: 'svc', + organizationId: '', + projectId: '', + }, + ]; + + const result = await service.ingestMetrics(records, 'proj-1', 'org-1'); + + // The skewed metric is still written, not rejected. + expect(result).toBe(1); + expect(mockIngestMetrics).toHaveBeenCalledOnce(); + + const skewCall = mockMeteringRecord.mock.calls.find( + ([e]) => e.type === 'ingestion.metric_timestamp_skew', + ); + expect(skewCall).toBeDefined(); + expect(skewCall![0].quantity).toBe(1); + expect(skewCall![0].organizationId).toBe('org-1'); + expect(skewCall![0].projectId).toBe('proj-1'); + expect((skewCall![0].metadata as { maxPastMs: number }).maxPastMs).toBeGreaterThan( + 24 * 60 * 60 * 1000, + ); + }); + + it('counts a metric with a future-skewed time', async () => { + mockIngestMetrics.mockResolvedValueOnce({ ingested: 1 }); + + const futureTime = new Date(Date.now() + 10 * 60 * 1000); + const records = [ + { + time: futureTime, + metricName: 'future_metric', + metricType: 'gauge' as const, + value: 1, + serviceName: 'svc', + organizationId: '', + projectId: '', + }, + ]; + + await service.ingestMetrics(records, 'proj-1', 'org-1'); + + const skewCall = mockMeteringRecord.mock.calls.find( + ([e]) => e.type === 'ingestion.metric_timestamp_skew', + ); + expect(skewCall).toBeDefined(); + expect(skewCall![0].quantity).toBe(1); + expect((skewCall![0].metadata as { maxFutureMs: number }).maxFutureMs).toBeGreaterThan(0); + }); + + it('does not count a fresh metric', async () => { + mockIngestMetrics.mockResolvedValueOnce({ ingested: 1 }); + + const records = [ + { + time: new Date(), + metricName: 'fresh_metric', + metricType: 'gauge' as const, + value: 1, + serviceName: 'svc', + organizationId: '', + projectId: '', + }, + ]; + + await service.ingestMetrics(records, 'proj-1', 'org-1'); + + expect( + mockMeteringRecord.mock.calls.find(([e]) => e.type === 'ingestion.metric_timestamp_skew'), + ).toBeUndefined(); + }); + }); }); describe('listMetricNames', () => { diff --git a/packages/backend/src/tests/modules/projects/routes.test.ts b/packages/backend/src/tests/modules/projects/routes.test.ts index 66261651..118cc85f 100644 --- a/packages/backend/src/tests/modules/projects/routes.test.ts +++ b/packages/backend/src/tests/modules/projects/routes.test.ts @@ -2,6 +2,11 @@ import { describe, it, expect, beforeEach, afterAll, beforeAll } from 'vitest'; import Fastify, { FastifyInstance } from 'fastify'; import { db } from '../../../database/index.js'; import { projectsRoutes } from '../../../modules/projects/routes.js'; +import { ingestionService } from '../../../modules/ingestion/service.js'; +import { meteringRecorder } from '../../../modules/metering/index.js'; +import { tracesService } from '../../../modules/traces/index.js'; +import { metricsService } from '../../../modules/metrics/index.js'; +import type { TransformedSpan } from '../../../modules/otlp/trace-transformer.js'; import { createTestContext, createTestUser, createTestProject, createTestOrganization } from '../../helpers/factories.js'; import crypto from 'crypto'; @@ -469,4 +474,298 @@ describe('Projects Routes', () => { expect(response.statusCode).toBe(401); }); }); + + describe('GET /:id/ingestion-health', () => { + let otherOrgProjectId: string; + + beforeEach(async () => { + // Second organization the test user is not a member of. + const otherOrg = await createTestOrganization(); + const otherProject = await createTestProject({ organizationId: otherOrg.id }); + otherOrgProjectId = otherProject.id; + }); + + it('returns null skew for every signal when nothing was recorded', async () => { + const response = await app.inject({ + method: 'GET', + url: `/api/v1/projects/${testProject.id}/ingestion-health`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(response.statusCode).toBe(200); + expect(response.json()).toEqual({ skew: { logs: null, spans: null, metrics: null } }); + }); + + it('aggregates skew events from the last 24h', async () => { + const olderEventTime = new Date(Date.now() - 60 * 60 * 1000); + const newestEventTime = new Date(Date.now() - 30 * 60 * 1000); + + await db + .insertInto('metering_events') + .values([ + { + organization_id: testOrganization.id, + project_id: testProject.id, + type: 'ingestion.timestamp_skew', + quantity: 4, + metadata: { maxPastMs: 97200000, maxFutureMs: 0 }, + time: olderEventTime, + }, + { + organization_id: testOrganization.id, + project_id: testProject.id, + type: 'ingestion.timestamp_skew', + quantity: 6, + metadata: { maxPastMs: 90000000, maxFutureMs: 600000 }, + time: newestEventTime, + }, + ]) + .execute(); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/projects/${testProject.id}/ingestion-health`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(response.statusCode).toBe(200); + const { skew } = response.json(); + expect(skew.logs.count24h).toBe(10); + expect(skew.logs.maxPastMs).toBe(97200000); // worst across events, not last + expect(skew.logs.maxFutureMs).toBe(600000); + // Pins lastSeenAt to the newest fixture row's time (not "now"), with a + // small tolerance for DB round-tripping. + expect(Math.abs(new Date(skew.logs.lastSeenAt).getTime() - newestEventTime.getTime())).toBeLessThan( + 5000, + ); + // Only logs had events; the other two signals stay null independently. + expect(skew.spans).toBeNull(); + expect(skew.metrics).toBeNull(); + }); + + it('aggregates each signal independently, not into a shared counter', async () => { + await db + .insertInto('metering_events') + .values([ + { + organization_id: testOrganization.id, + project_id: testProject.id, + type: 'ingestion.timestamp_skew', + quantity: 4, + metadata: { maxPastMs: 97200000, maxFutureMs: 0 }, + }, + { + organization_id: testOrganization.id, + project_id: testProject.id, + type: 'ingestion.span_timestamp_skew', + quantity: 2, + metadata: { maxPastMs: 88000000, maxFutureMs: 0 }, + }, + { + organization_id: testOrganization.id, + project_id: testProject.id, + type: 'ingestion.metric_timestamp_skew', + quantity: 1, + metadata: { maxPastMs: 0, maxFutureMs: 500000 }, + }, + ]) + .execute(); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/projects/${testProject.id}/ingestion-health`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(response.statusCode).toBe(200); + const { skew } = response.json(); + expect(skew.logs.count24h).toBe(4); + expect(skew.logs.maxPastMs).toBe(97200000); + expect(skew.spans.count24h).toBe(2); + expect(skew.spans.maxPastMs).toBe(88000000); + expect(skew.metrics.count24h).toBe(1); + expect(skew.metrics.maxFutureMs).toBe(500000); + }); + + it('ignores skew events older than 24h', async () => { + await db + .insertInto('metering_events') + .values({ + organization_id: testOrganization.id, + project_id: testProject.id, + type: 'ingestion.timestamp_skew', + quantity: 99, + metadata: { maxPastMs: 97200000, maxFutureMs: 0 }, + time: new Date(Date.now() - 25 * 60 * 60 * 1000), + }) + .execute(); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/projects/${testProject.id}/ingestion-health`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(response.json()).toEqual({ skew: { logs: null, spans: null, metrics: null } }); + }); + + it('does not leak skew from another organization', async () => { + // otherOrgProjectId belongs to an org the test user is not a member of. + const response = await app.inject({ + method: 'GET', + url: `/api/v1/projects/${otherOrgProjectId}/ingestion-health`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(response.statusCode).toBe(404); + }); + + it('reads the exact keys a real ingest writes (writer/reader contract, logs)', async () => { + // #279 repro: a log ~27h in the past, past the 24h skew threshold. + const skewedTime = new Date(Date.now() - 27 * 60 * 60 * 1000).toISOString(); + const ingestResult = await ingestionService.ingestLogs( + [{ time: skewedTime, service: 'ubuntu', level: 'critical', message: 'skewed log' }], + testProject.id, + ); + expect(ingestResult.received).toBe(1); + expect(ingestResult.rejected).toEqual([]); + + // Force the buffered metering event through to the DB rather than + // waiting for the recorder's interval flush. + await meteringRecorder.flush(); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/projects/${testProject.id}/ingestion-health`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(response.statusCode).toBe(200); + const { skew } = response.json(); + expect(skew.logs.count24h).toBe(1); + expect(skew.logs.maxPastMs).toBeGreaterThan(24 * 60 * 60 * 1000); + + // Required invariant: a skewed log is stored, never rejected. + const storedLog = await db + .selectFrom('logs') + .selectAll() + .where('project_id', '=', testProject.id) + .where('message', '=', 'skewed log') + .executeTakeFirst(); + expect(storedLog).toBeDefined(); + }); + + it('reads the exact keys a real ingest writes (writer/reader contract, spans)', async () => { + // span-metric-skew: spans are observed on end_time, not start_time. + const skewedEnd = new Date(Date.now() - 27 * 60 * 60 * 1000); + const spans: TransformedSpan[] = [ + { + trace_id: crypto.randomBytes(16).toString('hex'), + span_id: crypto.randomBytes(8).toString('hex'), + service_name: 'skewed-service', + operation_name: 'skewed-op', + start_time: new Date(skewedEnd.getTime() - 50).toISOString(), + end_time: skewedEnd.toISOString(), + duration_ms: 50, + }, + ]; + + const ingested = await tracesService.ingestSpans( + spans, + new Map(), + testProject.id, + testOrganization.id, + ); + expect(ingested).toBe(1); + + await meteringRecorder.flush(); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/projects/${testProject.id}/ingestion-health`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(response.statusCode).toBe(200); + const { skew } = response.json(); + expect(skew.spans.count24h).toBe(1); + expect(skew.spans.maxPastMs).toBeGreaterThan(24 * 60 * 60 * 1000); + + const storedSpan = await db + .selectFrom('spans') + .selectAll() + .where('span_id', '=', spans[0].span_id) + .executeTakeFirst(); + expect(storedSpan).toBeDefined(); + }); + + it('reads the exact keys a real ingest writes (writer/reader contract, metrics)', async () => { + const skewedTime = new Date(Date.now() - 27 * 60 * 60 * 1000); + + const ingested = await metricsService.ingestMetrics( + [ + { + time: skewedTime, + metricName: 'skewed.metric', + metricType: 'gauge', + value: 42, + serviceName: 'skewed-service', + }, + ], + testProject.id, + testOrganization.id, + ); + expect(ingested).toBe(1); + + await meteringRecorder.flush(); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/projects/${testProject.id}/ingestion-health`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(response.statusCode).toBe(200); + const { skew } = response.json(); + expect(skew.metrics.count24h).toBe(1); + expect(skew.metrics.maxPastMs).toBeGreaterThan(24 * 60 * 60 * 1000); + }); + + it('takes the numeric max of maxPastMs, not the lexicographic (text) max', async () => { + // "9500000" > "10000000" as text (compares '9' vs '1' first), but + // 10000000 is the larger number. A MAX() over uncasted metadata->>'x' + // text would wrongly report 9500000 here. + await db + .insertInto('metering_events') + .values([ + { + organization_id: testOrganization.id, + project_id: testProject.id, + type: 'ingestion.timestamp_skew', + quantity: 1, + metadata: { maxPastMs: 9500000, maxFutureMs: 0 }, + time: new Date(Date.now() - 60 * 60 * 1000), + }, + { + organization_id: testOrganization.id, + project_id: testProject.id, + type: 'ingestion.timestamp_skew', + quantity: 1, + metadata: { maxPastMs: 10000000, maxFutureMs: 0 }, + time: new Date(Date.now() - 30 * 60 * 1000), + }, + ]) + .execute(); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/projects/${testProject.id}/ingestion-health`, + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(response.statusCode).toBe(200); + const { skew } = response.json(); + expect(skew.logs.maxPastMs).toBe(10000000); + }); + }); }); diff --git a/packages/backend/src/tests/modules/traces/service.test.ts b/packages/backend/src/tests/modules/traces/service.test.ts index 2666a506..010a6480 100644 --- a/packages/backend/src/tests/modules/traces/service.test.ts +++ b/packages/backend/src/tests/modules/traces/service.test.ts @@ -1,7 +1,9 @@ -import { describe, it, expect, beforeEach } from 'vitest'; +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; import { TracesService } from '../../../modules/traces/service.js'; import { createTestContext, createTestTrace, createTestSpan, createTestLog } from '../../helpers/index.js'; import { db } from '../../../database/index.js'; +import { metering } from '../../../modules/metering/index.js'; +import { piiMaskingService } from '../../../modules/pii-masking/service.js'; import type { TransformedSpan, AggregatedTrace } from '../../../modules/otlp/trace-transformer.js'; import crypto from 'crypto'; @@ -259,6 +261,166 @@ describe('TracesService', () => { }); }); + // ========================================================================== + // ingestSpans - clock skew detection (span-metric-skew) + // ========================================================================== + describe('ingestSpans clock skew detection', () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('counts a span whose end_time is far in the past', async () => { + const recordSpy = vi.spyOn(metering, 'record'); + + const traceId = crypto.randomBytes(16).toString('hex'); + const skewedEnd = new Date(Date.now() - 27 * 60 * 60 * 1000); + + const spans: TransformedSpan[] = [ + { + trace_id: traceId, + span_id: crypto.randomBytes(8).toString('hex'), + service_name: 'skewed-service', + operation_name: 'skewed-op', + start_time: new Date(skewedEnd.getTime() - 50).toISOString(), + end_time: skewedEnd.toISOString(), + duration_ms: 50, + }, + ]; + + const result = await service.ingestSpans( + spans, + new Map(), + context.project.id, + context.organization.id, + ); + + expect(result).toBe(1); + + const skewCall = recordSpy.mock.calls.find( + ([e]) => e.type === 'ingestion.span_timestamp_skew', + ); + expect(skewCall).toBeDefined(); + expect(skewCall![0].quantity).toBe(1); + expect(skewCall![0].organizationId).toBe(context.organization.id); + expect(skewCall![0].projectId).toBe(context.project.id); + expect((skewCall![0].metadata as { maxPastMs: number }).maxPastMs).toBeGreaterThan( + 24 * 60 * 60 * 1000, + ); + + // The skewed span is still written, not rejected. + const storedSpan = await db + .selectFrom('spans') + .selectAll() + .where('span_id', '=', spans[0].span_id) + .executeTakeFirst(); + expect(storedSpan).toBeDefined(); + }); + + it('counts a span whose end_time is far in the future', async () => { + const recordSpy = vi.spyOn(metering, 'record'); + + const traceId = crypto.randomBytes(16).toString('hex'); + const futureEnd = new Date(Date.now() + 60 * 60 * 1000); + + const spans: TransformedSpan[] = [ + { + trace_id: traceId, + span_id: crypto.randomBytes(8).toString('hex'), + service_name: 'future-service', + operation_name: 'future-op', + start_time: new Date(futureEnd.getTime() - 50).toISOString(), + end_time: futureEnd.toISOString(), + duration_ms: 50, + }, + ]; + + const result = await service.ingestSpans( + spans, + new Map(), + context.project.id, + context.organization.id, + ); + + expect(result).toBe(1); + + const skewCall = recordSpy.mock.calls.find( + ([e]) => e.type === 'ingestion.span_timestamp_skew', + ); + expect(skewCall).toBeDefined(); + expect(skewCall![0].quantity).toBe(1); + expect((skewCall![0].metadata as { maxFutureMs: number }).maxFutureMs).toBeGreaterThan( + 5 * 60 * 1000, + ); + }); + + it('does not count a long-running span (old start_time, fresh end_time)', async () => { + const recordSpy = vi.spyOn(metering, 'record'); + + const traceId = crypto.randomBytes(16).toString('hex'); + const now = new Date(); + const oldStart = new Date(now.getTime() - 27 * 60 * 60 * 1000); + + const spans: TransformedSpan[] = [ + { + trace_id: traceId, + span_id: crypto.randomBytes(8).toString('hex'), + service_name: 'batch-job', + operation_name: 'long-running-op', + start_time: oldStart.toISOString(), + end_time: now.toISOString(), + duration_ms: now.getTime() - oldStart.getTime(), + }, + ]; + + const result = await service.ingestSpans( + spans, + new Map(), + context.project.id, + context.organization.id, + ); + + expect(result).toBe(1); + + expect( + recordSpy.mock.calls.find(([e]) => e.type === 'ingestion.span_timestamp_skew'), + ).toBeUndefined(); + }); + + it('does not count a span dropped by PII masking', async () => { + vi.spyOn(piiMaskingService, 'maskSpanBatch').mockResolvedValue([0]); + const recordSpy = vi.spyOn(metering, 'record'); + + const traceId = crypto.randomBytes(16).toString('hex'); + const skewedEnd = new Date(Date.now() - 27 * 60 * 60 * 1000); + + const spans: TransformedSpan[] = [ + { + trace_id: traceId, + span_id: crypto.randomBytes(8).toString('hex'), + service_name: 'masked-out-service', + operation_name: 'masked-op', + start_time: new Date(skewedEnd.getTime() - 50).toISOString(), + end_time: skewedEnd.toISOString(), + duration_ms: 50, + }, + ]; + + const result = await service.ingestSpans( + spans, + new Map(), + context.project.id, + context.organization.id, + ); + + // The masked-out span is dropped pre-storage, so nothing is ingested + // and it must not be counted as skew either. + expect(result).toBe(0); + expect( + recordSpy.mock.calls.find(([e]) => e.type === 'ingestion.span_timestamp_skew'), + ).toBeUndefined(); + }); + }); + // ========================================================================== // listTraces // ========================================================================== diff --git a/packages/backend/src/tests/modules/users/routes.test.ts b/packages/backend/src/tests/modules/users/routes.test.ts index 7c6ed5ec..f359fe48 100644 --- a/packages/backend/src/tests/modules/users/routes.test.ts +++ b/packages/backend/src/tests/modules/users/routes.test.ts @@ -1,9 +1,24 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; import Fastify, { type FastifyInstance } from 'fastify'; +import { hub } from '@logtide/core'; import { db } from '../../../database/index.js'; import { usersRoutes } from '../../../modules/users/routes.js'; import { usersService } from '../../../modules/users/service.js'; import { settingsService } from '../../../modules/settings/service.js'; +import { authenticationService } from '../../../modules/auth/authentication-service.js'; +import { AuthError, AuthErrorCode } from '../../../modules/auth/providers/types.js'; +import { createTestSession } from '../../helpers/auth.js'; + +// Mock internal observability so the auto-login failure path is fully +// exercised (isInternalLoggingEnabled is env-driven and off in tests) +vi.mock('@logtide/core', async (importOriginal) => ({ + ...(await importOriginal()), + hub: { captureLog: vi.fn() }, +})); +vi.mock('../../../utils/internal-logger.js', async (importOriginal) => ({ + ...(await importOriginal()), + isInternalLoggingEnabled: () => true, +})); // Mock settingsService vi.mock('../../../modules/settings/service.js', () => ({ @@ -73,6 +88,42 @@ describe('Users Routes', () => { expect(body.session.token).toBeDefined(); }); + it('returns 201 without a session and logs when auto-login fails', async () => { + const authSpy = vi.spyOn(authenticationService, 'authenticateWithProvider') + .mockRejectedValueOnce(new Error('session backend unavailable')); + + const response = await app.inject({ + method: 'POST', + url: '/register', + payload: { + email: 'no-session@example.com', + password: 'password123', + name: 'No Session', + }, + }); + + expect(response.statusCode).toBe(201); + const body = JSON.parse(response.body); + expect(body.user.email).toBe('no-session@example.com'); + expect(body.session).toBeUndefined(); + + // The swallowed failure must be visible to operators + expect(hub.captureLog).toHaveBeenCalledWith( + 'error', + '[Auth] Post-registration auto-login failed', + expect.objectContaining({ error: 'session backend unavailable' }) + ); + + // Only the registration auto-login was forced to fail; the account exists + // and is recoverable via a normal login through the provider path. + authSpy.mockRestore(); + const result = await authenticationService.authenticateWithProvider('local', { + email: 'no-session@example.com', + password: 'password123', + }); + expect(result.session.token).toBeDefined(); + }); + it('should return 400 for invalid email', async () => { const response = await app.inject({ method: 'POST', @@ -221,6 +272,25 @@ describe('Users Routes', () => { expect(response.statusCode).toBe(400); }); + + it('returns 503 when the local provider is unavailable', async () => { + const authSpy = vi.spyOn(authenticationService, 'authenticateWithProvider') + .mockRejectedValueOnce( + new AuthError("Authentication provider 'local' not found or disabled", AuthErrorCode.PROVIDER_UNAVAILABLE) + ); + + const response = await app.inject({ + method: 'POST', + url: '/login', + payload: { + email: 'login@example.com', + password: 'password123', + }, + }); + + expect(response.statusCode).toBe(503); + authSpy.mockRestore(); + }); }); describe('POST /logout', () => { @@ -231,10 +301,7 @@ describe('Users Routes', () => { name: 'Logout User', }); - const session = await usersService.login({ - email: 'logout@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); const response = await app.inject({ method: 'POST', @@ -267,10 +334,7 @@ describe('Users Routes', () => { name: 'Me User', }); - const session = await usersService.login({ - email: 'me@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); const response = await app.inject({ method: 'GET', @@ -310,16 +374,13 @@ describe('Users Routes', () => { describe('PUT /me', () => { it('should update user name', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'update@example.com', password: 'password123', name: 'Old Name', }); - const session = await usersService.login({ - email: 'update@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); const response = await app.inject({ method: 'PUT', @@ -338,16 +399,13 @@ describe('Users Routes', () => { }); it('should update email', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'oldemail@example.com', password: 'password123', name: 'Test User', }); - const session = await usersService.login({ - email: 'oldemail@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); const response = await app.inject({ method: 'PUT', @@ -366,16 +424,13 @@ describe('Users Routes', () => { }); it('should update password with correct current password', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'password@example.com', password: 'oldpassword', name: 'Test User', }); - const session = await usersService.login({ - email: 'password@example.com', - password: 'oldpassword', - }); + const session = await createTestSession(user.id); const response = await app.inject({ method: 'PUT', @@ -391,25 +446,22 @@ describe('Users Routes', () => { expect(response.statusCode).toBe(200); - // Verify can login with new password - const newSession = await usersService.login({ + // Verify the new password authenticates through the provider path + const result = await authenticationService.authenticateWithProvider('local', { email: 'password@example.com', password: 'newpassword123', }); - expect(newSession.token).toBeDefined(); + expect(result.session.token).toBeDefined(); }); it('should return 400 when changing password without current password', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'nopassword@example.com', password: 'password123', name: 'Test User', }); - const session = await usersService.login({ - email: 'nopassword@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); const response = await app.inject({ method: 'PUT', @@ -444,16 +496,13 @@ describe('Users Routes', () => { name: 'Existing User', }); - await usersService.createUser({ + const user = await usersService.createUser({ email: 'changemail@example.com', password: 'password123', name: 'Change Mail User', }); - const session = await usersService.login({ - email: 'changemail@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); const response = await app.inject({ method: 'PUT', @@ -472,16 +521,13 @@ describe('Users Routes', () => { describe('DELETE /me', () => { it('should delete user with correct password', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'delete@example.com', password: 'password123', name: 'Delete User', }); - const session = await usersService.login({ - email: 'delete@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); const response = await app.inject({ method: 'DELETE', @@ -497,25 +543,22 @@ describe('Users Routes', () => { expect(response.statusCode).toBe(204); // Verify user is deleted - const user = await db + const deletedUser = await db .selectFrom('users') .select('id') .where('email', '=', 'delete@example.com') .executeTakeFirst(); - expect(user).toBeUndefined(); + expect(deletedUser).toBeUndefined(); }); it('should return 400 with incorrect password', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'nodelete@example.com', password: 'password123', name: 'No Delete User', }); - const session = await usersService.login({ - email: 'nodelete@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); const response = await app.inject({ method: 'DELETE', @@ -532,16 +575,13 @@ describe('Users Routes', () => { }); it('should return 400 without password', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'needpassword@example.com', password: 'password123', name: 'Need Password User', }); - const session = await usersService.login({ - email: 'needpassword@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); const response = await app.inject({ method: 'DELETE', diff --git a/packages/backend/src/tests/modules/users/users-service.test.ts b/packages/backend/src/tests/modules/users/users-service.test.ts index 35481494..0b582637 100644 --- a/packages/backend/src/tests/modules/users/users-service.test.ts +++ b/packages/backend/src/tests/modules/users/users-service.test.ts @@ -1,6 +1,8 @@ import { describe, it, expect, beforeEach } from 'vitest'; import { db } from '../../../database/index.js'; import { UsersService } from '../../../modules/users/service.js'; +import { createTestSession } from '../../helpers/auth.js'; +import { authenticationService } from '../../../modules/auth/authentication-service.js'; describe('UsersService', () => { let usersService: UsersService; @@ -192,111 +194,11 @@ describe('UsersService', () => { }); }); - describe('login', () => { - it('should return session info for valid credentials', async () => { - await usersService.createUser({ - email: 'login@example.com', - password: 'password123', - name: 'Login User', - }); - - const session = await usersService.login({ - email: 'login@example.com', - password: 'password123', - }); - - expect(session.sessionId).toBeDefined(); - expect(session.userId).toBeDefined(); - expect(session.token).toBeDefined(); - expect(session.token).toHaveLength(64); - expect(session.expiresAt).toBeInstanceOf(Date); - expect(session.expiresAt.getTime()).toBeGreaterThan(Date.now()); - }); - - it('should throw error for non-existent user', async () => { - await expect( - usersService.login({ - email: 'nonexistent@example.com', - password: 'password123', - }) - ).rejects.toThrow('Invalid email or password'); - }); - - it('should throw error for wrong password', async () => { - await usersService.createUser({ - email: 'wrongpass@example.com', - password: 'correctPassword', - name: 'Test User', - }); - - await expect( - usersService.login({ - email: 'wrongpass@example.com', - password: 'wrongPassword', - }) - ).rejects.toThrow('Invalid email or password'); - }); - - it('should update last_login timestamp', async () => { - const user = await usersService.createUser({ - email: 'lastlogin@example.com', - password: 'password123', - name: 'Last Login User', - }); - - expect(user.lastLogin).toBeNull(); - - await usersService.login({ - email: 'lastlogin@example.com', - password: 'password123', - }); - - const updatedUser = await usersService.getUserById(user.id); - expect(updatedUser?.lastLogin).not.toBeNull(); - }); - - it('should reject login for disabled user', async () => { - await usersService.createUser({ - email: 'disabled@example.com', - password: 'password123', - name: 'Disabled User', - }); - - await db - .updateTable('users') - .set({ disabled: true }) - .where('email', '=', 'disabled@example.com') - .execute(); - - await expect( - usersService.login({ - email: 'disabled@example.com', - password: 'password123', - }) - ).rejects.toThrow('This account has been disabled'); - }); - - it('should allow multiple concurrent sessions', async () => { - await usersService.createUser({ - email: 'multi@example.com', - password: 'password123', - name: 'Multi Session User', - }); - - const session1 = await usersService.login({ - email: 'multi@example.com', - password: 'password123', - }); - - const session2 = await usersService.login({ - email: 'multi@example.com', - password: 'password123', - }); - - expect(session1.sessionId).not.toBe(session2.sessionId); - expect(session1.token).not.toBe(session2.token); - }); - }); + // Local login behavior (credential verification, disabled/SSO rejection, + // enumeration hardening, session creation) is covered on the provider path: + // see tests/modules/auth/local-provider.test.ts and authentication-service.test.ts, + // plus audit coverage in tests/modules/audit-log/record-integration.test.ts. + // usersService.login was removed as a redundant parallel implementation. describe('validateSession', () => { it('should return user profile for valid session', async () => { @@ -306,10 +208,7 @@ describe('UsersService', () => { name: 'Validate User', }); - const session = await usersService.login({ - email: 'validate@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); const profile = await usersService.validateSession(session.token); @@ -325,16 +224,13 @@ describe('UsersService', () => { }); it('should return null for expired session', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'expired@example.com', password: 'password123', name: 'Expired User', }); - const session = await usersService.login({ - email: 'expired@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); // Manually expire the session await db @@ -355,10 +251,7 @@ describe('UsersService', () => { name: 'Disabled User', }); - const session = await usersService.login({ - email: 'disabled@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); // Disable the user await db @@ -373,16 +266,13 @@ describe('UsersService', () => { }); it('should delete expired session on validation', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'cleanup@example.com', password: 'password123', name: 'Cleanup User', }); - const session = await usersService.login({ - email: 'cleanup@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); // Manually expire the session await db @@ -406,16 +296,13 @@ describe('UsersService', () => { describe('logout', () => { it('should delete the session', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'logout@example.com', password: 'password123', name: 'Logout User', }); - const session = await usersService.login({ - email: 'logout@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); await usersService.logout(session.token); @@ -513,13 +400,13 @@ describe('UsersService', () => { newPassword: 'newPassword', }); - // Should be able to login with new password - const session = await usersService.login({ + // Should be able to authenticate with the new password + const result = await authenticationService.authenticateWithProvider('local', { email: 'password@example.com', password: 'newPassword', }); - expect(session.token).toBeDefined(); + expect(result.session.token).toBeDefined(); }); it('should throw error when changing password without current password', async () => { @@ -599,10 +486,7 @@ describe('UsersService', () => { name: 'Cascade User', }); - const session = await usersService.login({ - email: 'cascade@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); await usersService.deleteUser(user.id, 'password123'); @@ -610,7 +494,7 @@ describe('UsersService', () => { const dbSession = await db .selectFrom('sessions') .select('id') - .where('id', '=', session.sessionId) + .where('id', '=', session.id) .executeTakeFirst(); expect(dbSession).toBeUndefined(); @@ -625,10 +509,7 @@ describe('UsersService', () => { name: 'Cleanup User', }); - const session = await usersService.login({ - email: 'cleanup@example.com', - password: 'password123', - }); + const session = await createTestSession(user.id); // Expire the session await db @@ -643,16 +524,13 @@ describe('UsersService', () => { }); it('should not delete valid sessions', async () => { - await usersService.createUser({ + const user = await usersService.createUser({ email: 'valid@example.com', password: 'password123', name: 'Valid User', }); - await usersService.login({ - email: 'valid@example.com', - password: 'password123', - }); + await createTestSession(user.id); const deleted = await usersService.cleanupExpiredSessions(); diff --git a/packages/backend/src/tests/queue/jobs/digest-dispatch.test.ts b/packages/backend/src/tests/queue/jobs/digest-dispatch.test.ts new file mode 100644 index 00000000..030e34bc --- /dev/null +++ b/packages/backend/src/tests/queue/jobs/digest-dispatch.test.ts @@ -0,0 +1,141 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import type { IJob } from '../../../queue/abstractions/types.js'; + +vi.mock('@logtide/core', () => ({ + hub: { + captureLog: vi.fn(), + }, +})); + +const mockAdd = vi.fn().mockResolvedValue({ id: 'job_1' }); +vi.mock('../../../queue/connection.js', () => ({ + createQueue: vi.fn(() => ({ + add: mockAdd, + })), +})); + +const mockExecute = vi.fn(); +vi.mock('../../../database/connection.js', () => ({ + db: { + selectFrom: vi.fn().mockReturnThis(), + select: vi.fn().mockReturnThis(), + updateTable: vi.fn().mockReturnThis(), + set: vi.fn().mockReturnThis(), + where: vi.fn().mockReturnThis(), + execute: (...args: unknown[]) => mockExecute(...args), + }, +})); + +import { isConfigDue, processDigestDispatch } from '../../../queue/jobs/digest-dispatch.js'; +import { db } from '../../../database/connection.js'; + +function makeJob(): IJob> { + return { id: 'dispatch_1', name: 'digest-dispatch', data: {} }; +} + +describe('isConfigDue', () => { + // 2026-07-13 is a Monday; 08:30 UTC + const mondayAt8 = new Date('2026-07-13T08:30:00.000Z'); + + const baseDaily = { + frequency: 'daily' as const, + delivery_hour: 8, + delivery_day_of_week: null, + last_sent_at: null, + }; + + it('daily config is due when the current UTC hour matches', () => { + expect(isConfigDue(baseDaily, mondayAt8)).toBe(true); + }); + + it('daily config is not due on a different hour', () => { + expect(isConfigDue({ ...baseDaily, delivery_hour: 9 }, mondayAt8)).toBe(false); + }); + + it('weekly config requires matching UTC day of week', () => { + const weekly = { ...baseDaily, frequency: 'weekly' as const, delivery_day_of_week: 1 }; + expect(isConfigDue(weekly, mondayAt8)).toBe(true); + expect(isConfigDue({ ...weekly, delivery_day_of_week: 2 }, mondayAt8)).toBe(false); + }); + + it('is not due again when last_sent_at is within the double-fire guard', () => { + const sentRecently = { ...baseDaily, last_sent_at: new Date('2026-07-13T08:05:00.000Z') }; + expect(isConfigDue(sentRecently, mondayAt8)).toBe(false); + }); + + it('is due when last_sent_at is older than the guard window', () => { + const sentYesterday = { ...baseDaily, last_sent_at: new Date('2026-07-12T08:30:00.000Z') }; + expect(isConfigDue(sentYesterday, mondayAt8)).toBe(true); + }); +}); + +describe('processDigestDispatch', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.useRealTimers(); + }); + + it('enqueues generation jobs only for due configs and stamps last_sent_at', async () => { + vi.useFakeTimers(); + vi.setSystemTime(new Date('2026-07-13T08:00:30.000Z')); + + mockExecute.mockResolvedValueOnce([ + { + id: 'conf_due', + organization_id: 'org_1', + frequency: 'daily', + delivery_hour: 8, + delivery_day_of_week: null, + last_sent_at: null, + }, + { + id: 'conf_not_due', + organization_id: 'org_2', + frequency: 'daily', + delivery_hour: 20, + delivery_day_of_week: null, + last_sent_at: null, + }, + ]); + // updateTable(...).execute() resolves via the same mockExecute + mockExecute.mockResolvedValue(undefined); + + await processDigestDispatch(makeJob()); + + expect(mockAdd).toHaveBeenCalledTimes(1); + const [jobName, payload, options] = mockAdd.mock.calls[0]; + expect(jobName).toBe('digest-generation'); + expect(payload).toEqual({ + organizationId: 'org_1', + digestConfigId: 'conf_due', + frequency: 'daily', + }); + // hour-keyed jobKey dedupes double enqueues within the same hour + expect(options.jobKey).toBe('digest:conf_due:2026-07-13T08'); + + expect(db.updateTable).toHaveBeenCalledWith('digest_configs'); + vi.useRealTimers(); + }); + + it('does nothing when no config is due', async () => { + vi.useFakeTimers(); + vi.setSystemTime(new Date('2026-07-13T03:00:00.000Z')); + + mockExecute.mockResolvedValueOnce([ + { + id: 'conf_1', + organization_id: 'org_1', + frequency: 'daily', + delivery_hour: 8, + delivery_day_of_week: null, + last_sent_at: null, + }, + ]); + + await processDigestDispatch(makeJob()); + + expect(mockAdd).not.toHaveBeenCalled(); + expect(db.updateTable).not.toHaveBeenCalled(); + vi.useRealTimers(); + }); +}); diff --git a/packages/backend/src/tests/queue/jobs/exception-parsing.test.ts b/packages/backend/src/tests/queue/jobs/exception-parsing.test.ts index 8eaae7d6..905f6591 100644 --- a/packages/backend/src/tests/queue/jobs/exception-parsing.test.ts +++ b/packages/backend/src/tests/queue/jobs/exception-parsing.test.ts @@ -409,6 +409,160 @@ ValueError: Repeated error`; ); }); + it('reopens a resolved error group and flags a regression when it recurs', async () => { + const owner = await createTestUser({ name: 'Owner User' }); + const org = await createTestOrganization({ ownerId: owner.id, name: 'Test Org' }); + const project = await createTestProject({ organizationId: org.id, userId: owner.id }); + + const pythonStackTrace = `Traceback (most recent call last): + File "/app/regress.py", line 10, in + do_something() +ValueError: Regressed error`; + + const log1 = await createTestLog({ projectId: project.id, level: 'error', message: pythonStackTrace, service: 'test-service' }); + const log2 = await createTestLog({ projectId: project.id, level: 'error', message: pythonStackTrace, service: 'test-service' }); + + const mkJob = (logId: string, message: string) => + ({ + data: { + logs: [{ id: logId, message, level: 'error' as const, service: 'test-service' }], + organizationId: org.id, + projectId: project.id, + } as ExceptionParsingJobData, + } as Job); + + await processExceptionParsing(mkJob(log1.id, log1.message)); + + // Operator resolves the group + await db.updateTable('error_groups').set({ status: 'resolved' }).where('organization_id', '=', org.id).execute(); + + vi.clearAllMocks(); + mockState.queueAdd.mockResolvedValue({ id: 'job-id' }); + + // The same error recurs + await processExceptionParsing(mkJob(log2.id, log2.message)); + + expect(mockState.queueAdd).toHaveBeenCalledWith( + 'error-notification', + expect.objectContaining({ isRegression: true }), + expect.any(Object) + ); + + const group = await db + .selectFrom('error_groups') + .select('status') + .where('organization_id', '=', org.id) + .executeTakeFirstOrThrow(); + expect(group.status).toBe('open'); + }); + + describe('auto-merge at ingestion', () => { + async function ingest( + org: string, + project: string, + frames: Array<{ file: string; function: string; line: number }>, + message = 'boom', + type = 'TypeError' + ) { + const log = await createTestLog({ projectId: project, level: 'error', message: `${type}: ${message}`, service: 'svc' }); + const job = { + data: { + logs: [ + { + id: log.id, + message: log.message, + level: 'error' as const, + service: 'svc', + metadata: { exception: { type, message, language: 'nodejs', stacktrace: frames } }, + }, + ], + organizationId: org, + projectId: project, + } as ExceptionParsingJobData, + } as Job; + await processExceptionParsing(job); + } + + async function groupCount(org: string): Promise { + const rows = await db.selectFrom('error_groups').select('id').where('organization_id', '=', org).execute(); + return rows.length; + } + + it('folds a stack that differs only in deeper app frames into one group', async () => { + const owner = await createTestUser({ name: 'Owner' }); + const org = await createTestOrganization({ ownerId: owner.id, name: 'Org' }); + const project = await createTestProject({ organizationId: org.id, userId: owner.id }); + + // Same top app frame + message, different SECOND app frame -> different + // fingerprint (would split) but same merge key (auto-folds). + await ingest(org.id, project.id, [ + { file: '/app/src/charge.js', function: 'charge', line: 10 }, + { file: '/app/src/handlerA.js', function: 'handleA', line: 5 }, + ]); + await ingest(org.id, project.id, [ + { file: '/app/src/charge.js', function: 'charge', line: 10 }, + { file: '/app/src/handlerB.js', function: 'handleB', line: 7 }, + ]); + + expect(await groupCount(org.id)).toBe(1); + }); + + it('keeps errors from different throw sites in separate groups', async () => { + const owner = await createTestUser({ name: 'Owner' }); + const org = await createTestOrganization({ ownerId: owner.id, name: 'Org' }); + const project = await createTestProject({ organizationId: org.id, userId: owner.id }); + + await ingest(org.id, project.id, [{ file: '/app/src/charge.js', function: 'charge', line: 10 }]); + await ingest(org.id, project.id, [{ file: '/app/src/refund.js', function: 'refund', line: 20 }]); + + expect(await groupCount(org.id)).toBe(2); + }); + + it('folds messages that differ only by dynamic values', async () => { + const owner = await createTestUser({ name: 'Owner' }); + const org = await createTestOrganization({ ownerId: owner.id, name: 'Org' }); + const project = await createTestProject({ organizationId: org.id, userId: owner.id }); + + await ingest( + org.id, + project.id, + [ + { file: '/app/src/net.js', function: 'call', line: 3 }, + { file: '/app/src/a.js', function: 'a', line: 1 }, + ], + 'Timeout after 5023ms' + ); + await ingest( + org.id, + project.id, + [ + { file: '/app/src/net.js', function: 'call', line: 3 }, + { file: '/app/src/b.js', function: 'b', line: 2 }, + ], + 'Timeout after 812ms' + ); + + expect(await groupCount(org.id)).toBe(1); + }); + + it('does not compute a merge key for a library-only exception', async () => { + const owner = await createTestUser({ name: 'Owner' }); + const org = await createTestOrganization({ ownerId: owner.id, name: 'Org' }); + const project = await createTestProject({ organizationId: org.id, userId: owner.id }); + + await ingest(org.id, project.id, [ + { file: '/app/node_modules/pg/lib/client.js', function: 'query', line: 1 }, + ]); + + const group = await db + .selectFrom('error_groups') + .select('merge_key') + .where('organization_id', '=', org.id) + .executeTakeFirstOrThrow(); + expect(group.merge_key).toBeNull(); + }); + }); + it('should handle empty logs array', async () => { const owner = await createTestUser({ name: 'Owner User' }); const org = await createTestOrganization({ ownerId: owner.id, name: 'Test Org' }); diff --git a/packages/backend/src/tests/setup.ts b/packages/backend/src/tests/setup.ts index 5e5ed51d..34bf8a47 100644 --- a/packages/backend/src/tests/setup.ts +++ b/packages/backend/src/tests/setup.ts @@ -25,7 +25,25 @@ beforeAll(async () => { console.log('Running database migrations...'); await migrateToLatest(); console.log('Database migrations completed'); - + + // Re-seed the 'local' auth provider (migration 010 seeds it, but tracked + // migrations never re-run, so a test file that deletes the row would + // otherwise leave the whole database without local login forever). + await db + .insertInto('auth_providers') + .values({ + type: 'local', + name: 'Email & Password', + slug: 'local', + enabled: true, + is_default: true, + display_order: 0, + icon: 'mail', + config: {}, + }) + .onConflict((oc) => oc.column('slug').doNothing()) + .execute(); + try { const dbUrl = process.env.DATABASE_URL; if (dbUrl) { diff --git a/packages/backend/src/utils/internal-logger.ts b/packages/backend/src/utils/internal-logger.ts index 59d62287..c9f15559 100644 --- a/packages/backend/src/utils/internal-logger.ts +++ b/packages/backend/src/utils/internal-logger.ts @@ -58,7 +58,7 @@ export async function initializeInternalLogging(): Promise { dsn, service: process.env.SERVICE_NAME || 'logtide-backend', environment: process.env.NODE_ENV || 'development', - release: process.env.npm_package_version || '1.1.0', + release: process.env.npm_package_version || '1.2.0', batchSize: 5, // Smaller batch for internal logs to see them faster flushInterval: 5000, maxBufferSize: 1000, diff --git a/packages/backend/src/worker.ts b/packages/backend/src/worker.ts index e7efc061..b58efb5b 100644 --- a/packages/backend/src/worker.ts +++ b/packages/backend/src/worker.ts @@ -25,8 +25,8 @@ import { enrichmentService } from './modules/siem/enrichment-service.js'; import { retentionService } from './modules/retention/index.js'; import { auditLogService } from './modules/audit-log/index.js'; import { sigmaSyncService } from './modules/sigma/sync-service.js'; -// Email digest feature is incomplete and disabled for 1.0.0-beta (see #154 below). -// import { digestScheduler } from './modules/digests/scheduler.js'; +import { digestScheduler } from './modules/digests/scheduler.js'; +import { processDigestDispatch } from './queue/jobs/digest-dispatch.js'; import { initializeWorkerLogging, shutdownInternalLogging, isInternalLoggingEnabled } from './utils/internal-logger.js'; import { hub } from '@logtide/core'; import { reservoir, reservoirReady } from './database/reservoir.js'; @@ -92,12 +92,13 @@ const pipelineWorker = createWorker('log-pipeline', async (j const digestWorker = createWorker('digest-generation', async (job) => { await processDigestGeneration(job); }); -// The email digest reports feature (#154) is incomplete: it only summarizes log -// volume (no error groups, security or uptime sections, no HTML layout) and has no -// UI to configure it. The scheduler is intentionally left unregistered so no partial -// digests are ever sent in the 1.0.0-beta release. The worker, service, database -// schema and tests stay in place so the remaining scope can be finished under #154. -// await digestScheduler.registerAllDigests(); +// Hourly dispatch cron scans digest configs and enqueues due digests (#154). +// A single static cron (instead of one per org) keeps config CRUD live on both +// queue backends: graphile-worker cron items cannot change while the runner is up. +const digestDispatchWorker = createWorker>('digest-dispatch', async (job) => { + await processDigestDispatch(job); +}); +await digestScheduler.registerDispatchCron(); // Create worker for outbound webhook delivery (#218) const webhookDeliveryWorker = createWorker('webhook-delivery', async (job) => { @@ -335,6 +336,15 @@ digestWorker.on('failed', (job, err) => { } }); +digestDispatchWorker.on('failed', (job, err) => { + if (isInternalLoggingEnabled()) { + hub.captureLog('error', `Digest dispatch job failed: ${err.message}`, { + error: { name: err.name, message: err.message, stack: err.stack }, + jobId: job?.id, + }); + } +}); + webhookDeliveryWorker.on('failed', (job, err) => { console.error(`Webhook delivery job ${job?.id} failed:`, err); if (isInternalLoggingEnabled()) { @@ -912,6 +922,7 @@ async function gracefulShutdown(signal: string) { await monitorNotificationWorker.close(); await pipelineWorker.close(); await digestWorker.close(); + await digestDispatchWorker.close(); await webhookDeliveryWorker.close(); await receiverEventsWorker.close(); console.log('[Worker] Workers closed'); diff --git a/packages/frontend/package.json b/packages/frontend/package.json index fd933557..7e7b7107 100644 --- a/packages/frontend/package.json +++ b/packages/frontend/package.json @@ -1,6 +1,6 @@ { "name": "@logtide/frontend", - "version": "1.1.0", + "version": "1.2.0", "private": true, "description": "LogTide Frontend Dashboard", "type": "module", @@ -20,13 +20,14 @@ "sourcemaps:upload": "logtide sourcemaps upload build/client --release $npm_package_version" }, "dependencies": { + "@internationalized/date": "^3.12.2", "@logtide/core": "0.7.0", "@logtide/shared": "workspace:*", "@logtide/sveltekit": "0.7.0", "@lucide/svelte": "^0.482.0", "@tanstack/svelte-table": "^8.21.3", "canvas-confetti": "^1.9.4", - "echarts": "^6.0.0", + "echarts": "^6.1.0", "leaflet": "^1.9.4", "shiki": "^3.18.0", "svelte-dnd-action": "^0.9.69", diff --git a/packages/frontend/src/hooks.client.ts b/packages/frontend/src/hooks.client.ts index d5e59755..f1727148 100644 --- a/packages/frontend/src/hooks.client.ts +++ b/packages/frontend/src/hooks.client.ts @@ -9,7 +9,7 @@ if (dsn) { dsn, service: 'logtide-frontend-client', environment: env.PUBLIC_NODE_ENV || 'production', - release: env.PUBLIC_APP_VERSION || '1.1.0', + release: env.PUBLIC_APP_VERSION || '1.2.0', debug: env.PUBLIC_NODE_ENV === 'development', browser: { // Core Web Vitals (LCP, INP, CLS, TTFB) diff --git a/packages/frontend/src/hooks.server.ts b/packages/frontend/src/hooks.server.ts index b208ad34..3e890f2a 100644 --- a/packages/frontend/src/hooks.server.ts +++ b/packages/frontend/src/hooks.server.ts @@ -82,7 +82,7 @@ export const handle = dsn dsn, service: 'logtide-frontend', environment: privateEnv?.NODE_ENV || 'production', - release: process.env.npm_package_version || '1.1.0', }) as unknown as Handle, + release: process.env.npm_package_version || '1.2.0', }) as unknown as Handle, requestLogHandle, configHandle ) diff --git a/packages/frontend/src/lib/api/admin.ts b/packages/frontend/src/lib/api/admin.ts index 75880776..472c0bdf 100644 --- a/packages/frontend/src/lib/api/admin.ts +++ b/packages/frontend/src/lib/api/admin.ts @@ -390,6 +390,9 @@ export interface IngestionHealthStats { detectionEnqueueFailed: number; exceptionEnqueueFailed: number; identifierFailed: number; + timestampSkew: number; + spanTimestampSkew: number; + metricTimestampSkew: number; }; enrichment: { ipReputation: EnrichmentSourceStatus & { totalIps: number }; diff --git a/packages/frontend/src/lib/api/alerts.ts b/packages/frontend/src/lib/api/alerts.ts index 6bf4e48e..1438f469 100644 --- a/packages/frontend/src/lib/api/alerts.ts +++ b/packages/frontend/src/lib/api/alerts.ts @@ -5,6 +5,23 @@ import type { LogLevel, MetadataFilter, MetadataFilterInput } from '@logtide/sha export type AlertType = 'threshold' | 'rate_of_change'; export type BaselineType = 'same_time_yesterday' | 'same_day_last_week' | 'rolling_7d_avg' | 'percentile_p95'; +/** Optional values to prefill the alert builder (create-from-error, duplicate). */ +export interface AlertBuilderPrefill { + name?: string; + service?: string | null; + levels?: string[]; + threshold?: number; + timeWindow?: number; + alertType?: AlertType; + baselineType?: BaselineType | null; + deviationMultiplier?: number | null; + minBaselineValue?: number | null; + cooldownMinutes?: number | null; + sustainedMinutes?: number | null; + metadataFilters?: MetadataFilterInput[]; + channelIds?: string[]; +} + export interface AlertRule { id: string; organizationId: string; diff --git a/packages/frontend/src/lib/api/auth.ts b/packages/frontend/src/lib/api/auth.ts index 3a016b41..48c35387 100644 --- a/packages/frontend/src/lib/api/auth.ts +++ b/packages/frontend/src/lib/api/auth.ts @@ -23,6 +23,13 @@ export interface AuthResponse { }; } +// Registration can succeed (201) without a session when the backend's +// auto-login fails transiently; the account is then recoverable via /login. +export interface RegisterResponse { + user: AuthResponse['user']; + session?: AuthResponse['session']; +} + export interface ErrorResponse { error: string; details?: any; @@ -53,7 +60,7 @@ export interface AuthConfig { } export class AuthAPI { - async register(input: RegisterInput): Promise { + async register(input: RegisterInput): Promise { const response = await fetch(`${getApiBaseUrl()}/auth/register`, { method: 'POST', headers: { diff --git a/packages/frontend/src/lib/api/digests.ts b/packages/frontend/src/lib/api/digests.ts new file mode 100644 index 00000000..8d3c112b --- /dev/null +++ b/packages/frontend/src/lib/api/digests.ts @@ -0,0 +1,122 @@ +import { getApiUrl } from '$lib/config'; +import { getAuthToken } from '$lib/utils/auth'; + +export type DigestFrequency = 'daily' | 'weekly'; + +export interface DigestConfig { + id: string; + frequency: DigestFrequency; + delivery_hour: number; + delivery_day_of_week: number | null; + enabled: boolean; + last_sent_at: string | null; + created_at: string; + updated_at: string; +} + +export interface DigestRecipient { + id: string; + email: string; + user_id: string | null; + subscribed: boolean; + created_at: string; +} + +export interface DigestConfigInput { + frequency: DigestFrequency; + deliveryHour: number; + deliveryDayOfWeek?: number | null; + enabled: boolean; +} + +async function fetchWithAuth(url: string, options: RequestInit = {}): Promise { + const token = getAuthToken(); + const headers: HeadersInit = { + ...(options.body ? { 'Content-Type': 'application/json' } : {}), + ...(token ? { Authorization: `Bearer ${token}` } : {}), + ...(options.headers || {}), + }; + + return fetch(url, { + ...options, + headers, + credentials: 'include', + }); +} + +async function unwrapError(response: Response, fallback: string): Promise { + const error = await response.json().catch(() => ({ error: fallback })); + throw new Error(error.error || fallback); +} + +export const digestsAPI = { + async getConfig( + organizationId: string + ): Promise<{ config: DigestConfig | null; recipients: DigestRecipient[] }> { + const response = await fetchWithAuth( + `${getApiUrl()}/api/v1/digests/config?organizationId=${organizationId}` + ); + if (!response.ok) await unwrapError(response, 'Failed to load digest settings'); + return response.json(); + }, + + async saveConfig(organizationId: string, input: DigestConfigInput): Promise { + const response = await fetchWithAuth( + `${getApiUrl()}/api/v1/digests/config?organizationId=${organizationId}`, + { method: 'PUT', body: JSON.stringify(input) } + ); + if (!response.ok) await unwrapError(response, 'Failed to save digest settings'); + return (await response.json()).config; + }, + + async deleteConfig(organizationId: string): Promise { + const response = await fetchWithAuth( + `${getApiUrl()}/api/v1/digests/config?organizationId=${organizationId}`, + { method: 'DELETE' } + ); + if (!response.ok) await unwrapError(response, 'Failed to delete digest settings'); + }, + + async addRecipient(organizationId: string, email: string): Promise { + const response = await fetchWithAuth( + `${getApiUrl()}/api/v1/digests/recipients?organizationId=${organizationId}`, + { method: 'POST', body: JSON.stringify({ email }) } + ); + if (!response.ok) await unwrapError(response, 'Failed to add recipient'); + return (await response.json()).recipient; + }, + + async removeRecipient(organizationId: string, recipientId: string): Promise { + const response = await fetchWithAuth( + `${getApiUrl()}/api/v1/digests/recipients/${recipientId}?organizationId=${organizationId}`, + { method: 'DELETE' } + ); + if (!response.ok) await unwrapError(response, 'Failed to remove recipient'); + }, + + async resubscribeRecipient( + organizationId: string, + recipientId: string + ): Promise { + const response = await fetchWithAuth( + `${getApiUrl()}/api/v1/digests/recipients/${recipientId}/resubscribe?organizationId=${organizationId}`, + { method: 'POST' } + ); + if (!response.ok) await unwrapError(response, 'Failed to resubscribe recipient'); + return (await response.json()).recipient; + }, + + /** + * Public one-click unsubscribe. No auth: the token from the email link is + * the credential. + */ + async unsubscribe(token: string): Promise<{ success: boolean; email: string }> { + const response = await fetch(`${getApiUrl()}/api/v1/digests/unsubscribe`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ token }), + }); + if (!response.ok) await unwrapError(response, 'Invalid or expired unsubscribe link'); + return response.json(); + }, +}; diff --git a/packages/frontend/src/lib/api/exceptions.ts b/packages/frontend/src/lib/api/exceptions.ts index 82319db7..db2f4d7e 100644 --- a/packages/frontend/src/lib/api/exceptions.ts +++ b/packages/frontend/src/lib/api/exceptions.ts @@ -204,6 +204,58 @@ export async function updateErrorGroupStatus( return response.json(); } +export interface DuplicateErrorGroup { + id: string; + occurrenceCount: number; + firstSeen: string; + lastSeen: string; +} + +export async function getDuplicateErrorGroups( + groupId: string, + organizationId: string +): Promise { + const token = getAuthToken(); + const searchParams = new URLSearchParams({ organizationId }); + + const response = await fetch( + `${getApiUrl()}/api/v1/error-groups/${groupId}/duplicates?${searchParams}`, + { headers: { Authorization: `Bearer ${token}` } } + ); + + if (!response.ok) { + const error = await response.json().catch(() => ({})); + throw new Error(error.error || `Failed to load duplicate groups (HTTP ${response.status})`); + } + + const data = await response.json(); + return data.duplicates ?? []; +} + +export async function mergeErrorGroups( + groupId: string, + sourceIds: string[], + organizationId: string +): Promise { + const token = getAuthToken(); + + const response = await fetch(`${getApiUrl()}/api/v1/error-groups/${groupId}/merge`, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + Authorization: `Bearer ${token}`, + }, + body: JSON.stringify({ organizationId, sourceIds }), + }); + + if (!response.ok) { + const error = await response.json().catch(() => ({})); + throw new Error(error.error || `Failed to merge error groups (HTTP ${response.status})`); + } + + return response.json(); +} + export async function getErrorGroupTrend(params: { groupId: string; organizationId: string; diff --git a/packages/frontend/src/lib/api/fetch-interceptor.ts b/packages/frontend/src/lib/api/fetch-interceptor.ts index 32dd038a..6dc0e0ed 100644 --- a/packages/frontend/src/lib/api/fetch-interceptor.ts +++ b/packages/frontend/src/lib/api/fetch-interceptor.ts @@ -38,6 +38,19 @@ function isAuthEndpoint(url: string): boolean { return url.includes('/auth/'); } +// Ingestion / telemetry endpoints authenticate with an API key or DSN, not the +// session. A 401 here means a bad or expired telemetry key (e.g. the client-side +// SDK in hooks.client.ts, or an inbound receiver), NOT an expired user session, +// so it must never log the user out. Otherwise a misconfigured browser SDK key +// bounces the user to /login on every page view. +function isTelemetryEndpoint(url: string): boolean { + return ( + url.includes('/api/v1/ingest') || + url.includes('/v1/otlp/') || + url.includes('/api/v1/receivers/') + ); +} + function handleUnauthorized(): void { if (handling) return; handling = true; @@ -74,7 +87,8 @@ export function installAuthFetchInterceptor(): void { response.status === 401 && getAuthToken() && isApiRequest(urlOf(input)) && - !isAuthEndpoint(urlOf(input)) + !isAuthEndpoint(urlOf(input)) && + !isTelemetryEndpoint(urlOf(input)) ) { handleUnauthorized(); } diff --git a/packages/frontend/src/lib/api/projects.ts b/packages/frontend/src/lib/api/projects.ts index 5758b21d..16a422af 100644 --- a/packages/frontend/src/lib/api/projects.ts +++ b/packages/frontend/src/lib/api/projects.ts @@ -16,6 +16,19 @@ export interface UpdateProjectInput { statusPagePassword?: string; } +export interface ProjectSkewHealth { + count24h: number; + maxPastMs: number; + maxFutureMs: number; + lastSeenAt: string; +} + +export interface ProjectSkewHealthMap { + logs: ProjectSkewHealth | null; + spans: ProjectSkewHealth | null; + metrics: ProjectSkewHealth | null; +} + export class ProjectsAPI { constructor(private getToken: () => string | null) {} @@ -81,6 +94,16 @@ export class ProjectsAPI { return response.json(); } + async getIngestionHealth(id: string): Promise<{ skew: ProjectSkewHealthMap }> { + const response = await this.request(`/projects/${id}/ingestion-health`); + + if (!response.ok) { + throw new Error('Failed to fetch project ingestion health'); + } + + return response.json(); + } + async getProject(id: string): Promise<{ project: Project }> { const response = await this.request(`/projects/${id}`); diff --git a/packages/frontend/src/lib/components/CommandPalette.svelte b/packages/frontend/src/lib/components/CommandPalette.svelte index ad4a63cd..45a56c4f 100644 --- a/packages/frontend/src/lib/components/CommandPalette.svelte +++ b/packages/frontend/src/lib/components/CommandPalette.svelte @@ -20,6 +20,11 @@ import Moon from '@lucide/svelte/icons/moon'; let open = $state(false); + let query = $state(''); + + let trimmedQuery = $derived(query.trim()); + // Trace/span IDs are 16 or 32 lowercase hex characters. + let looksLikeTraceId = $derived(/^[0-9a-f]{16}$|^[0-9a-f]{32}$/i.test(trimmedQuery)); $effect(() => { const unsubscribe = shortcutsStore.subscribe((s) => { @@ -32,6 +37,7 @@ if (newOpen) { shortcutsStore.openCommandPalette(); } else { + query = ''; shortcutsStore.closeCommandPalette(); } } @@ -51,6 +57,7 @@ function navigate(href: string) { goto(href); + query = ''; shortcutsStore.closeCommandPalette(); } @@ -61,10 +68,31 @@ - + No results found. + {#if trimmedQuery} + + {#if looksLikeTraceId} + navigate(`/dashboard/traces?traceId=${encodeURIComponent(trimmedQuery)}`)}> + + Open trace {trimmedQuery} + + {/if} + navigate(`/dashboard/search?q=${encodeURIComponent(trimmedQuery)}`)}> + + Search logs for "{trimmedQuery}" + + navigate(`/dashboard/errors?search=${encodeURIComponent(trimmedQuery)}`)}> + + Search errors for "{trimmedQuery}" + + + + + {/if} + {#each navItems as item} {@const Icon = item.icon} diff --git a/packages/frontend/src/lib/components/CreateAlertDialog.svelte b/packages/frontend/src/lib/components/CreateAlertDialog.svelte index 1082f4ec..1c2000c2 100644 --- a/packages/frontend/src/lib/components/CreateAlertDialog.svelte +++ b/packages/frontend/src/lib/components/CreateAlertDialog.svelte @@ -1,5 +1,5 @@ diff --git a/packages/frontend/src/lib/components/EmptyMetrics.svelte b/packages/frontend/src/lib/components/EmptyMetrics.svelte new file mode 100644 index 00000000..3cc77624 --- /dev/null +++ b/packages/frontend/src/lib/components/EmptyMetrics.svelte @@ -0,0 +1,201 @@ + + +
+ +
+
+ +
+

No Metrics Yet

+

+ Send OTLP metrics from your application to explore counters, gauges and histograms here. +

+
+ + +
+ + + + +

Get API Key

+

Generate credentials

+ + + + + + + +

OTLP Docs

+

Setup guide

+ + + + + + + +

OpenTelemetry

+

Official docs

+ + + +
+ + + + +
+ + Send Metrics with OpenTelemetry +
+ + Configure your application to export OTLP metrics to LogTide + + + + + + Node.js + Python + Go + + + {#each Object.entries(codeExamples) as [key, code]} + +
+ 
{code}
+ +
+ + {/each} + + + + + +
+ + +

Once metrics arrive, you can:

+
    +
  • + + Explore metrics by name and service +
    • +
  • + + Track golden signals at a glance +
    • +
  • + + Visualize time series over any range +
    • +
  • + + Spot trends and anomalies early +
    • +
+ + +
+
diff --git a/packages/frontend/src/lib/components/EmptyTraces.svelte b/packages/frontend/src/lib/components/EmptyTraces.svelte index f9eea66f..7c642901 100644 --- a/packages/frontend/src/lib/components/EmptyTraces.svelte +++ b/packages/frontend/src/lib/components/EmptyTraces.svelte @@ -65,9 +65,10 @@ import ( "go.opentelemetry.io/otel/sdk/trace" ) -exporter, _ := otlptracehttp.New(ctx, +exporter, _ := otlptracehttp.New(context.Background(), otlptracehttp.WithEndpoint("${apiUrlValue.replace('https://', '').replace('http://', '')}"), otlptracehttp.WithURLPath("/v1/otlp/traces"), + otlptracehttp.WithInsecure(), // remove for HTTPS otlptracehttp.WithHeaders(map[string]string{ "X-API-Key": "YOUR_API_KEY", }), diff --git a/packages/frontend/src/lib/components/Footer.svelte b/packages/frontend/src/lib/components/Footer.svelte index 38c5aadc..aae173e2 100644 --- a/packages/frontend/src/lib/components/Footer.svelte +++ b/packages/frontend/src/lib/components/Footer.svelte @@ -1,7 +1,7 @@ diff --git a/packages/frontend/src/lib/components/ThemeToggle.svelte b/packages/frontend/src/lib/components/ThemeToggle.svelte index 2b6483c6..df816a52 100644 --- a/packages/frontend/src/lib/components/ThemeToggle.svelte +++ b/packages/frontend/src/lib/components/ThemeToggle.svelte @@ -1,27 +1,49 @@ - + + + + + + {#each options as opt} + themeStore.setPreference(opt.value)}> + + {opt.label} + {#if current === opt.value} + + {/if} + + {/each} + + diff --git a/packages/frontend/src/lib/components/TimeRangeButtons.svelte b/packages/frontend/src/lib/components/TimeRangeButtons.svelte new file mode 100644 index 00000000..319b3219 --- /dev/null +++ b/packages/frontend/src/lib/components/TimeRangeButtons.svelte @@ -0,0 +1,36 @@ + + +
+ {#each options as opt (opt.value)} + + {/each} +
diff --git a/packages/frontend/src/lib/components/TimeRangePicker.svelte b/packages/frontend/src/lib/components/TimeRangePicker.svelte index d7ac9855..5811026e 100644 --- a/packages/frontend/src/lib/components/TimeRangePicker.svelte +++ b/packages/frontend/src/lib/components/TimeRangePicker.svelte @@ -3,6 +3,9 @@ import Input from "$lib/components/ui/input/input.svelte"; import Label from "$lib/components/ui/label/label.svelte"; import Clock from "@lucide/svelte/icons/clock"; + import { RangeCalendar } from "$lib/components/ui/range-calendar"; + import { CalendarDate, type DateValue } from "@internationalized/date"; + import type { DateRange } from "bits-ui"; export type TimeRangeType = "last_hour" | "last_24h" | "last_7d" | "custom"; @@ -144,12 +147,49 @@ emitChange(); } - function handleCustomTimeChange() { + function emitChange() { + onchange?.(getTimeRange()); + } + + // ── Custom range as calendar (date) + time input (hour) ────────────────── + // customFromTime / customToTime stay the source of truth ("YYYY-MM-DDTHH:mm"); + // the calendar drives the date part and the time inputs the hour part. + const pad = (n: number) => String(n).padStart(2, "0"); + const datePart = (s: string) => (s ? s.slice(0, 10) : ""); + const timePart = (s: string) => (s && s.length >= 16 ? s.slice(11, 16) : "00:00"); + const combine = (d: string, t: string) => (d ? `${d}T${t || "00:00"}` : ""); + + function toCalendarDate(s: string): DateValue | undefined { + const d = datePart(s); + if (!d) return undefined; + const [y, m, day] = d.split("-").map(Number); + if (!y || !m || !day) return undefined; + return new CalendarDate(y, m, day); + } + + function dateStr(d: DateValue): string { + return `${d.year}-${pad(d.month)}-${pad(d.day)}`; + } + + let calendarValue = $derived({ + start: toCalendarDate(customFromTime), + end: toCalendarDate(customToTime), + }); + + function handleRangeChange(range: DateRange | undefined) { + if (!range) return; + if (range.start) customFromTime = combine(dateStr(range.start), timePart(customFromTime)); + if (range.end) customToTime = combine(dateStr(range.end), timePart(customToTime)); emitChange(); } - function emitChange() { - onchange?.(getTimeRange()); + function handleTimeInput(which: "from" | "to", value: string) { + if (which === "from") { + customFromTime = combine(datePart(customFromTime), value); + } else { + customToTime = combine(datePart(customToTime), value); + } + emitChange(); } @@ -190,24 +230,33 @@ {#if timeRangeType === "custom"} -
-
- - +
+
-
- - +
+
+ + handleTimeInput("from", (e.currentTarget as HTMLInputElement).value)} + /> +
+
+ + handleTimeInput("to", (e.currentTarget as HTMLInputElement).value)} + /> +
{/if} diff --git a/packages/frontend/src/lib/components/dashboard/StatsCard.svelte b/packages/frontend/src/lib/components/dashboard/StatsCard.svelte index d0ef9850..a09f32bc 100644 --- a/packages/frontend/src/lib/components/dashboard/StatsCard.svelte +++ b/packages/frontend/src/lib/components/dashboard/StatsCard.svelte @@ -10,11 +10,22 @@ trend?: { value: number; isPositive: boolean; + /** Suffix after the number. Defaults to '%'. Pass '' for a plain count, ' pp' for percentage points. */ + unit?: string; + /** Trailing label. Defaults to 'from last period'. */ + label?: string; }; onclick?: () => void; } let { title, value, icon: Icon, description, trend, onclick }: Props = $props(); + + function formatTrend(t: NonNullable): string { + const unit = t.unit ?? '%'; + const decimals = unit === '' ? 0 : 2; + const sign = t.isPositive ? '+' : ''; + return `${sign}${t.value.toFixed(decimals)}${unit} ${t.label ?? 'from last period'}`; + } {#if onclick} @@ -36,7 +47,7 @@ {/if} {#if trend}

- {trend.isPositive ? '+' : ''}{trend.value.toFixed(2)}% from last period + {formatTrend(trend)}

{/if} diff --git a/packages/frontend/src/lib/components/metrics/OverviewPanel.svelte b/packages/frontend/src/lib/components/metrics/OverviewPanel.svelte index ec49c956..d88c1050 100644 --- a/packages/frontend/src/lib/components/metrics/OverviewPanel.svelte +++ b/packages/frontend/src/lib/components/metrics/OverviewPanel.svelte @@ -1,9 +1,9 @@ {#if displayServices.length === 0} -
- -

No metrics found

-

Start sending OTLP metrics to see them here

-
+ {:else} {#each displayServices as service, si}
diff --git a/packages/frontend/src/lib/components/metrics/ServiceSelector.svelte b/packages/frontend/src/lib/components/metrics/ServiceSelector.svelte index 02065a6c..f80c3fa8 100644 --- a/packages/frontend/src/lib/components/metrics/ServiceSelector.svelte +++ b/packages/frontend/src/lib/components/metrics/ServiceSelector.svelte @@ -1,6 +1,6 @@ + +{#if show && skew} + + + {TITLE[signal]} + + {#if signal === 'logs'} +

+ In the last 24 hours, {skew.count24h.toLocaleString('en-US')} + {noun} arrived with a timestamp up to {direction}. + They are stored and searchable, but threshold alert rules only count logs inside their + time window, so these logs cannot trigger an alert. +

+

+ This usually means the shipper is sending a time field that does not match + the current instant. Omitting the field entirely makes LogTide use the server ingestion + time instead. +

+ {:else if signal === 'spans'} +

+ In the last 24 hours, {skew.count24h.toLocaleString('en-US')} + {noun} ended with a timestamp up to {direction}. + They are stored, but they fall outside the time windows trace views use, so these + spans will not appear there for that period. +

+

+ This usually means a clock or exporter timestamp problem on the sending host. +

+ {:else} +

+ In the last 24 hours, {skew.count24h.toLocaleString('en-US')} + {noun} arrived with a timestamp up to {direction}. + They are stored, but dashboards only show data points inside their time window, so + these metrics will not appear in dashboards for that period. +

+

+ This usually means a clock or exporter timestamp problem on the sending host. +

+ {/if} +

+ Most recent at {lastSeen}. +

+ + +{/if} diff --git a/packages/frontend/src/lib/components/projects/IngestionSkewBanner.test.ts b/packages/frontend/src/lib/components/projects/IngestionSkewBanner.test.ts new file mode 100644 index 00000000..66f32343 --- /dev/null +++ b/packages/frontend/src/lib/components/projects/IngestionSkewBanner.test.ts @@ -0,0 +1,134 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { render, screen } from '@testing-library/svelte'; +import IngestionSkewBanner from './IngestionSkewBanner.svelte'; + +describe('IngestionSkewBanner', () => { + beforeEach(() => { + vi.useFakeTimers(); + vi.setSystemTime(new Date('2026-07-17T09:00:00.000Z')); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + it('renders nothing when there is no skew', () => { + const { container } = render(IngestionSkewBanner, { props: { skew: null, signal: 'logs' } }); + expect(container.querySelector('[role="alert"]')).toBeNull(); + }); + + it('renders nothing when the count is zero', () => { + const { container } = render(IngestionSkewBanner, { + props: { + skew: { count24h: 0, maxPastMs: 0, maxFutureMs: 0, lastSeenAt: '2026-07-17T09:00:00.000Z' }, + signal: 'logs', + }, + }); + expect(container.querySelector('[role="alert"]')).toBeNull(); + }); + + it('reports past skew with a localized count and an hour figure', () => { + render(IngestionSkewBanner, { + props: { + skew: { count24h: 1234, maxPastMs: 97200000, maxFutureMs: 0, lastSeenAt: '2026-07-17T09:00:00.000Z' }, + signal: 'logs', + }, + }); + + expect(screen.getByRole('alert')).toBeInTheDocument(); + expect(screen.getByText(/1,234/)).toBeInTheDocument(); + expect(screen.getByText(/27 hours in the past/)).toBeInTheDocument(); + }); + + it('reports future skew', () => { + render(IngestionSkewBanner, { + props: { + skew: { count24h: 5, maxPastMs: 0, maxFutureMs: 3600000, lastSeenAt: '2026-07-17T09:00:00.000Z' }, + signal: 'logs', + }, + }); + + expect(screen.getByText(/1 hour ahead of the server clock/)).toBeInTheDocument(); + }); + + it('renders the most recent skew time as a relative phrase (plural minutes)', () => { + render(IngestionSkewBanner, { + props: { + // 5 minutes before the fixed system time. + skew: { count24h: 5, maxPastMs: 3600000, maxFutureMs: 0, lastSeenAt: '2026-07-17T08:55:00.000Z' }, + signal: 'logs', + }, + }); + + expect(screen.getByText(/Most recent at 5 minutes ago\./)).toBeInTheDocument(); + }); + + it('renders the most recent skew time with singular minute', () => { + render(IngestionSkewBanner, { + props: { + // 1 minute before the fixed system time. + skew: { count24h: 5, maxPastMs: 3600000, maxFutureMs: 0, lastSeenAt: '2026-07-17T08:59:00.000Z' }, + signal: 'logs', + }, + }); + + expect(screen.getByText(/Most recent at 1 minute ago\./)).toBeInTheDocument(); + }); + + it('renders the most recent skew time in hours', () => { + render(IngestionSkewBanner, { + props: { + // 2 hours before the fixed system time. + skew: { count24h: 5, maxPastMs: 3600000, maxFutureMs: 0, lastSeenAt: '2026-07-17T07:00:00.000Z' }, + signal: 'logs', + }, + }); + + expect(screen.getByText(/Most recent at 2 hours ago\./)).toBeInTheDocument(); + }); + + describe('per-signal copy', () => { + const skew = { + count24h: 3, + maxPastMs: 97200000, + maxFutureMs: 0, + lastSeenAt: '2026-07-17T09:00:00.000Z', + }; + + it('logs: keeps the alert-consequence copy', () => { + render(IngestionSkewBanner, { props: { skew, signal: 'logs' } }); + + expect(screen.getByText('Logs are arriving with an out-of-range timestamp')).toBeInTheDocument(); + expect(screen.getByText(/cannot trigger an alert/)).toBeInTheDocument(); + expect(screen.getByText(/Omitting the field entirely/)).toBeInTheDocument(); + }); + + it('spans: talks about trace views, not alerts or an omittable field', () => { + const { container } = render(IngestionSkewBanner, { props: { skew, signal: 'spans' } }); + + expect(screen.getByText('Spans are arriving with an out-of-range timestamp')).toBeInTheDocument(); + expect(screen.getByText(/fall outside the time windows trace views use/)).toBeInTheDocument(); + expect(screen.getByText(/clock or exporter timestamp problem/)).toBeInTheDocument(); + expect(container.textContent).not.toMatch(/cannot trigger an alert/); + expect(container.textContent).not.toMatch(/Omitting the field entirely/); + }); + + it('metrics: talks about dashboards, not alerts or an omittable field', () => { + const { container } = render(IngestionSkewBanner, { props: { skew, signal: 'metrics' } }); + + expect(screen.getByText('Metrics are arriving with an out-of-range timestamp')).toBeInTheDocument(); + expect(screen.getByText(/will not appear in dashboards/)).toBeInTheDocument(); + expect(screen.getByText(/clock or exporter timestamp problem/)).toBeInTheDocument(); + expect(container.textContent).not.toMatch(/cannot trigger an alert/); + expect(container.textContent).not.toMatch(/Omitting the field entirely/); + }); + + it('uses the singular noun when count24h is 1', () => { + render(IngestionSkewBanner, { + props: { skew: { ...skew, count24h: 1 }, signal: 'spans' }, + }); + + expect(screen.getByText(/1\s+span ended with a timestamp/)).toBeInTheDocument(); + }); + }); +}); diff --git a/packages/frontend/src/lib/components/ui/range-calendar/index.ts b/packages/frontend/src/lib/components/ui/range-calendar/index.ts new file mode 100644 index 00000000..f9c08dc7 --- /dev/null +++ b/packages/frontend/src/lib/components/ui/range-calendar/index.ts @@ -0,0 +1,6 @@ +import Root from "./range-calendar.svelte"; + +export { + Root, + Root as RangeCalendar, +}; diff --git a/packages/frontend/src/lib/components/ui/range-calendar/range-calendar.svelte b/packages/frontend/src/lib/components/ui/range-calendar/range-calendar.svelte new file mode 100644 index 00000000..1cd1676d --- /dev/null +++ b/packages/frontend/src/lib/components/ui/range-calendar/range-calendar.svelte @@ -0,0 +1,82 @@ + + + + {#snippet children({ months, weekdays })} + + + + + + + + + + +
+ {#each months as month (month.value)} + + + + {#each weekdays as weekday (weekday)} + + {weekday.slice(0, 2)} + + {/each} + + + + {#each month.weeks as weekDates (weekDates)} + + {#each weekDates as date (date)} + + + + {/each} + + {/each} + + + {/each} +
+ {/snippet} + diff --git a/packages/frontend/src/lib/stores/current-project.ts b/packages/frontend/src/lib/stores/current-project.ts new file mode 100644 index 00000000..8b28b16f --- /dev/null +++ b/packages/frontend/src/lib/stores/current-project.ts @@ -0,0 +1,43 @@ +import { writable } from 'svelte/store'; +import { browser } from '$app/environment'; + +// Remembers the last project the user looked at, so pages that have a project +// selector default to the same one across navigations and reloads instead of +// each falling back to "the first project". +const STORAGE_KEY = 'logtide_current_project'; + +function load(): string | null { + if (!browser) return null; + try { + return localStorage.getItem(STORAGE_KEY); + } catch { + return null; + } +} + +function createCurrentProjectStore() { + // In-memory source of truth, hydrated from localStorage once. get() reads this + // so a selection survives within a session even if localStorage is blocked. + let current = load(); + const { subscribe, set } = writable(current); + + return { + subscribe, + set: (projectId: string | null) => { + current = projectId; + if (browser) { + try { + if (projectId) localStorage.setItem(STORAGE_KEY, projectId); + else localStorage.removeItem(STORAGE_KEY); + } catch { + // localStorage may be unavailable + } + } + set(projectId); + }, + /** Current value without subscribing. */ + get: (): string | null => current, + }; +} + +export const currentProjectStore = createCurrentProjectStore(); diff --git a/packages/frontend/src/lib/stores/organization.ts b/packages/frontend/src/lib/stores/organization.ts index b1850e12..7ab11b80 100644 --- a/packages/frontend/src/lib/stores/organization.ts +++ b/packages/frontend/src/lib/stores/organization.ts @@ -8,9 +8,39 @@ interface OrganizationState { loading: boolean; } +const ORG_ID_KEY = 'currentOrganizationId'; +const ORG_OBJECT_KEY = 'currentOrganization'; + +/** + * Restore the last selected organization object from localStorage. + * Persisting the whole object (not just its id) lets a hard reload of a + * sub-route render the org name immediately instead of flashing + * "Select organization" while the org list is still being fetched. + */ +function loadCachedOrganization(): OrganizationWithRole | null { + if (!browser) return null; + try { + const raw = localStorage.getItem(ORG_OBJECT_KEY); + return raw ? (JSON.parse(raw) as OrganizationWithRole) : null; + } catch { + return null; + } +} + +function persistCurrentOrganization(org: OrganizationWithRole | null): void { + if (!browser) return; + if (org) { + localStorage.setItem(ORG_ID_KEY, org.id); + localStorage.setItem(ORG_OBJECT_KEY, JSON.stringify(org)); + } else { + localStorage.removeItem(ORG_ID_KEY); + localStorage.removeItem(ORG_OBJECT_KEY); + } +} + const initialState: OrganizationState = { organizations: [], - currentOrganization: null, + currentOrganization: loadCachedOrganization(), loading: false, }; @@ -25,15 +55,18 @@ function createOrganizationStore() { update((state) => { const currentOrganization = state.currentOrganization || organizations[0] || null; - const savedOrgId = browser ? localStorage.getItem('currentOrganizationId') : null; + const savedOrgId = browser ? localStorage.getItem(ORG_ID_KEY) : null; const restoredOrg = savedOrgId ? organizations.find((org) => org.id === savedOrgId) : null; + const resolved = restoredOrg || currentOrganization; + persistCurrentOrganization(resolved); + return { ...state, organizations, - currentOrganization: restoredOrg || currentOrganization, + currentOrganization: resolved, }; }); }, @@ -49,13 +82,7 @@ function createOrganizationStore() { organization = organizationOrId; } - if (browser) { - if (organization) { - localStorage.setItem('currentOrganizationId', organization.id); - } else { - localStorage.removeItem('currentOrganizationId'); - } - } + persistCurrentOrganization(organization); return { ...state, @@ -66,11 +93,14 @@ function createOrganizationStore() { addOrganization: (organization: OrganizationWithRole) => { - update((state) => ({ - ...state, - organizations: [organization, ...state.organizations], - currentOrganization: organization, - })); + update((state) => { + persistCurrentOrganization(organization); + return { + ...state, + organizations: [organization, ...state.organizations], + currentOrganization: organization, + }; + }); }, updateOrganization: (id: string, updates: Partial) => { @@ -84,6 +114,11 @@ function createOrganizationStore() { ? { ...state.currentOrganization, ...updates } : state.currentOrganization; + // Keep the cached object fresh so a reload does not show a stale name. + if (state.currentOrganization?.id === id) { + persistCurrentOrganization(currentOrganization); + } + return { ...state, organizations, @@ -101,6 +136,10 @@ function createOrganizationStore() { ? organizations[0] || null : state.currentOrganization; + if (state.currentOrganization?.id === id) { + persistCurrentOrganization(currentOrganization); + } + return { ...state, organizations, @@ -122,7 +161,7 @@ function createOrganizationStore() { const newOrg = await apiCall(); update((state) => { - if (browser) localStorage.setItem('currentOrganizationId', newOrg.id); + persistCurrentOrganization(newOrg); return { ...state, @@ -147,16 +186,16 @@ function createOrganizationStore() { const orgs = await apiCall(); update((state) => { - const savedOrgId = browser ? localStorage.getItem('currentOrganizationId') : null; + const savedOrgId = browser ? localStorage.getItem(ORG_ID_KEY) : null; const restoredOrg = savedOrgId ? orgs.find((org) => org.id === savedOrgId) : null; const currentOrganization = restoredOrg || orgs[0] || null; - if (browser && currentOrganization && !savedOrgId) { - localStorage.setItem('currentOrganizationId', currentOrganization.id); - } + // Refresh the cache with the authoritative fetched object (fresh name/role), + // or drop it if the previously selected org no longer exists. + persistCurrentOrganization(currentOrganization); return { ...state, @@ -181,8 +220,8 @@ function createOrganizationStore() { clear: () => { - if (browser) localStorage.removeItem('currentOrganizationId'); - set(initialState); + persistCurrentOrganization(null); + set({ organizations: [], currentOrganization: null, loading: false }); }, }; } diff --git a/packages/frontend/src/lib/stores/stores-persistence.test.ts b/packages/frontend/src/lib/stores/stores-persistence.test.ts new file mode 100644 index 00000000..322e49b0 --- /dev/null +++ b/packages/frontend/src/lib/stores/stores-persistence.test.ts @@ -0,0 +1,47 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { get as storeGet } from 'svelte/store'; + +// $app/environment is a SvelteKit virtual module that does not resolve under +// plain vitest; mock it as a browser context so the localStorage path runs. +vi.mock('$app/environment', () => ({ browser: true })); + +import { timeRangeStore } from './time-range.js'; +import { currentProjectStore } from './current-project.js'; + +// Regression: get() must return the in-memory value, not re-read localStorage. +// If localStorage.setItem is blocked (private mode, quota, disabled), set() still +// updates subscribers, so get() must agree with them within the session instead +// of falling back to a stale/empty localStorage read. +describe('session persistence stores fall back to memory when localStorage is blocked', () => { + let setItem: ReturnType; + + beforeEach(() => { + localStorage.clear(); + setItem = vi.spyOn(Storage.prototype, 'setItem').mockImplementation(() => { + throw new DOMException('blocked', 'SecurityError'); + }); + }); + + afterEach(() => { + setItem.mockRestore(); + localStorage.clear(); + }); + + it('timeRangeStore.get() returns the value set in-session', () => { + const value = { type: 'custom', from: '2026-07-01T00:00', to: '2026-07-02T00:00' }; + timeRangeStore.set(value); + + expect(timeRangeStore.get()).toEqual(value); + expect(storeGet(timeRangeStore)).toEqual(value); + // localStorage write was attempted but rejected, so it stays empty. + expect(localStorage.getItem('logtide_time_range')).toBeNull(); + }); + + it('currentProjectStore.get() returns the value set in-session', () => { + currentProjectStore.set('project-123'); + + expect(currentProjectStore.get()).toBe('project-123'); + expect(storeGet(currentProjectStore)).toBe('project-123'); + expect(localStorage.getItem('logtide_current_project')).toBeNull(); + }); +}); diff --git a/packages/frontend/src/lib/stores/theme.ts b/packages/frontend/src/lib/stores/theme.ts index 15d5ce04..f532f928 100644 --- a/packages/frontend/src/lib/stores/theme.ts +++ b/packages/frontend/src/lib/stores/theme.ts @@ -1,68 +1,86 @@ -import { writable } from 'svelte/store'; +import { writable, get } from 'svelte/store'; import { browser } from '$app/environment'; +// The theme actually applied to the DOM. export type Theme = 'light' | 'dark'; +// What the user picked. "system" follows the OS and tracks it live. +export type ThemePreference = 'light' | 'dark' | 'system'; const STORAGE_KEY = 'logtide-theme'; -function getInitialTheme(): Theme { +function systemTheme(): Theme { if (!browser) return 'dark'; + return window.matchMedia('(prefers-color-scheme: light)').matches ? 'light' : 'dark'; +} +function getStoredPreference(): ThemePreference { + if (!browser) return 'dark'; try { const stored = localStorage.getItem(STORAGE_KEY); - if (stored === 'light' || stored === 'dark') { + if (stored === 'light' || stored === 'dark' || stored === 'system') { return stored; } } catch { // localStorage may be unavailable (incognito, etc.) } + // No explicit choice yet: follow the OS. + return 'system'; +} - // Check OS preference - if (window.matchMedia('(prefers-color-scheme: light)').matches) { - return 'light'; - } - - return 'dark'; +function resolve(pref: ThemePreference): Theme { + return pref === 'system' ? systemTheme() : pref; } function createThemeStore() { - const initialTheme = getInitialTheme(); - const { subscribe, set, update } = writable(initialTheme); + const initialPref = getStoredPreference(); + const preference = writable(initialPref); + // Public store yields the RESOLVED theme, so existing consumers keep working. + const resolved = writable(resolve(initialPref)); - function applyTheme(theme: Theme) { - if (browser) { - const root = document.documentElement; - if (theme === 'dark') { - root.classList.add('dark'); - } else { - root.classList.remove('dark'); - } + function apply(theme: Theme) { + if (!browser) return; + document.documentElement.classList.toggle('dark', theme === 'dark'); + } - try { - localStorage.setItem(STORAGE_KEY, theme); - } catch { - // localStorage may be unavailable - } + function persist(pref: ThemePreference) { + if (!browser) return; + try { + localStorage.setItem(STORAGE_KEY, pref); + } catch { + // localStorage may be unavailable } } - // Apply initial theme if (browser) { - applyTheme(initialTheme); + apply(resolve(initialPref)); + // Track OS changes while the preference follows the system. + window + .matchMedia('(prefers-color-scheme: dark)') + .addEventListener('change', () => { + if (get(preference) === 'system') { + const t = systemTheme(); + resolved.set(t); + apply(t); + } + }); + } + + function setPreference(pref: ThemePreference) { + preference.set(pref); + persist(pref); + const t = resolve(pref); + resolved.set(t); + apply(t); } return { - subscribe, - set: (theme: Theme) => { - set(theme); - applyTheme(theme); - }, + subscribe: resolved.subscribe, + preference: { subscribe: preference.subscribe }, + setPreference, + // Backward-compatible helpers. + set: (theme: Theme) => setPreference(theme), toggle: () => { - update((current) => { - const newTheme = current === 'dark' ? 'light' : 'dark'; - applyTheme(newTheme); - return newTheme; - }); + setPreference(get(resolved) === 'dark' ? 'light' : 'dark'); }, }; } diff --git a/packages/frontend/src/lib/stores/time-range.ts b/packages/frontend/src/lib/stores/time-range.ts new file mode 100644 index 00000000..c9aa3685 --- /dev/null +++ b/packages/frontend/src/lib/stores/time-range.ts @@ -0,0 +1,54 @@ +import { writable } from 'svelte/store'; +import { browser } from '$app/environment'; + +// Remembers the last time window the user picked, so it carries across the +// time-driven pages (logs, traces) instead of each resetting to its own default. +// Consumers validate `type` against what they support and fall back otherwise, +// since not every page offers the same presets. +const STORAGE_KEY = 'logtide_time_range'; + +export interface StoredTimeRange { + type: string; + from: string; + to: string; +} + +function load(): StoredTimeRange | null { + if (!browser) return null; + try { + const raw = localStorage.getItem(STORAGE_KEY); + if (!raw) return null; + const parsed = JSON.parse(raw); + if (parsed && typeof parsed.type === 'string') return parsed as StoredTimeRange; + return null; + } catch { + return null; + } +} + +function createTimeRangeStore() { + // Source of truth in memory, hydrated from localStorage once. get() reads this + // so the value survives within a session even if localStorage is blocked (a + // failed setItem must not make get() return null while subscribers see it). + let current = load(); + const { subscribe, set } = writable(current); + + return { + subscribe, + set: (value: StoredTimeRange | null) => { + current = value; + if (browser) { + try { + if (value) localStorage.setItem(STORAGE_KEY, JSON.stringify(value)); + else localStorage.removeItem(STORAGE_KEY); + } catch { + // localStorage may be unavailable + } + } + set(value); + }, + get: (): StoredTimeRange | null => current, + }; +} + +export const timeRangeStore = createTimeRangeStore(); diff --git a/packages/frontend/src/lib/utils/datetime.ts b/packages/frontend/src/lib/utils/datetime.ts index 9b39f4e7..52403e28 100644 --- a/packages/frontend/src/lib/utils/datetime.ts +++ b/packages/frontend/src/lib/utils/datetime.ts @@ -64,6 +64,25 @@ export function formatTimeAgo(date: Date | string, now: Date = new Date()): stri return `${diffDays} day${diffDays > 1 ? 's' : ''} ago`; } +/** + * Format a date as a short, consistent absolute date (e.g. "Apr 12, 2026"). + * Use for record dates (joined, created) where an absolute value reads better + * than a relative one, pairing it with formatTimeAgo() in a tooltip. + */ +export function formatDateShort(date: Date | string): string { + const dateObj = typeof date === 'string' ? new Date(date) : date; + + if (isNaN(dateObj.getTime())) { + return 'Invalid date'; + } + + return dateObj.toLocaleDateString('en-US', { + year: 'numeric', + month: 'short', + day: 'numeric' + }); +} + /** * Get the browser's timezone */ diff --git a/packages/frontend/src/routes/dashboard/+page.svelte b/packages/frontend/src/routes/dashboard/+page.svelte index 24fe5287..e5fc2278 100644 --- a/packages/frontend/src/routes/dashboard/+page.svelte +++ b/packages/frontend/src/routes/dashboard/+page.svelte @@ -19,6 +19,8 @@ } from '$lib/stores/custom-dashboards'; import DashboardContainer from '$lib/components/custom-dashboards/DashboardContainer.svelte'; import DashboardSwitcher from '$lib/components/custom-dashboards/DashboardSwitcher.svelte'; + import * as Select from '$lib/components/ui/select'; + import RefreshCw from '@lucide/svelte/icons/refresh-cw'; import PanelConfigDialog from '$lib/components/custom-dashboards/PanelConfigDialog.svelte'; import AddPanelDialog from '$lib/components/custom-dashboards/AddPanelDialog.svelte'; import CreateDashboardDialog from '$lib/components/custom-dashboards/CreateDashboardDialog.svelte'; @@ -168,6 +170,34 @@ // ─── Edit mode ──────────────────────────────────────────────────────── + // Auto-refresh (global, for the active dashboard's panels) + let autoRefreshMs = $state(0); + const autoRefreshOptions = [ + { value: '0', label: 'Auto-refresh off', short: 'Off' }, + { value: '30000', label: 'Every 30s', short: '30s' }, + { value: '60000', label: 'Every 1m', short: '1m' }, + { value: '300000', label: 'Every 5m', short: '5m' }, + ]; + const autoRefreshShort = $derived( + autoRefreshOptions.find((o) => o.value === String(autoRefreshMs))?.short ?? 'Off' + ); + + function setAutoRefresh(v: string | undefined) { + autoRefreshMs = v ? parseInt(v, 10) || 0 : 0; + if (browser) localStorage.setItem('logtide_dashboard_autorefresh', String(autoRefreshMs)); + } + + $effect(() => { + if (!browser) return; + const ms = autoRefreshMs; + // Pause while editing so a refresh does not clobber unsaved layout edits. + if (ms <= 0 || $editMode) return; + const id = setInterval(() => { + void customDashboardsStore.fetchAllPanelData(); + }, ms); + return () => clearInterval(id); + }); + function startEdit() { customDashboardsStore.enterEditMode(); } @@ -235,6 +265,11 @@ // ─── Shortcuts ──────────────────────────────────────────────────────── onMount(() => { + const savedAutoRefresh = localStorage.getItem('logtide_dashboard_autorefresh'); + if (savedAutoRefresh) { + const parsed = parseInt(savedAutoRefresh, 10); + if (!isNaN(parsed) && parsed >= 0) autoRefreshMs = parsed; + } shortcutsStore.setScope('dashboard'); shortcutsStore.register([ { @@ -307,6 +342,21 @@ {$dashboardSaving ? 'Saving…' : 'Save'} {:else if $activeDashboard} + + + + {autoRefreshShort} + + + {#each autoRefreshOptions as opt} + {opt.label} + {/each} + +
+ {#if recentSearches.length > 0 && !searchQuery.trim()} +
+ Recent: + {#each recentSearches as term (term)} + + {/each} + +
+ {/if} +
@@ -1762,6 +1898,17 @@ {/if} {#if viewMode === "table"} + { columnStore?.set(cols); }} @@ -1805,7 +1952,7 @@ {:else}
- +
Time @@ -1824,7 +1971,7 @@ {@const globalIndex = i} - {formatDateTime(log.time)} + {formatDateTime(log.time)} - {log.message} + + {#each highlightSegments(log.message, searchQuery) as seg} + {#if seg.match}{seg.text}{:else}{seg.text}{/if} + {/each} + {#each customColumns as col (col)} {@const cellValue = resolveMetadataPath(log.metadata, col)}
-
- - - -
+ handleTimeRangeChange(v as typeof timeRange)} + /> + + + + + + + Recipients + + Who receives the digest. Each email includes its own one-click unsubscribe link. + + + + {#if !config} +

+ Save the schedule first, then add recipients. +

+ {:else} + {#if canManage} +
{ + e.preventDefault(); + addRecipient(); + }} + class="flex gap-2" + > + + + + {/if} + + {#if recipients.length === 0} +

No recipients yet.

+ {:else} +
+ {#each recipients as recipient (recipient.id)} +
+
+ + {recipient.email} + {#if recipient.subscribed} + Subscribed + {:else} + Unsubscribed + {/if} +
+ {#if canManage} +
+ {#if !recipient.subscribed} + + {/if} + +
+ {/if} +
+ {/each} +
+ {/if} + {/if} + + + {/if} +
diff --git a/packages/frontend/src/routes/dashboard/settings/members/+page.svelte b/packages/frontend/src/routes/dashboard/settings/members/+page.svelte index e9e8691d..26a0fb9b 100644 --- a/packages/frontend/src/routes/dashboard/settings/members/+page.svelte +++ b/packages/frontend/src/routes/dashboard/settings/members/+page.svelte @@ -11,6 +11,7 @@ import { Badge } from '$lib/components/ui/badge'; import type { OrganizationWithRole, OrganizationMemberWithUser, PendingInvitation, OrgRole } from '@logtide/shared'; import { canManageMembers } from '@logtide/shared'; + import { formatDateShort, formatTimeAgo } from '$lib/utils/datetime'; import Users from '@lucide/svelte/icons/users'; import Crown from '@lucide/svelte/icons/crown'; import Shield from '@lucide/svelte/icons/shield'; @@ -231,18 +232,6 @@ return Users; } - function formatTimeAgo(date: Date | string): string { - const d = date instanceof Date ? date : new Date(date); - const now = new Date(); - const diff = now.getTime() - d.getTime(); - const days = Math.floor(diff / (1000 * 60 * 60 * 24)); - - if (days === 0) return 'Today'; - if (days === 1) return 'Yesterday'; - if (days < 7) return `${days} days ago`; - return d.toLocaleDateString('en-US'); - } - function formatExpiresIn(date: Date | string): string { const d = date instanceof Date ? date : new Date(date); const now = new Date(); @@ -355,7 +344,7 @@ - {formatTimeAgo(member.createdAt)} + {formatDateShort(member.createdAt)} {#if canManage} diff --git a/packages/frontend/src/routes/dashboard/settings/pii-masking/+page.svelte b/packages/frontend/src/routes/dashboard/settings/pii-masking/+page.svelte index e46109a8..3cc2e1b9 100644 --- a/packages/frontend/src/routes/dashboard/settings/pii-masking/+page.svelte +++ b/packages/frontend/src/routes/dashboard/settings/pii-masking/+page.svelte @@ -282,6 +282,7 @@ // Update locally to avoid full page refresh rules = rules.map((r) => r.id === rule.id ? { ...r, enabled: !r.enabled } : r); } + toastStore.success(!rule.enabled ? `${rule.displayName} enabled` : `${rule.displayName} disabled`); } catch (e) { const msg = e instanceof Error ? e.message : 'Failed to update rule'; toastStore.error(msg); @@ -308,6 +309,7 @@ // Update locally to avoid full page refresh rules = rules.map((r) => r.id === rule.id ? { ...r, action: newAction } : r); } + toastStore.success(`${rule.displayName} set to ${newAction}`); } catch (e) { const msg = e instanceof Error ? e.message : 'Failed to update action'; toastStore.error(msg); diff --git a/packages/frontend/src/routes/dashboard/settings/usage/+page.svelte b/packages/frontend/src/routes/dashboard/settings/usage/+page.svelte index 42714dae..fc5f181b 100644 --- a/packages/frontend/src/routes/dashboard/settings/usage/+page.svelte +++ b/packages/frontend/src/routes/dashboard/settings/usage/+page.svelte @@ -19,6 +19,7 @@ TableRow, } from '$lib/components/ui/table'; import { Button } from '$lib/components/ui/button'; + import TimeRangeButtons from '$lib/components/TimeRangeButtons.svelte'; import Spinner from '$lib/components/Spinner.svelte'; import type { OrganizationWithRole } from '@logtide/shared'; import BarChart3 from '@lucide/svelte/icons/bar-chart-3'; @@ -208,19 +209,11 @@
-
- {#each RANGES as range} - - {/each} -
+ ({ value: String(r.days), label: r.label }))} + value={String(selectedDays)} + onChange={(v) => { selectedDays = parseInt(v, 10) as RangeDays; }} + />