Architecture

Architecture

clients ──HTTPS + token──▶ api ───────┐
devices ──syslog UDP/TCP/TLS──▶ syslogd ┴─▶ enrich (GeoIP, threat intel) ─▶ ingest worker pool ─▶ batched transactions,
                                                                          (500 events or 500 ms)  one per day file
                                                                                                           │
                               api ──▶ search engine ──▶ day files, newest first ◀──────────────────────────┘
                                       (events ⋈ events_fts in each)
                                       retention worker (deletes old day files)
                                       intel service (feed refresh, reload on change)

• pkg/ocsf: event model and validation.
• pkg/parser: automatic parsing, custom parsers (pattern, JSON, key=value), templates and pattern drafting.
• pkg/sources: applies each source's parser to its lines.
• pkg/ai: parser suggestions from a local model via Ollama.
• pkg/backup: backups (VACUUM INTO of every file, in a .tar.gz), the schedule, uploads, and restores applied at startup.
• pkg/storage: SQLite setup (WAL, synchronous=NORMAL, busy_timeout=5000), schema, queries. Events are stored one file per UTC day by when they happened (siemlite-events/2026-10-04.db); everything else is in siemlite.db. A search reads the days newest first and stops as soon as the page is full; within a day, a search with many matches reads them newest first instead of sorting them all. Each day's events_fts is an external-content FTS5 table kept in sync by a trigger and one bulk update per batch, so log text is not stored twice. It indexes the parsed message, program, host, user, addresses and the values of the fields (not their names), not the original line, which an hourly job clears 24 hours after the event arrived (by arrival, tracked with arrival-number watermarks, so late events keep theirs for a day too). Event ids carry their day, and an arrival number lets the alert engine find late events filed in older days.
• pkg/ingest: enriches each event, then a buffered channel and worker pool that flushes in batches.
• pkg/syslogd: UDP, TCP and TLS syslog listeners with a sender allowlist.
• pkg/enrich: enrichment document and GeoIP/ASN lookups (.mmdb, reloaded on change).
• pkg/intel: feed parsing, the in-memory indicator matcher and feed refresh.
• pkg/search: combines time window, OCSF filters and FTS5.
• pkg/retention: deletes expired days' files, and older events from the day the cutoff falls in.
• pkg/alerts: the alert engine: runs each rule over the events since its last check.
• pkg/mcp: a small Model Context Protocol server (JSON-RPC over HTTP, tools filtered by access level); pkg/agents: AI agent tokens and the "AI agents (MCP)" integration; the tools themselves are in api/mcp.go and run the REST handlers as the token.
• pkg/audit: writes the audit log as INTERNAL events.
• pkg/auth: access tokens, users, roles, source limits, sessions and permission checks.
• api: HTTPS server and endpoints. web: the embedded UI.