docs(queries): Query limits & performance, LCQL reference, and compiled SDK snippets

[tomaz-lc]

Details

This PR expands the Query Console / LCQL documentation under docs/4-data-queries/ and includes a CI/security hardening pass on the repository's GitHub Actions workflows. On the docs side it adds a new "Query Limits & Performance" page, greatly expands the LCQL examples and reference material, documents how to discover the per-organization Search API endpoint, refreshes and annotates the Query Console screenshot, and corrects several accuracy issues in the programmatic examples.

It also introduces mechanisms so the docs stay correct over time: custom syntax highlighting for LCQL code blocks, extraction of every Python and Go SDK example into standalone files that are compiled in CI so they cannot drift from the published SDKs, and a CI check that catches ordered-list numbering silently breaking in the rendered site.

What's included

New page - Query Limits & Performance (docs/4-data-queries/query-limits-and-performance.md):

• Query types (stateless, projection, aggregation, stateful) and how each executes (paged vs whole-timeline).
• Data streams (event, detection, audit).
• Concurrent-query limits and the HTTP 429 behavior, query timeouts, and aggregation cardinality guidance.
• Query progress and cost reporting (pre-flight estimate vs per-page actual billing), including a worked progress-bar example - the clamp(batchesCompleted / batchesInScope, 0, 100%) formula the Query Console uses - in Python, Go, and bash + jq.
• A guide to writing efficient queries, plus a troubleshooting section.

LCQL Examples (docs/4-data-queries/lcql-examples.md):

• New "Time Range" section explaining how the time range is supplied per interface (the query string for raw LCQL and Replay, the Console time picker, the Search API startTime/endTime parameters, and the CLI --start/--end flags) and the supported relative, absolute, and bounded formats.
• New "Common Operators and Patterns" section, and expanded stateful operators (with child / with descendant / with events) with sample data, plus subtree-wildcard examples.
• Correctness fixes (for example COUNT vs COUNT_UNIQUE on a GROUP BY key), added projections, and high-cardinality GROUP BY guidance.

Programmatic reference (docs/4-data-queries/index.md):

• LCQL field paths corrected to / notation throughout.
• Documents Search API endpoint discovery via /v1/orgs/{oid}/url with a per-region reference table, recommending discovery over hardcoding a direct endpoint.
• Go SDK examples corrected to use the real SDK methods and the correct client initialization; CLI saved-query flags corrected.

Query Console UI (docs/4-data-queries/query-console-ui.md):

• Replaced the overview screenshot with an annotated version (numbered UI elements) and fixed the numbered-list rendering (the sub-lists were flattening and the 1-10 list fragmented into restarting blocks; re-indented to 4-space nesting so it renders as one continuous list).
• Documented two elements not covered before: the status-line progress indicator, and the Search Details panel (opened from the status-line info icon) with its timings, completion breakdown, and copyable Query ID to share with support when troubleshooting.
• Corrected the documented query billing unit to 200,000 events.

Tooling:

• Custom Pygments lexer for LCQL (hooks/lcql_lexer.py) so lcql code blocks are highlighted; registered via the existing MkDocs hooks mechanism and fails safe to plain text (registration is wrapped so any failure degrades to unhighlighted blocks instead of breaking the build).
• All Python and Go SDK examples, plus the progress-bar helper, extracted to snippets/python/*.py and snippets/golang/*/main.go, embedded with pymdownx.snippets.
• CLAUDE.md guidance to verify SDK, CLI, and LCQL snippets against authoritative public sources.

CI and security hardening:

• New workflows: LCQL lexer unit tests; snippet compilation (go build / go vet for Go, py_compile + SDK import check for Python); and a PR docs preview that builds the site and uploads it as a short-lived (1-day) artifact for review. All are read-only, fork-guarded, and never execute snippet code.
• deploy-pages.yml: permissions scoped to jobs (build gets contents: read; only deploy gets pages: write + id-token: write), with persist-credentials: false and job timeouts.
• mkdocs.yml: strict: true so build warnings (broken nav, unresolved internal links, bad redirects) fail every build path including deploy, not only the PR preview.
• Removed docs.yml, a redundant second deploy path that pushed to an unserved gh-pages branch while exposing a write token to the build steps.
• link-checker.yml: job-scoped permissions, fork guard, the PR internal-link check is now blocking (external stays advisory), the PR-comment step is split into its own minimally-scoped job, the third-party action is SHA-pinned, result artifacts use 1-day retention, and the results-artifact download is continue-on-error so the comment job does not fail spuriously when the build breaks before the link checker runs.
• snippet-tests.yml: the Python SDK the snippets are checked against is pinned (limacharlie==5.5.1) so the import-path check is reproducible, matching the pinned Go module.
• publish-release-notes.yml / scripts/publish-release-note.py: sanitize the machine-fed release note before it is written into the docs. The body is HTML-escaped and additionally rejected if any Markdown link/image destination uses a scheme other than http/https/mailto - html.escape does not neutralize those, so a [x](javascript:...) or data: destination would otherwise render as a live one-click link on the docs site. The URL must be https, on an allowlisted host, and free of characters that could break out of the [GitHub Release](URL) Markdown link it is embedded in; component/version use re.fullmatch (no trailing newline); and a length cap plus job timeout bound the work.
• scripts/check-list-numbering.py (new) + a docs-preview step: fails a PR only on NEW ordered-list numbering breaks - an item the author numbered >1 that renders as a restarted 1. because its continuation content is indented below Python-Markdown's 4-space rule (the Query Console regression fixed in this PR). It is false-positive-free by construction: the author's source numbering is the tie breaker between an accidental split and intentional adjacent lists. Five pre-existing cases are recorded in scripts/list-numbering-baseline.json and are left unchanged here.

Screenshots

LCQL formatting before:

LCQL formatting after:

Blast radius / isolation

Documentation content changes are scoped to the Query Console / LCQL section (docs/4-data-queries/). The PR also adds new supporting files (hooks/lcql_lexer.py, snippets/, tests/, scripts/check-list-numbering.py) and makes CI changes: three new workflows, an ordered-list numbering check wired into the PR docs preview, plus hardening of the existing deploy, link-check, and release-note workflows. The docs lexer fails safe (blocks render as plain text if it ever fails to register). Deploy behavior is functionally unchanged - deploy-pages.yml remains the serving path; only its permission scoping is tightened, strict is enforced, and the redundant docs.yml is removed.

How this was verified

• mkdocs build passes on every build path (now strict) and markdownlint reports 0 errors across all docs.
• The LCQL lexer unit tests pass (57 tests) and run in CI.
• All 8 Go snippets compile (go build / go vet) and all 8 Python snippets byte-compile with their SDK import paths verified against the pinned package; both run in CI. The progress-bar Python and Go snippets produce the expected percentage on a sample page.
• All workflow YAML validates; the release-note publish script passes py_compile, and its URL/body validation is exercised against the XSS and break-out vectors above (rejected) alongside legitimate inputs (accepted).
• The ordered-list numbering check was validated against the pre-fix Query Console page (caught) and its fixed version (clean) plus adversarial false-positive/true-positive cases, and reports zero new breaks over its baseline.

Follow-up

• I will update the remaining inline screenshots (time picker, schema fields, table columns) with the new UI elements and information in a separate PR later this week.
• Settings/ops actions (not code): scope the repository_dispatch token used by the release-notes publisher as tightly as possible; and after this merges, delete the stale gh-pages branch and clear the Pages source-branch setting so the site is served solely from the Actions artifact.
• Five pre-existing pages have ordered-list numbering breaks recorded in the baseline (sophos, windows/installation, plaso, hayabusa-bigquery, velociraptor-bigquery); each is a mechanical 4-space re-indent and can be cleaned up in a follow-up, dropping its baseline entry.

🤖 Generated with Claude Code