Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,18 +142,21 @@ Cooldown behavior:

- [`demos/rule-evaluation-and-dedup-demo/README.md`](demos/rule-evaluation-and-dedup-demo/README.md) explains the third demo and links its committed before/after dedup artifacts
- [`demos/config-change-investigation-demo/README.md`](demos/config-change-investigation-demo/README.md) explains the config-change investigation demo and its committed artifacts
- [`docs/reviewer-pack.md`](docs/reviewer-pack.md) is the top-level no-guessing reviewer pack
- [`docs/reviewer-path.md`](docs/reviewer-path.md) maps common review questions to the right demo and artifacts
- [`docs/architecture.md`](docs/architecture.md) diagrams the local file-based detection workflow
- [`docs/sample-output.md`](docs/sample-output.md) summarizes the committed sample artifacts
- [`docs/roadmap.md`](docs/roadmap.md) sketches the next demo directions
- [`docs/roadmap.md`](docs/roadmap.md) sketches the v0.7 / v1.0 consolidation direction
- [`data/processed/summary.json`](data/processed/summary.json) captures the default run in machine-readable form
- [`data/processed/richer_sample/summary.json`](data/processed/richer_sample/summary.json) captures the richer scenario pack
- [`tests/`](tests/) keeps regression coverage close to the CLI behavior and windowing logic

## Next Demo Directions
## v0.7 / v1.0 Direction

- add a focused auth/login anomaly triage walkthrough on top of the existing window features
- add a compact config-change drift follow-up scenario using the current deterministic evidence model
- keep sample-output docs and public repo presentation aligned with the checked-in demo state
- stabilize the four-demo matrix and avoid broad platform expansion
- freeze reviewer-visible artifact names unless a rename is intentionally coordinated across docs, tests, and sample outputs
- use [`docs/reviewer-pack.md`](docs/reviewer-pack.md) and [`docs/architecture.md`](docs/architecture.md) as the consolidation entrypoints
- add at most one more demo before v1-style consolidation

## Scope

Expand Down
43 changes: 43 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Architecture

`telemetry-lab` is a local, file-based detection workflow lab. It is organized around committed sample inputs, deterministic demo pipelines, and reviewer-facing artifacts.

```mermaid
flowchart TD
Inputs["Committed sample inputs<br/>JSONL, CSV, YAML configs"]
CLI["Local CLI entrypoints<br/>run, run-ai-demo, run-rule-dedup-demo, run-config-change-demo"]
Window["telemetry-window-demo<br/>normalize -> windows -> features -> alerts"]
AI["ai-assisted-detection-demo<br/>rules -> cases -> JSON-only drafting"]
Dedup["rule-evaluation-and-dedup-demo<br/>raw hits -> cooldown -> suppression reasons"]
Config["config-change-investigation-demo<br/>config changes -> bounded evidence correlation"]
Artifacts["Reviewer artifacts<br/>CSV, JSON, JSONL, Markdown, PNG"]
Review["Reviewer inspection<br/>README, reviewer path, reviewer pack, tests"]

Inputs --> CLI
CLI --> Window
CLI --> AI
CLI --> Dedup
CLI --> Config
Window --> Artifacts
AI --> Artifacts
Dedup --> Artifacts
Config --> Artifacts
Artifacts --> Review
```

## Design Rules

- Detection decisions stay deterministic and inspectable.
- The AI-assisted demo is limited to bounded JSON-only case drafting.
- Artifacts are file-based and suitable for local regeneration or GitHub review.
- Artifact names are reviewer-visible contracts during the v0.7 / v1.0 consolidation path.
- The repository does not provide production monitoring, real-time ingestion, dashboards, alert routing, case management, autonomous response, or final incident verdicts.

## Demo Boundaries

| Demo | Boundary |
| --- | --- |
| `telemetry-window-demo` | Converts raw events into window features and alerts; does not become a live stream processor. |
| `ai-assisted-detection-demo` | Drafts constrained summaries from deterministic cases; does not decide incident outcomes or call tools. |
| `rule-evaluation-and-dedup-demo` | Explains cooldown and suppression behavior; does not route alerts. |
| `config-change-investigation-demo` | Correlates risky changes with bounded local evidence; does not monitor live infrastructure. |
2 changes: 1 addition & 1 deletion docs/reviewer-brief.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,4 +71,4 @@ The other demos emit reviewer-facing artifacts such as `dedup_report.md`, `inves

## Next milestone

Add another compact investigation path that strengthens the bridge from telemetry features to analyst-facing monitoring and detection workflows.
Consolidate the current reviewer path for v0.7 / v1.0: keep the demo matrix stable, freeze reviewer-visible artifact names, maintain one top-level reviewer pack, and keep the architecture diagram aligned with actual CLI and artifact behavior.
68 changes: 68 additions & 0 deletions docs/reviewer-pack.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Reviewer Pack

This is the top-level reviewer pack for `telemetry-lab`.

Use it as a short, no-guessing path through the repository. The project is a local, file-based detection workflow lab: it is not a SIEM, not a dashboard, and not a production monitoring platform.

## What to Review

Start with the stable demo matrix in [`docs/reviewer-path.md`](reviewer-path.md). It maps common review questions to the demo and artifacts that answer them:

| Review question | Demo | Primary evidence |
| --- | --- | --- |
| How are raw events converted to alert features? | `telemetry-window-demo` | `data/processed/features.csv`, `data/processed/alerts.csv`, `data/processed/summary.json` |
| How is AI constrained? | `ai-assisted-detection-demo` | `demos/ai-assisted-detection-demo/artifacts/case_summaries.json`, `demos/ai-assisted-detection-demo/artifacts/audit_traces.jsonl`, guardrails in `demos/ai-assisted-detection-demo/README.md` |
| How are duplicate alerts reduced? | `rule-evaluation-and-dedup-demo` | `demos/rule-evaluation-and-dedup-demo/artifacts/rule_hits_before_dedup.json`, `demos/rule-evaluation-and-dedup-demo/artifacts/rule_hits_after_dedup.json`, `demos/rule-evaluation-and-dedup-demo/artifacts/dedup_explanations.json` |
| How are risky config changes investigated? | `config-change-investigation-demo` | `demos/config-change-investigation-demo/artifacts/investigation_hits.json`, `demos/config-change-investigation-demo/artifacts/investigation_report.md` |

## Architecture

See [`docs/architecture.md`](architecture.md) for the repository-level workflow diagram.

The important shape is:

1. committed sample inputs and configs
2. deterministic local pipelines
3. bounded AI drafting only where explicitly marked
4. committed or regenerated artifacts
5. reviewer inspection through docs, tests, and reports

## Artifact Naming Contract

The current artifact names are reviewer-facing contracts for the v0.7 / v1.0 consolidation path.

- Keep the demo matrix stable unless a demo is intentionally retired.
- Prefer additive artifacts over renaming existing reviewer-visible outputs.
- If an artifact must be renamed, update the README, reviewer path, this reviewer pack, demo README, tests, and committed sample outputs in the same change.
- Do not introduce platform-style names that imply alert routing, case management, real-time ingestion, or autonomous response.

## Fast Verification

From the repository root:

```bash
python -m pip install -e ".[dev]"
python -m telemetry_window_demo.cli run --config configs/default.yaml
python -m telemetry_window_demo.cli run-ai-demo
python -m telemetry_window_demo.cli run-rule-dedup-demo
python -m telemetry_window_demo.cli run-config-change-demo
pytest
```

Use the same Python interpreter for install, tests, and demo commands.

## Review Anchors

- [`docs/reviewer-brief.md`](reviewer-brief.md): short problem / value summary
- [`docs/reviewer-path.md`](reviewer-path.md): demo choice by review question
- [`docs/architecture.md`](architecture.md): local file-based workflow diagram
- [`docs/sample-output.md`](sample-output.md): committed output counts and sample artifacts
- [`tests/test_reviewer_docs.py`](../tests/test_reviewer_docs.py): regression checks for reviewer-facing docs

## Boundaries

- No production monitoring claims
- No real-time ingestion or streaming state
- No dashboard, alert routing, or case-management service
- No autonomous response actions
- No final incident verdicts from the AI-assisted demo
57 changes: 36 additions & 21 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,40 @@
# Roadmap

This repository is intentionally small, so the next steps should be new demos that make the existing telemetry pipeline easier to understand rather than a broad platform build-out.

# Roadmap

`telemetry-lab` is moving from demo expansion toward v0.7 / v1.0 consolidation.

The repo now has four reviewer-verifiable demos and a clear [`docs/reviewer-path.md`](reviewer-path.md). The priority is to keep the demo matrix stable, preserve artifact names, and make review faster without implying a SIEM, dashboard, or production monitoring platform.

Recently added:

- [rule-evaluation-and-dedup-demo](../demos/rule-evaluation-and-dedup-demo/README.md) now shows raw rule hits, retained alerts, and suppression reasons side by side.
- [config-change-investigation-demo](../demos/config-change-investigation-demo/README.md) now shows risky configuration changes, bounded evidence attachment, and deterministic investigation summaries.
- Core telemetry-window validation now gives clearer failures for malformed inputs, invalid run configs, rule parameters, plot CSV values, and output window bounds.

## 1. Auth/Login Anomaly Triage Demo

Goal:
Add a demo that walks from bursty login failures into follow-on signals such as source spread, eventual success, or repeated target concentration.

Why it helps the portfolio:
This strengthens the repo's analyst-facing story. It shows how simple window features and rule output can support a concrete triage narrative instead of stopping at generic alert generation.

## 2. Config-Change Drift Follow-Up Demo

Goal:
Add a compact follow-up scenario centered on repeated config drift, rollback attempts, and evidence that the remediation path actually reduced nearby denials or noisy follow-on signals.

Why it helps the portfolio:
This would build on the current config-change investigation demo without changing the repo's local, file-based character. It would show not just the initial risky change, but also how deterministic evidence can support a short remediation narrative.
- [`docs/reviewer-path.md`](reviewer-path.md) maps common review questions to the right demo and artifacts.
- [`docs/reviewer-pack.md`](reviewer-pack.md) collects the top-level reviewer flow and artifact naming contract.
- [`docs/architecture.md`](architecture.md) describes the local file-based workflow shape.

## v0.7 / v1.0 Consolidation

1. Stabilize the demo matrix.
2. Freeze reviewer-visible artifact names unless a rename is intentional and documented across README, reviewer docs, demo docs, tests, and sample outputs.
3. Keep one top-level reviewer pack as the primary no-guessing entrypoint.
4. Keep the architecture diagram aligned with actual CLI and artifact behavior.
5. Prefer regression tests and documentation accuracy over adding new workflow surface area.

## Optional Final Demo

At most one more demo should be added before v1-style consolidation.

Good candidates:

- auth/login anomaly triage from bursty login failures into follow-on signals
- config-change drift follow-up showing rollback attempts and reduced nearby denials

Only add one if it clearly strengthens the detection workflow portfolio without turning the repo into a platform.

## Non-Directions

- No production monitoring claims
- No realtime ingestion or streaming state
- No dashboard or service deployment
- No alert routing or case management
- No autonomous response
29 changes: 29 additions & 0 deletions tests/test_reviewer_docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,4 +74,33 @@ def test_readme_links_reviewer_path_and_uses_lab_framing() -> None:

assert "A local, file-based detection workflow lab" in readme
assert "not a SIEM, dashboard, or monitoring platform" in readme
assert "[`docs/reviewer-pack.md`](docs/reviewer-pack.md)" in readme
assert "[`docs/reviewer-path.md`](docs/reviewer-path.md)" in readme
assert "[`docs/architecture.md`](docs/architecture.md)" in readme


def test_top_level_reviewer_pack_covers_matrix_and_artifact_contract() -> None:
reviewer_pack = _read_repo_file("docs/reviewer-pack.md")

assert "top-level reviewer pack" in reviewer_pack
assert "Artifact Naming Contract" in reviewer_pack
assert "[`docs/reviewer-path.md`](reviewer-path.md)" in reviewer_pack
assert "[`docs/architecture.md`](architecture.md)" in reviewer_pack

for question, demo_name, artifact_paths in REVIEWER_DEMO_MATRIX:
assert question in reviewer_pack
assert f"`{demo_name}`" in reviewer_pack
for artifact_path in artifact_paths:
assert f"`{artifact_path}`" in reviewer_pack


def test_architecture_doc_keeps_local_file_based_boundaries() -> None:
architecture = _read_repo_file("docs/architecture.md")

assert "flowchart TD" in architecture
assert "local, file-based detection workflow lab" in architecture
assert "Artifact names are reviewer-visible contracts" in architecture
assert "does not provide production monitoring" in architecture

for _, demo_name, _ in REVIEWER_DEMO_MATRIX:
assert f"`{demo_name}`" in architecture
Loading