LLM Analyst

LLM Analyst

prohori explain INCIDENT asks a large language model to explain an incident: summary, why it matters, evidence, next steps and caveats. The full page is docs/LLM_ANALYST.md.

Not released yet: run it from a clone (uv sync --extra llm).

• Off by default. With PROHORI_LLM_PROVIDER=none (the default), nothing is sent and the command says so.
• Check before you send. --show-payload prints exactly what would be sent, and sends nothing.
• Redacted. Account names, ids and emails become pseudonyms; --mask-ips hides addresses too. Event descriptions from the logs are never sent. If a replaced value would still be sent, nothing is sent.
• Free-tier terms. On Gemini's free tier, Google may use what is sent to improve its products. Prohori reminds you on every free-tier request.
• Ask questions. prohori ask "..." answers from ATT&CK (all 706 active techniques) and the runbooks, citing them as [S1], [S2]; add -f INCIDENT to ask about an incident. Offline, it shows the best-matching sources.
• Injection-resistant. Text from logs is fenced as data, and an answer that cites an address, account, id, technique or link that is not in the incident or the sources is rejected.
• Never stuck. If the model cannot answer (no key, daily budget or rate limit reached, timeout, blocked answer), you get a local summary and a note saying why.

uv sync --extra llm
uv run prohori explain samples/incidents/INC-ee7c8818fbb34989.json --show-payload
# with PROHORI_LLM_PROVIDER=gemini and GEMINI_API_KEY in .env:
uv run prohori explain samples/incidents/INC-ee7c8818fbb34989.json
uv run prohori ask "How should I respond to password spraying?"

Setting	Default
PROHORI_LLM_PROVIDER	none (mock, gemini)
PROHORI_LLM_MODEL	gemini-3.5-flash-lite
PROHORI_LLM_DAILY_REQUESTS	20
PROHORI_LLM_MASK_IPS	false

Evaluation

An offline evaluation scores recorded answers against 30 hand-written references: 15 attacks, 7 false positives and 8 incidents with an instruction hidden in a user agent, a request path or a username. It counts factual errors, indicators named and points covered, and whether injected instructions were followed and reached the analyst.

No real model has been recorded yet. The baselines are the local summary, the mock and a scripted model that obeys every injection. The answer checks stopped every obeyed injection that named a new value or link (5 of 8); the 3 that made a claim with real values only got through.

uv run python -m prohori.llm.evaluate score                      # offline, from the recordings
uv run python -m prohori.llm.evaluate record --system gemini     # optional: needs a key and budget

Results and method: docs/LLM_ANALYST.md.