Skip to content

Threat Intelligence and Detection

Krishnendu Bhattacherjee edited this page Nov 10, 2026 · 2 revisions

Threat Intelligence and Detection

Indicator enrichment integrates VirusTotal, AbuseIPDB and URLhaus. Sigma supports a bounded subset, hunting uses a KQL-style filter, and ATT&CK annotations describe recorded hits rather than effectiveness. See the hunting grammar.

Optional YARA scanner

The base install works without YARA. To enable the T3.2 scanner, install the separate pinned extra (Windows x64 wheel, BSD-3-Clause):

python -m pip install --only-binary=:all: -r requirements-yara.txt

Scan immutable bytes acquired by a trusted caller; the worker receives no sample path. This harmless example compiles one original demonstration rule:

from src.engine.yara import RuleSource, scan_snapshot

source = '''rule demo_marker {
    meta:
        id = "demo.marker"
        author = "TriageLens example"
        source = "https://example.invalid/triagelens-demo"
        revision = "v1"
        license = "MIT"
    strings:
        $marker = "TriageLens harmless marker"
    condition:
        $marker
}'''
result = scan_snapshot(b"TriageLens harmless marker", (RuleSource("demo", source),))
print(result.status, result.size, result.engine, result.version)

match and no_match describe this exact SHA-256 snapshot and rule-pack digest, never a malicious/benign disposition. Missing optional package returns missing_engine; unsupported versions/platforms or failed isolation return unavailable. Compile errors identify the zero-based source index, category/code and line when available, without snippets or exception text. Timeout, resource exhaustion, worker failure and truncation remain separate incomplete outcomes. Matches contain attributed rule metadata and bounded offsets/lengths, never bytes. Missing or invalid ATT&CK annotations retain the hit with explicit diagnostics.

The optional pack in rules/yara has two original text-pattern rules and two adapted rules from Marcin Ulikowski's yara-rulz. The upstream MIT notice is in rules/yara/LICENSE-elceef.txt. The adaptations retain the detection strings and conditions, replace metadata to meet the scanner contract, and add TriageLens technique mappings. The upstream source URL tracks a moving branch; the recorded upstream dates are not immutable commit identifiers. The local files and scanner pack digest identify the actual scanned rules. To scan a harmless snapshot with all four reviewed sources:

from pathlib import Path
from src.engine.yara import RuleSource, scan_snapshot

paths = sorted(Path('rules/yara').glob('*.yar'))
pack = tuple(RuleSource(p.stem, p.read_text(encoding='utf-8')) for p in paths)
result = scan_snapshot(b'powershell -encodedcommand example', pack)
print(result.status, [match.rule for match in result.matches])

These patterns can match benign text, need not indicate execution, and have no measured real-world detection quality. ATT&CK IDs describe rule mappings, not verified observations; a human must investigate matches.

Limits: 16 MiB/snapshot, 30 sources (one rule per unique namespace), 1 MiB total source and 10,000 patterns. Includes and imports are rejected conservatively even inside comments/strings; native includes are also disabled. Every rule requires string metadata id, author, source (HTTPS reference), revision and license; optional mitre_attack contains comma-separated technique IDs. Other metadata is rejected. Only reviewed text sources are accepted, never serialized native rules. Archive signatures are rejected; bytes are never executed or unpacked.

A fresh Windows AppContainer with no network capabilities runs each scan in a Job Object: 512 MiB committed memory, 10 seconds user CPU, one process, kill on parent exit. Parent monitoring enforces a 10-second compile/startup deadline and 20-second worker wall deadline; the YARA-X scanner also has a 5-second scan timeout. Windows checks CPU limits periodically; these are not hard real-time guarantees. Results cap occurrences at 100/pattern, 1,000/file and 256 KiB JSON; reaching a cap is reported as truncated. Limits can tighten bounds. One scan is admitted per Python process; concurrent calls return busy.

The launcher stages trusted interpreter/stdlib/extension files in a disposable temporary directory, grants its fresh identity read/execute access only there, and supplies a minimal environment. It never changes installed-runtime ACLs or grants sample-directory/database access. Normal exit removes the profile/runtime; an abrupt parent exit kills the worker but may leave the empty profile and copied runtime for later operator cleanup. Runtime staging has a 60-second overall budget checked before IPC; blocking filesystem/OS setup calls are not preemptible. The API can scan an explicitly registered local Windows directory. Set the roots before starting the API; an empty setting disables directory scanning. Roots must be fixed local directories outside the project, system, program and known credential directories. For example, replace the path below with an existing local directory:

$env:YARA_SCAN_ROOTS = '{"samples":"C:\\triagelens-samples"}'
uvicorn src.api.main:app --host 127.0.0.1 --port 8000

In another PowerShell window, submit a registered root ID, optional relative directory, and caller-generated job ID. Use harmless files for local testing:

$job = [guid]::NewGuid().ToString()
$body = @{ job_id = $job; root_id = 'samples'; relative_path = '' } | ConvertTo-Json
$run = Invoke-RestMethod -Method Post -Uri 'http://127.0.0.1:8000/api/yara/scans' -ContentType 'application/json' -Body $body
$run
Invoke-RestMethod "http://127.0.0.1:8000/api/yara/jobs/$job"
Invoke-RestMethod "http://127.0.0.1:8000/api/yara/scans/$($run.scan_ids[0])"

The job reports rejected files and incomplete scope. A retry with the same job ID and request returns its recorded state; reuse with different inputs is rejected. If the process stops during a job, its recorded running state requires a new job ID for another attempt. File reads use pinned Windows handles and reject reparse points, hard links, alternate streams and recognized archives. Limits are 1,000 entries, four directory levels, 100 files, 16 MiB/file, 128 MiB/job, and a 60-second job deadline. Blocking filesystem calls and database operations are not preemptible. API scan admission is serialized within the Windows logon session; direct calls to scan_snapshot retain their separate per-process admission. No file or path is sent to a provider. Scan results and qualified YARA hits are stored separately from AI output and labels; a hit links to existing alerts only on an exact lowercase SHA-256 indicator. An unmatched hit remains in the scan record. Linking does not imply execution.

To explicitly look up a stored snapshot hash with the configured VirusTotal provider, use POST /api/yara/scans/{scan_id}/enrichment. This sends only the hash to the provider under the existing quota and cache rules; it is never automatic. unavailable, an incomplete job, and no_match do not establish that a file is safe. Offline adversarial file tests are recorded under T3.5 in TASKS.md. No detection-quality claim is made.

Wiki home · Project README

Clone this wiki locally