Skip to content

Detection Engine

Jump to bottom
Mohhudib edited this page Jun 5, 2026 · 10 revisions

Detection Engine

Hybrid R-Sentry uses six detection layers that work together to produce a high-confidence threat score. Each layer catches a different aspect of ransomware behavior. The first five layers run in the inotify-based agent (monitor.py); Layer 6 is the optional eBPF kernel sensor (monitor_ebpf.py) that provides deeper, lower-latency visibility.

Layer 1 — Canary Files

What it does: Places strategically named bait files (AAA_*.txt) across the monitored directory. Any process that reads, modifies, moves, or deletes a canary file triggers an immediate CRITICAL alert.

Why it works: Ransomware encrypts files indiscriminately — it will inevitably touch the canary files. Legitimate processes have no reason to touch files named AAA_.

Key details:

  • 30 canary files placed by default (configurable via CANARY_COUNT)
  • 4 naming prefixes: AAA_, aaa_, ZZZ_, zzz_ — 4× the surface area vs. a single prefix
  • Placement strategy: BFS across directory tree, 5 canaries per directory spread across prefixes
  • Any canary touch, move, or deletion → CRITICAL → auto-containment fires immediately
  • All 4 prefix variants are excluded from .gitignore to prevent accidental commits
  • In eBPF mode, canary inodes are registered in the canary_inodes BPF map at startup — renames are blocked at the kernel level (-EPERM) before any file is overwritten
  • Markov repositioner (inotify mode only) is blocked from placing canaries into .git/, /proc/, /sys/, /dev/, /run/
  • Agent validates WATCH_PATH at startup and exits if it's inside a git repo

Layer 2 — Shannon Entropy Analysis

What it does: Monitors the Shannon entropy (randomness) of files as they are modified. Ransomware encrypts files, which dramatically increases their entropy. The engine tracks a rolling window of entropy values per file and fires when the delta exceeds the threshold.

Why it works: Encrypted data is indistinguishable from random noise — it always has very high entropy (close to 8 bits). Normal file modifications do not cause sudden entropy spikes.

Key details:

  • Threshold: 3.5 bits delta → MEDIUM alert
  • Absolute threshold: 7.2 bits current entropy → HIGH alert
  • Rolling window: last 30 samples per file
  • Spike window: delta must occur within 10 seconds
  • Memory cap: 5 000 files with LRU eviction — prevents OOM on large watch paths
  • Partial reads: only the first 65 KB of each file is sampled

Severity mapping:

Condition Severity
Delta >= 5.0 bits HIGH
Absolute entropy >= 7.2 bits HIGH
Delta >= 3.5 bits MEDIUM

Layer 3 — Process Lineage Scoring

What it does: When a file event is triggered by an active process (PID > 0), the engine walks the process ancestry chain and scores the process on multiple suspicion signals.

Scoring factors:

Signal Weight Description
Suspicious parent process +30 Parent is netcat, ncat, etc.
Suspicious spawn path +25 Process executable in /tmp/, /dev/shm/
Deep ancestry chain +15 More than 5 ancestor processes
Unreadable executable +10 Cannot read the process binary
No controlling TTY +5 Background/hidden process
Rapid spawn +5 Process age < 2 seconds
Process exited rapidly +40 Process exits before scoring completes (baseline score)
Benign parent -15 Parent is systemd, bash, python3, etc.

dpkg hash verification: The process binary SHA-256 is verified against a database of 416 K dpkg hashes. Verdict is one of:

  • MATCH — binary matches a known legitimate package
  • MISMATCH — binary is present but hash differs (tampered)
  • UNKNOWN — binary not in dpkg database (unpackaged or custom binary)

A 512-entry LRU cache avoids repeated SHA-256 computation on the same binary.

Score range: 0–100. A score >= 40 triggers an alert.

Layer 4 — Ransomware Extension Detection

What it does: Watches for file rename events where the destination extension matches known ransomware patterns. Also detects newly created files with ransomware extensions.

Trigger conditions:

Condition Severity
Document file renamed to .enc, .locked, .wcry, .crypted, .crypt, .encrypted etc. CRITICAL
Non-document file renamed to a ransomware extension HIGH
New file created with a ransomware extension HIGH

Why it works: Ransomware often renames files in-place after encrypting them, appending its signature extension. This layer catches that behavior even before entropy thresholds are reached.

Layer 5 — Markov Chain Adaptive Repositioning

What it does: Tracks which directories are accessed most frequently and automatically moves canary files to predicted high-risk locations.

Why it works: Ransomware typically starts in one area and spreads. The Markov chain models directory access transitions and predicts where the attacker will go next, pre-positioning canary files there.

Key details:

  • Repositioning triggers when transition probability >= 70%
  • Minimum 10 observations required before repositioning
  • Repositioning is logged as MARKOV_REPOSITION event (not treated as a threat)
  • Runs every 300 seconds by default
  • _is_safe_target() blocks all system paths — never repositions into .git/, /proc/, /sys/, /dev/, /run/

Layer 6 — eBPF Kernel Sensor

What it does: A kernel-level sensor (agent/monitor_ebpf.py) that attaches to multiple syscall probes and (optionally) a BPF LSM hook to detect and block ransomware behavior at the kernel boundary — below the userspace filesystem layer. Provides faster detection, richer behavioral context, and the ability to block canary file renames in nanoseconds.

Requirements: Linux kernel ≥ 6.19 with BCC 0.35 installed (sudo apt install python3-bpfcc bpfcc-tools -y). Falls back to inotify automatically if BCC is not available (SENSOR_BACKEND=inotify in .env).

Key capabilities:

Feature Description
5-syscall coverage Monitors openat (file access), vfs_write (binary writes), unlink (file deletion), rename (velocity tracking), execve (child spawning) simultaneously via kprobes/tracepoints
proc_profile BPF map Per-process behavioral profile (10 000 entries): files_opened, files_written, files_deleted, files_renamed, child_procs, write_bytes, score (0–100), alerted flag
Behavioral scoring (5 signals) +35: rapid unlink+write (≥2 del/sec); +25: rename velocity ≥3; +15: total_ops>15 && deleted>3; +15: write/read symmetry; +10: child spawning + file ops
Entropy verification On behavior_event: if entropy ≥ 6.5 → BLOCK + SIGSTOP (silent encryption detected); if < 6.5 → ALERT only (prevents false positives)
BPF LSM canary blocking When lsm=bpf is active: LSM_PROBE(path_rename) + canary_inodes BPF map blocks canary renames with -EPERM in nanoseconds — no userspace roundtrip needed
blocked_pids BPF map Velocity burst sets a kernel-level block on the PID — all subsequent renames by that process are denied by the LSM hook
Ransomware family profiling Recognizes LockBit 5.0 (16-char extension, two-pass), Akira (.akiracrypt, intermittent), ESXi-targeting (.vmdk/.vmx/.vmem)
IGNORE_COMMS suppression Filters out 40+ known benign processes (Docker, Redis, git, apt, npm, uvicorn, Celery, Firefox, etc.) before any scoring
Run modes --mode audit (alert only, no SIGSTOP); --selftest (unit-tests the engine); --seed-canaries (place canaries without running sensor); --print-bpf (print compiled BPF program)

Sensor backend selection: Set SENSOR_BACKEND=ebpf (default) or SENSOR_BACKEND=inotify in .env.

Markov repositioning: Disabled when using eBPF backend — the eBPF sensor tracks file access patterns via proc_profile directly, making Markov repositioning redundant.

BPF LSM activation: The canary_inodes kernel-level blocking requires the lsm=bpf kernel boot parameter. Without it, the sensor falls back to SIGSTOP via the userspace contain() callback. To enable:

# Add to /etc/default/grub:
GRUB_CMDLINE_LINUX="lsm=bpf"
sudo update-grub && sudo reboot

Why eBPF? Userspace inotify events can be delayed or coalesced by the kernel. eBPF probes fire inside the kernel syscall path — sub-millisecond latency and no way for ransomware to evade by timing userspace polling.

Combined Threat Score

When both entropy and lineage signals fire, they are combined into a single weighted score:

combined_score = lineage_score × 0.6 + (entropy_delta / 8.0 × 100) × 0.4
Combined Score Event Type Severity
>= 70 COMBINED_ALERT CRITICAL
40–69 COMBINED_ALERT HIGH
Entropy only ENTROPY_SPIKE MEDIUM
Lineage only (>= 40) PROCESS_ANOMALY HIGH

False Positive Suppression

The system includes an exception whitelist (agent/exceptions.py) that suppresses alerts for known-safe activity:

  • Whitelisted processes: apt, git, pip, docker, firefox, gpg, celery, uvicorn, etc.
  • Whitelisted paths: ~/.cache/, /var/cache/apt/, browser profile directories
  • Whitelisted extensions: .zip, .gpg, .pdf, .jpg, .mp4, .pyc, .deb, and more
  • Smart /tmp filter: files in /tmp/ with document extensions (.docx, .pdf, etc.) bypass the whitelist — prevents ransomware staging in /tmp/ being silently ignored

Canary hits are never whitelisted — they always trigger regardless of process or path.

Clone this wiki locally