Skip to content

Known Issues and Fixes

Jump to bottom
Mohammad edited this page Jun 8, 2026 · 4 revisions

Known Issues & Fixes

This page documents all significant problems encountered during development and how they were resolved.

Setup Issues

PEP 668 — externally-managed-environment

Problem: pip install -r requirements.txt blocked by Kali's system Python protection. Fix: Always use a virtual environment. setup.sh handles this automatically:

bash setup.sh

scipy Compile Error — OpenBLAS not found

Problem: scipy 1.13.1 has no pre-built wheel for Python 3.13 on Kali. Compilation fails because libopenblas-dev and libatlas-base-dev don't exist on this Kali version. Fix: setup.sh installs python3-scipy, python3-numpy, python3-pandas via apt and then installs from pip using --only-binary :all:. Emergency manual fix if needed:

sudo apt install -y python3-scipy python3-numpy python3-pandas
cp -r /usr/lib/python3/dist-packages/scipy venv/lib/python3.13/site-packages/
cp -r /usr/lib/python3/dist-packages/numpy venv/lib/python3.13/site-packages/
cp -r /usr/lib/python3/dist-packages/pandas venv/lib/python3.13/site-packages/

asyncpg Version Incompatibility

Problem: asyncpg==0.29.0 is not available for Python 3.13. Fix: Use asyncpg==0.31.0 in requirements.txt. Already fixed.

asyncpg ModuleNotFoundError in venv

Problem: venv was created with --system-site-packages, causing it to use system SQLAlchemy which couldn't see venv's asyncpg. Fix: Recreate venv without --system-site-packages. Use venv/bin/uvicorn explicitly instead of system uvicorn.

Runtime Issues

Celery DATABASE_URL Empty

Problem: Celery fork workers don't inherit environment variables. DATABASE_URL is empty, causing ArgumentError: Could not parse SQLAlchemy URL. Fix: tasks.py reads the .env file directly as plain text — no dotenv dependency needed:

def _env(key, default=""):
    value = os.getenv(key, "")
    if value:
        return value
    env_file = Path(__file__).resolve().parent.parent.parent / ".env"
    if env_file.exists():
        for line in env_file.read_text().splitlines():
            if line.strip().startswith(f"{key}="):
                return line.split("=", 1)[1].strip().strip('"').strip("'")
    return default

Python 3.13 + Celery asyncio Crash

Problem: asyncio.run() breaks in forked Celery workers on Python 3.13. Fix: Create a new event loop per task:

def _run(coro):
    loop = asyncio.new_event_loop()
    try:
        return loop.run_until_complete(coro)
    finally:
        loop.close()

Agent Not Picking Up WATCH_PATH

Problem: Agent started with sudo drops environment variables, so WATCH_PATH is not set and defaults to /home (entire home directory). Fix: Use sudo -E to preserve environment and source .env first:

cd ~/hybrid-rsentry && set -a && source .env && set +a
sudo -E ~/hybrid-rsentry/venv/bin/python -m agent.monitor

Canary Files Corrupting Git Refs

Problem: When WATCH_PATH pointed inside the project directory, canary files named AAA_* ended up inside .git/refs/heads/, breaking git.

Three-layer prevention now in place (no longer occurs on new installs):

  1. AAA_*.txt and **/AAA_*.txt excluded in .gitignore
  2. _validate_watch_path() in monitor.py — agent exits at startup with a clear error if WATCH_PATH is inside a git repo
  3. _is_safe_target() in adaptive.py — Markov repositioner refuses to move canaries into .git/, /proc/, /sys/, /dev/, /run/

Emergency fix if corruption has already occurred:

find ~/hybrid-rsentry/.git/refs -name "AAA_*" -delete
git pull origin main

tasks.py Empty on Disk

Problem: backend/workers/tasks.py was committed as an empty file. All Celery tasks were missing, causing ImportError on startup. Fix: File was fully rebuilt from scratch and pushed via GitHub API.

agent/graph.py Empty on Disk

Problem: agent/graph.py was empty, causing ImportError: cannot import name 'FilesystemGraph'. Fix: FilesystemGraph class rebuilt from scratch with BFS canary placement and orphan cleanup.

Alert Count Mismatch Across Dashboard

Problem: StatsBar used limit: 500 and counted array length. AlertFeed fetched all alerts including acknowledged ones. Numbers didn't match. Fix:

  • Added /api/alerts/counts backend endpoint returning exact counts with no limit
  • StatsBar now uses this endpoint
  • AlertFeed now filters acknowledged: false and removes alerts immediately on ACK

High False Positive Rate — Firefox Cache

Problem: Agent was generating hundreds of ENTROPY_SPIKE alerts from Firefox browser cache activity because:

  1. WATCH_PATH defaulted to /home (entire home dir)
  2. lineage.py listed python, bash, sh as suspicious parents — everything on Linux
  3. lineage.py listed /.cache/ as a suspicious spawn path — matched Firefox

Fix:

  • Added WATCH_PATH=/home/mohammad/Documents to .env
  • Removed python, bash, sh from suspicious parents in lineage.py
  • Removed /.cache/ from suspicious spawn paths in lineage.py
  • Created agent/exceptions.py with comprehensive whitelist
  • Added whitelist check at start of monitor._handle_event()

NVIDIA API 429 Rate Limit

Problem: Multiple Celery workers calling NVIDIA API simultaneously triggers rate limiting. Fix: Lua-based atomic Redis rate limiter in ai_analyst.py enforces minimum delay between calls per provider. Two separate API keys (events vs alerts) prevent them from blocking each other. Cerebras (AI_API_KEY_CEREBRAS) can be added as the primary provider with higher rate limits.

AI Analyst Page Shows "Waiting for events" Forever

Problem: tasks.py was publishing final AI results as ai_analysis_update but App.jsx only handled ai_analysis. Analysis cards never appeared. Fix: App.jsx now handles both ai_analysis and ai_analysis_update message types.

Risk Score Stuck at 100 After Acknowledging Alerts

Problem: Acknowledging alerts via bulk SQL or ACK button did not trigger update_host_risk, so risk score stayed at 100. Fix: acknowledge_alert endpoint now calls update_host_risk.delay(host_id) after each acknowledgment.

auto-acknowledge Pointing to Wrong ID

Problem: analyze_alert endpoint was passing event_id instead of alert_id to the analyze_alert_ai Celery task. Auto-acknowledge after AI analysis never fired. Fix: Endpoint now passes the correct alert_id. The event_id is embedded in event_data so the AI Analyst page can still match and render the correct card.

Vite/CORS/Frontend Issues

CORS Errors — Dashboard Shows No Alerts

Problem: Vite grabbed a different port (e.g. port 3002) because port 3000 was already occupied. api/client.js had a hardcoded http://localhost:8000 base URL, bypassing the Vite proxy. The backend's CORS policy only allowed port 3000, so all requests from port 3002 were blocked. Symptom: Dashboard loads but no alerts or events appear. Browser console shows CORS errors. Fix: Changed client.js base URL from 'http://localhost:8000' to '' (empty string). Axios now uses relative paths (/api/...) which Vite proxies to port 8000 regardless of which port Vite is on. Also fix the port conflict:

sudo pkill -f uvicorn && sudo pkill -f celery && sudo pkill -f agent.monitor
kill $(lsof -t -i:3000) 2>/dev/null
sleep 2 && bash start.sh

Duplicate Processes After git pull

Problem: After git pull, old root-owned uvicorn was still holding port 8000. New start.sh backend failed silently with [Errno 98] Address already in use. Frontend hit the old stale backend with stale data. Symptom: tail /tmp/rsentry-backend.log shows ERROR: [Errno 98] Address already in use Fix: Always kill all running processes before restarting after a pull:

sudo pkill -f uvicorn
sudo pkill -f celery
sudo pkill -f "agent.monitor"
kill $(lsof -t -i:3000) 2>/dev/null
sleep 2
bash start.sh

Risk Score Stays Elevated After Containment Completes

Problem: update_host_risk was called from events.py at the same time as auto_ack_containment, so the risk recalculation raced against the acknowledgment write — the score often recalculated before the alerts were marked acknowledged and stayed elevated. Fix: Removed update_host_risk.delay() from events.py. It is now called at the end of auto_ack_containment and auto_ack_by_event, after the DB commit, so the score always reflects the cleared alert state.

LockBit / Qilin Not Contained — Ultra-Fast Bulk Encryption

Problem: LockBit 5.0 and Qilin encrypt 1 000+ files in under 1 second using a create-new + delete-original pattern (not os.rename()). The rename velocity tracker never fires. BPF behavioral score reached 60 (above SCORE_ALERT=50, below SCORE_BLOCK=70). Python _handle_behavior required entropy >= 6.5 to trigger containment, but files were already deleted/restored before the 1 ms poll ran — entropy read as 0.0. Fix (v2.2.0):

  1. Signal 6 added to __calc_score: del_per_sec >= 20 && files_deleted >= 5 → +15. Pushes score 60 → 75, crossing SCORE_BLOCK before the alerted flag is set.
  2. _block_on_unlink_score injected into __handle_unlink: when score >= SCORE_BLOCK, PID is immediately written to blocked_pids BPF map at the kernel level.
  3. Entropy gate relaxed in _handle_behavior: if entropy >= 6.5 or ev.score >= 70: — high-confidence kernel scores now trigger SIGSTOP without a readable sample file.

Docker rsentry_frontend Container Holding Port 3000

Problem: After a docker compose up that includes the production frontend container (rsentry_frontend mapped to 0.0.0.0:3000→80/tcp), the Vite dev server cannot bind port 3000. Symptom: npm start exits with Error: listen EADDRINUSE: address already in use :::3000 Fix:

docker stop rsentry_frontend
cd ~/hybrid-rsentry/frontend && npm start

eBPF Issues

eBPF Sensor Fails Silently — python3-bpfcc Not Installed

Problem: agent.monitor defaults to SENSOR_BACKEND=ebpf but python3-bpfcc was not installed. The agent starts and logs backend=eBPF mode=enforce but immediately exits — no events are detected. Symptom: tail /tmp/rsentry-agent.log shows something like [ebpf] python3-bpfcc not installed or the log file stays empty after agent start. Fix:

sudo apt install python3-bpfcc bpfcc-tools -y

Then restart the agent. Temporary workaround (no restart needed for other services):

# Add to .env
SENSOR_BACKEND=inotify
# Then restart just the agent
sudo pkill -f agent.monitor
cd ~/hybrid-rsentry && set -a && source .env && set +a && sudo -E ~/hybrid-rsentry/venv/bin/python -m agent.monitor

npm Issues

npm ENOTEMPTY During Install

Problem: npm install failed with ENOTEMPTY: directory not empty, rename node_modules/typescript. Fix:

rm -rf node_modules
npm install

Clone this wiki locally