Troubleshooting

Troubleshooting

Core Agent Issues

"Connection refused" when starting

Symptom: Agent fails to connect to the LLM server.

Fix: Verify your LLM server is running and the URL is correct:

curl http://localhost:5000/v1/models

Check NEBULUS_BASE_URL in your .env file.

Streaming errors with MLX/Ollama

Symptom: Garbled output or connection drops during streaming.

Fix: Some backends don't support streaming well. Disable it:

NEBULUS_STREAMING=false

"No module named 'chromadb'"

Symptom: Import error when using RAG features.

Fix: Install the full requirements:

pip install -r requirements.txt

Tests fail with import errors

Symptom: ModuleNotFoundError when running tests.

Fix: Make sure you're in the project root and the virtualenv is activated:

cd /path/to/nebulus-atom
source .venv/bin/activate
pip install -r requirements-dev.txt

Swarm Issues

Overlord can't connect to Slack

Symptom: slack_bolt connection errors on startup.

Checklist:

1. Verify SLACK_BOT_TOKEN starts with xoxb-
2. Verify SLACK_APP_TOKEN starts with xapp-
3. Confirm Socket Mode is enabled in Slack app settings
4. Check that the bot is invited to the configured channel
5. Verify SLACK_CHANNEL_ID is correct (use Slack's "Copy Channel ID")

Minion container fails to start

Symptom: DockerException or ImageNotFound when spawning minions.

Fix: Build the minion image:

docker build -t nebulus-minion:latest -f nebulus_swarm/minion/Dockerfile .

Verify Docker is accessible:

docker ps
docker images | grep minion

Minion times out

Symptom: Minion killed by watchdog after 30 minutes.

Possible causes:

• LLM server is slow or overloaded
• Issue is too complex for the configured timeout
• Network issues between minion and LLM server

Fix: Increase the timeout:

MINION_TIMEOUT_MINUTES=60

Or route complex issues to a faster model. See Model Router.

GitHub rate limiting

Symptom: RateLimitExceededException during queue sweeps.

Fix: The Overlord tracks rate limits and backs off automatically. If you're hitting limits:

1. Reduce cron frequency: CRON_SCHEDULE=0 */6 * * * (every 6 hours)
2. Reduce watched repos
3. Use a GitHub App token instead of a personal access token (higher limits)

"No module named 'slack_bolt'" in tests

Symptom: Test imports fail because slack_bolt isn't installed.

Fix: This is expected in development. Tests mock slack_bolt:

import sys
from unittest.mock import MagicMock
sys.modules.setdefault("slack_bolt", MagicMock())

All test files that import Overlord modules include these mocks.

Docker socket permission denied

Symptom: Overlord can't spawn minions due to Docker socket permissions.

Fix: The Overlord container needs access to the Docker socket:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

On Linux, ensure the container user has docker group access, or run with appropriate permissions.

Dashboard shows "Overlord unreachable"

Symptom: Swarm Dashboard can't connect to the Overlord API.

Checklist:

1. Verify the Overlord is running: curl http://localhost:8080/health
2. Check OVERLORD_URL is set correctly for the dashboard
3. If using Docker, ensure the dashboard can reach the Overlord's network
4. Check firewall rules aren't blocking port 8080

State database corruption

Symptom: SQLite errors or missing data.

Fix: The state database can be reset (you'll lose history):

# Backup first
cp /var/lib/overlord/state.db /var/lib/overlord/state.db.bak

# Delete and let Overlord recreate
rm /var/lib/overlord/state.db
docker restart overlord

Overlord Daemon Issues (Phase 3)

Daemon starts but no scheduled tasks fire

Symptom: Daemon logs "Overlord daemon starting..." but no tasks execute.

Checklist:

1. Verify overlord.yml has a schedule: section with valid cron expressions
2. Check that tasks are enabled (enabled: true or omitted — default is enabled)
3. Look for "Invalid cron" errors in the logs — malformed cron expressions are silently skipped
4. The scheduler sleeps until the next fire time — an hourly task won't fire immediately on startup

# Verify schedule is loaded
nebulus-atom overlord daemon start
# Look for: "Scheduled task 'scan' with cron '0 * * * *' (next: ...)"

Slack commands not responding

Symptom: @atom status in Slack gets no response.

Checklist:

1. Verify all three Slack tokens are set: SLACK_BOT_TOKEN, SLACK_APP_TOKEN, SLACK_CHANNEL_ID
2. Check daemon logs for "Slack bot configured for channel ..." — if missing, tokens aren't loaded
3. Verify the bot is invited to the channel
4. Ensure the message starts with the bot mention (the bot name must match)
5. Check for "Slack bot crashed" in logs — indicates a connection failure

Common fix: If the daemon says "Slack tokens not set — running without Slack integration", your environment variables aren't reaching the process. Check your .env file or service configuration.

Daemon starts without Slack despite .env being present

Symptom: You have a .env file with SLACK_BOT_TOKEN etc., but the daemon logs show "Slack tokens not set — running without Slack integration".

Root cause (fixed in v2.4.0): Prior to v2.4.0, the daemon CLI command did not auto-load .env — you had to source .env manually before starting. The daemon now auto-loads .env from the current working directory on startup.

Diagnosis: When the daemon starts, it prints an environment variable detection banner:

Loaded .env from /path/to/project/.env
SLACK_BOT_TOKEN: yes | SLACK_APP_TOKEN: yes | SLACK_CHANNEL_ID: yes
Logging to ~/.atom/overlord/daemon.log (level=INFO)

If any token shows no, check:

1. Your .env file is in the current working directory
2. The variable names are spelled correctly (case-sensitive)
3. Values don't have extra quotes or whitespace

Log file: Check ~/.atom/overlord/daemon.log for the startup banner which also reports token detection status.

Proposals missed while daemon was offline

Symptom: You replied "approve" or "deny" in a proposal thread while the daemon was stopped, and the proposal stayed PENDING.

Explanation: Slack Socket Mode does not buffer historical events. If the daemon is offline when you reply, that event is lost. Starting with v2.5.0, the daemon performs startup reconciliation — on launch, it reads pending proposal threads from the Slack API and applies any approve/deny replies it missed.

Behavior after reconciliation:

• Approved proposals: State transitions to APPROVED, but the DispatchPlan is lost (it was in-memory). A Slack thread notification tells you to re-dispatch if needed.
• Denied proposals: State transitions to DENIED with a reconciliation note.
• No matching replies: Proposal stays PENDING until it expires or you reply again.

What to look for in logs:

Reconciliation applied 1 approvals, 0 denials (scanned=3, skipped=2)

If reconciliation isn't working:

1. Verify the Slack app has channels:history scope (needed to read thread replies)
2. Check that proposals have a thread_ts (proposals created without Slack won't be reconciled)
3. Look for "Failed to fetch thread history" warnings in daemon logs

---

Proposals not receiving thread replies

Symptom: Replying "approve" in a proposal thread does nothing.

Checklist:

1. Verify you're replying in the thread, not as a new message in the channel
2. Check that the Slack app has channels:history scope (needed to read thread replies)
3. Look for the proposal in the database:

   sqlite3 ~/.atom/overlord/proposals.db "SELECT id, state, thread_ts FROM proposals"

4. If thread_ts is NULL, the proposal wasn't posted to Slack (missing Slack bot)

Proposals auto-expiring too quickly

Symptom: Proposals expire before you can approve them.

Fix: The default TTL is 30 minutes. The cleanup loop runs every 5 minutes. If proposals expire faster than expected, check:

1. System clock — the daemon uses UTC timestamps
2. Whether another process is modifying the proposals database
3. Logs for "Cleaning up expired proposals" entries

Detections not appearing in Slack

Symptom: Scheduled scans run but no detection results are posted.

Possible causes:

• Autonomy filtering: Detections are filtered by the current autonomy level. In cautious mode, all detections are reported. In proactive mode, only high-severity detections pass through
• No issues found: If all projects are healthy, there's nothing to report
• No Slack bot: In headless mode, detections are logged but not posted

Debug:

# Run a manual scan to see detections
@atom scan

# Check autonomy level
@atom autonomy

Daily digest not sending

Symptom: No daily digest appears in Slack.

Checklist:

1. Verify digest_enabled: true in overlord.yml notifications config
2. Check that the digest cron has fired — default is 0 8 * * * (8 AM UTC)
3. If the buffer is empty (no events since last digest), no digest is sent — this is by design
4. Verify the Slack bot is connected (digest requires Slack)

"croniter" import error

Symptom: ModuleNotFoundError: No module named 'croniter' when starting daemon.

Fix: Install the dependency:

pip install croniter

Or install from requirements:

pip install -r requirements.txt

Daemon won't shut down cleanly

Symptom: Daemon hangs on Ctrl+C or SIGTERM.

Possible causes:

• A scheduled task is mid-execution (the daemon waits for it to complete)
• The Slack bot has an active connection that's slow to close

Fix: Use the CLI stop command, which sends SIGTERM and waits up to 5 seconds:

nebulus-atom overlord daemon stop

If that fails or times out:

# Force kill (last resort — may leave stale PID file and proposal state)
kill -9 $(pgrep -f "overlord daemon")
# Clean up stale PID file
rm -f ~/.atom/overlord/daemon.pid

Stale PID file

Symptom: overlord daemon status shows "Stale PID file (PID XXXXX is not running)".

Cause: The daemon crashed or was force-killed without clean shutdown, leaving behind ~/.atom/overlord/daemon.pid.

Fix: The start and stop commands handle stale PID files automatically. You can also remove it manually:

rm -f ~/.atom/overlord/daemon.pid

Proposals database locked

Symptom: sqlite3.OperationalError: database is locked

Possible causes:

• Multiple daemon instances running simultaneously
• A crashed daemon left a write lock

Fix:

1. Check if the daemon is running:

   nebulus-atom overlord daemon status

2. If the database is locked after a crash, the lock clears automatically after a few seconds
3. As a last resort, copy and replace:

   cp ~/.atom/overlord/proposals.db ~/.atom/overlord/proposals.db.bak
   sqlite3 ~/.atom/overlord/proposals.db.bak ".dump" | sqlite3 ~/.atom/overlord/proposals_new.db
   mv ~/.atom/overlord/proposals_new.db ~/.atom/overlord/proposals.db

Async test failures in full suite

Symptom: test_run_and_shutdown or test_daemon_starts_and_stops fail when running the full test suite, but pass in isolation.

Explanation: This is a known pytest-asyncio issue with add_signal_handler. When multiple async test files run in the same process, signal handler registration can conflict. These tests pass reliably when run alone:

# These pass
python3 -m pytest tests/test_overlord_daemon.py -v
python3 -m pytest tests/test_overlord_phase3_e2e.py -v

# May fail in full suite due to signal handler conflicts
python3 -m pytest tests/ -v

This does not indicate a bug in the daemon code.

Pre-commit Hook Issues

ruff fails on commit

Symptom: Commit rejected by ruff linting.

Fix: ruff auto-fixes most issues. Re-stage and commit:

git add -u
git commit -m "your message"

If the issue persists, run ruff manually to see details:

python3 -m ruff check <file>

ruff-format changes files

Symptom: ruff-format reformats files during commit.

Fix: This is normal. Re-stage the formatted files and commit again:

git add -u
git commit -m "your message"

Getting Help

• Check the GitHub Issues for known problems
• Review logs: docker compose logs -f
• For Overlord: check /var/log/overlord/overlord.log (if configured)
• Enable debug logging: LOG_LEVEL=DEBUG