Spatial Open Intelligence Shed — a self-hosted OSINT workstation for situational awareness research. 47 live data feeds, a FollowTheMoney entity graph on PostgreSQL 16 with pgvector, hybrid RAG with GPU-accelerated reranking, two-sided Bayesian source adjudication, W3C PROV evidence DAG, and an automated 24h intelligence briefing generated by a 7-model NVIDIA NIM cascade with Ollama fallback. Built entirely on free and open-source resources. No commercial API is a hard dependency.
IntelShed is an integration layer over existing libraries, datasets, and tools — not a framework. It runs standalone on a workstation or forms a 3-node mesh (primary, field, sensor) with Ed25519-authenticated delta sync, geo-sharded entity exchange, and FedAvg gradient propagation across a LAN. Designed for analysts and researchers who need offline-capable intelligence infrastructure without vendor lock-in.
FastAPI · React · Cesium · PostgreSQL + pgvector · DuckDB · Ollama · NVIDIA NIM · Docker · PyTorch + PyG · mesh networking
See
THIRD_PARTY_NOTICES.mdfor attributions.
- Local-first — operator owns data, processing, and output. No cloud dependency, no vendor lock-in, no surrender of data sovereignty.
- Fail-soft by design — the system never binary-fails. It dims. No GPU → CPU fallback, no internet → Pi edge autonomy, no LLM → deterministic briefing.
- Auditable evidence — W3C PROV evidence DAG with Merkle checkpoints and tamper detection. Every claim traces to its source.
- Anti-hallucination by design — 4-layer guard stack, constraint caps on corroboration, red-team veto gate in the multi-agent orchestrator.
- Mesh-native — 3-node LAN topology with Ed25519 identity, geo-sharded entity exchange, and FedAvg gradient propagation. Not a single-point-of-failure system.
- System overview
- Quick start
- What works without keys
- Architecture
- Intelligence pipeline
- Mesh network topology
- Benchmark
- FAQ
- Known limitations
- Contributing
- License
- Star History
| Category | Description |
|---|---|
| Feeds | 47 live data feeds — GDELT 2.0 raw files (no API key, no rate limits), USGS earthquakes, GDACS, NWS alerts, AIS maritime (AISstream + MyShipTracking + AISHub), ADS-B aircraft, MalwareBazaar, URLhaus, OpenPhish, IODA outages, CAMS air quality, SMARD energy, Pegel water levels, EONET wildfires, NOAA space weather, CISA KEV, WHO DON, Cloudflare Radar, BGP (bgpv6 + RIPEstat). Parallel fetch via asyncio.gather for non-GDELT feeds; GDELT stays serial with 6s pacing. Fail-soft: stale cache or empty payload on upstream errors. |
| Globe | Cesium 3D globe with 26 toggleable layers + MapLibre 2D split view. Offline Protomaps via PMTiles (regional ~500 MB or planet ~130 GB). Layer cap with LRU eviction (max 15 active), heap guard at 800 MB. 38 layer-specific React hooks. |
| Briefing | Automated 24h intelligence digest (LOCAL / REGION / GLOBAL buckets), 22+ alert types, maritime anomaly detection (AIS trajectory analysis), space weather monitoring, agentic loop with two-pass critique-refine, threat clusters via Louvain community detection, CII (Country Instability Index), satellite change detection, GNN 24h forecast. Generated on schedule or on demand. |
| Entity graph | FollowTheMoney schema on PostgreSQL 16 (~126k entities, 126k edges, 309k statements). Splink dual-pipeline entity resolution, LLM-in-the-loop adjudication for ambiguous sameAs pairs, GNN-predicted edges (4,000+), geo-sharded by 10×10 deg grid. DuckDB sidecar for analytical workloads. |
| RAG | Hybrid retrieval: pgvector HNSW (cosine) + tsvector GIN (FTS) + reciprocal rank fusion + BGE reranker (GPU CUDA, 14× speedup over CPU). 2,000+ indexed chunks. Citation check enforcement. Situation feedback loop (opt-in). |
| Trust layer | Claim atomization → multilingual-e5-large embeddings → constraint caps (geo >2000 km, temporal >7 d, event-type incompatibility) → two-sided Bayesian adjudication (Beta sensitivity + specificity per source, F1 quality, online posterior updates). Corroboration rate calibrated from 100% → 16.8% via constraint layer. pgvector HNSW for similarity search (8.3× improvement). |
| Graph analytics | PageRank, betweenness centrality, Louvain community detection, weakly connected components. PageRank feeds back into provenance reliability scoring (max +0.1 boost). Graph runtime: Node/Edge/Graph DAG with topological scheduling and parallel execution (MEGA PLAN Phase 1+2). |
| Evidence | W3C PROV evidence DAG with Merkle root checkpoints (every 100 items), tamper detection via verify_dag(), queryable lineage (up/down BFS), WORM SQLite triggers. Zero-Trust Ingest PEP with hash-chained evidence capture. 6 API endpoints. |
| Multi-agent | 8-phase orchestrator (Coverage → Retrieval → Spatial → Corroboration → Temporal → Synthesis → ACH → Red-Team) with shared blackboard, loop detection (state_hash MD5, threshold=3), per-phase evaluators, red-team veto gate, atomic claim decomposition. 13 phase modules. Depths: standard / deep / due_diligence. |
| Causal intelligence | 4 modules: rule-based chain detection (temporal + geo + keyword templates), FtM causal edge schema (triggers/enables/prevents/mitigates), counterfactual perturbation on entity graph, LLM-based cause-effect extraction from claim text. |
| Anomaly detection | Isolation Forest on feed time series, River HalfSpaceTrees on live streams, spatial LISA (Local Moran's I, pure-numpy) on fusion grid cells with p-values and HH/LL/HL/LH cluster types. Aircraft anomaly detection (very_low_altitude, rapid_descent, military_callsign, high_speed_low_alt). |
| Fusion | Multi-source correlation heatmap, LISA spatial autocorrelation (5/10 hotspots typically significant at p<0.01), 3-source rule for HIGH severity situations, delta grid tracking. 76+ active situation tracks in typical operation. |
| GNN pipeline | GAT correlation (static, temporal edge features), T-GCN forecaster (GCN+GRU, 24h region hotspot prediction), GNNExplainer (edge importance attribution), FedAvg gradient exchange across mesh nodes (~10 MB diffs), DP-SGD differential privacy on gradients. |
| OSINT | Dark web search (8 engines + Tor proxy with exit-node rotation), Shodan InternetDB IP lookup, passive DNS (RapidDNS + VirusTotal), domain intelligence (crt.sh + Wayback + RDAP), MITRE ATT&CK, IOC extraction, Flowsint enrichment chain (47 enrichers, 80 enricher files). CelesTrak TLE + SGP4 satellite propagation. |
| Predictive | LightGBM forecasting on snapshot time series, 24h entity count forecast, prediction ground-truth pilot, prediction ledger with automated retraining. |
| LLM | Ollama (qwen3:8b) local fallback + NVIDIA NIM 7-model cascade: step-3.5-flash → step-3.7-flash → kimi-k2.6 → llama-3.1-8b → mistral-3.5 → nemotron-3-super → minimax-m2.7 → Ollama. Central dispatch via llm_provider.py. 4-layer anti-hallucination stack. Fail-soft: degrades to deterministic briefing without GPU. |
| Mesh | 3-node LAN topology: primary (Docker + Caddy TLS), field (venv), sensor (Pi). Ed25519 node identity, epidemic gossip, briefing delta sync with ETag/304, HMAC-signed telemetry, geo-sharded entity exchange, FedAvg gradient propagation. Edge autonomy: Pi generates briefings without hub or LLM (13 rule categories). 7/7 mesh scaling components shipped. |
| Operations agent | ORGA cycle (Observe → Reason → Gate → Act → Verify → Rollback → Audit), per-action trust scoring, immutable audit ledger, shadow-mode diagnosis, VCR record/replay. Phase 1+2 shipped: read-only actions + 5 T2/T3 write executors with trust graduation. |
| Security | 4-layer anti-hallucination (prompt guard → RAG integrity → session guard → output guard). RBAC + JWT + WebAuthn passkeys. CSP single source of truth (csp_policy.py → header / meta / Caddyfile). Rate limiting (Redis sliding window). HMAC + Ed25519 node auth. Secret management with scoped API keys. |
| MCP | 77+ tools for Cursor/Claude integration — briefing, mesh, feeds, globe control, OSINT, threat intel, backup, Flowsint enrichment, dark web search, IOC extraction. Per-tool RBAC policy + quota. JMESPath server-side projections. docs/MCP.md |
| Docker | 5 services: backend (Python 3.12-slim, GPU passthrough, 20 GB mem limit) + PostgreSQL 16 (pgvector + postgis + pg_trgm) + Caddy (TLS, SPA, /api proxy) + Redis 7 + Tor. 748 API routes. docker compose up -d --build |
| Tests | 6,374+ tests across 354 modules. Independent benchmark v2: 5/5 hard assertions pass, 5/5 promotion gates pass. ruff + tsc + prettier pre-commit hooks. CI: 8 GitHub Actions jobs. |
| Codebase | 572 backend Python source files (21,126 items incl. cache/tests), 71 frontend .tsx components, 49 React hooks, 36 lib utilities, 60+ feed bridges, 31 route files, 13 agent phase modules, 98 scripts, 110 docs. 251 config fields. 100 feature flags enabled in production .env. |
| Approach | Situational awareness for decision support — not an attack tool. |
Full feature catalog with env vars → docs/FEATURES.md
Prerequisites: Docker, Ollama with models pulled (qwen3:8b, nomic-embed-text), free Cesium Ion token. Optional: NVIDIA NIM API key for cloud reasoning models.
ollama pull qwen3:8b
ollama pull nomic-embed-text # required for RAG embeddingsgit clone https://github.com/sookoothaii/intelshed.git
cd intelshed
git submodule update --init --recursive # Pi sync scripts in offgrid-raspi/
cp backend/.env.example backend/.env # API keys, feature flags
cp frontend/.env.example frontend/.env # required: VITE_CESIUM_ION_TOKEN
docker compose up -d --build| Service | URL |
|---|---|
| UI | https://localhost (accept self-signed cert) |
| API | https://localhost/api/docs |
| Health | https://localhost/api/health/ping |
API key: docker compose exec backend python -c "import os; print(os.getenv('WORLDBASE_API_KEY','NOT_SET'))" — use as X-API-Key header.
If
WORLDBASE_API_KEYis not set inbackend/.env, all API endpoints are open. Set a key for any instance exposed beyond localhost.
Docker notes:
docker compose down -vwipes the database volume — use with caution.- Caddy uses a self-signed TLS certificate. On Windows,
curlmay return empty even with-k; use a browser orInvoke-WebRequest -SkipCertificateCheck. - Do not run venv backend and Docker stack simultaneously — separate databases cause silent data divergence.
- PostgreSQL 16 on host port
5434, data in volumeworldbase-pg. DuckDB sidecar for analytical workloads. - Feature flags at
GET /api/admin/flagsmay showenabled: falsewhen not explicitly set in.env. Code defaults inconfig.py/features.pyapply at runtime.
Requires Python 3.12+, Node.js 20+.
git clone https://github.com/sookoothaii/intelshed.git
cd intelshed
git submodule update --init --recursive
cp backend/.env.example backend/.env
cp frontend/.env.example frontend/.env
python -m venv backend/venv
source backend/venv/bin/activate # Linux/macOS
# backend\venv\Scripts\activate # Windows
pip install -r backend/requirements.txt
# Start backend + frontend (see start.ps1 or start.sh)| Service | URL (venv mode) |
|---|---|
| UI | http://localhost:5176 |
| API | http://localhost:8002/docs |
| Health | http://localhost:8002/api/health |
| Ollama | http://127.0.0.1:11434 |
Verify (venv): ./scripts/smoke-test.ps1 (Windows) or ./scripts/smoke-test.sh (Linux) — 33 checks.
Optional Python packages (venv only — Docker image includes these):
pip install sentence-transformers # BGE reranker (RAG_RERANK=1)
pip install scikit-learn # Isolation Forest (WORLDBASE_ANOMALY_DETECTION=1)
pip install lightgbm numpy # LightGBM forecasting (WORLDBASE_PREDICTIVE=1)For Cursor / Claude automation — full guide: docs/MCP.md.
# backend/.env
WORLDBASE_MCP=1
WORLDBASE_MCP_WRITE=1
WORLDBASE_AGENT_BUS=1 # globe fly_to / layer toggle via MCP
# frontend/.env
VITE_WORLDBASE_AGENT_BUS=1 # HUD must be openMCP endpoint: https://localhost/api/mcp (Docker) or http://127.0.0.1:8002/api/mcp (venv). 77+ tools, per-tool RBAC policy enforced.
| GLOBE | MAP | DATA | SPLIT |
|---|---|---|---|
 |
 |
 |
 |
Full set: docs/screenshots/
./scripts/start-docker.sh # Linux
# .\scripts\start-docker.ps1 # WindowsSensor node sync → offgrid-raspi/docs/WORLDBASE_PI_SYNC.md
Most feeds operate without API keys. All feeds are fail-soft — stale cache or empty payload on upstream errors.
| Tag | Layers | Key |
|---|---|---|
no-key |
GDELT, USGS, EONET, GDACS, SMARD, IODA, Pegel, NWS, CAMS, ADS-B (adsb.fi), MalwareBazaar, URLhaus, OpenPhish, CISA KEV, space weather, RIPEstat BGP, Cloudflare Radar | — |
recommended |
NASA FIRMS | free signup |
optional |
OpenSky OAuth, ENTSO-E, Blitzortung, AISstream, ReliefWeb, ACLED, Flowsint, Windy | varies |
required |
Cesium terrain/imagery | Ion token |
Docker: GET https://localhost/api/health · GET https://localhost/api/trust · GET https://localhost/api/credentials/status (add X-API-Key header if set). venv: replace with http://127.0.0.1:8002/api/....
Docker mode:
┌─────────────────────────────────────────────────────────────────┐
│ Caddy TLS reverse proxy :443 │
│ https://localhost → SPA + /api/* proxy to backend │
│ CSP · HSTS · WebSocket/SSE flush │
└───────────────────────────┬─────────────────────────────────────┘
│
┌───────────────────────────▼─────────────────────────────────────┐
│ React 18 + Cesium 1.142 (+ MapLibre 2D) served by Caddy │
│ Globe · DATA · AI chat · Agent Bus · Intel graph · Situations │
│ 71 .tsx components · 49 hooks · 38 layer hooks · LRU cap │
└───────────────────────────┬─────────────────────────────────────┘
│ /api/*
┌───────────────────────────▼─────────────────────────────────────┐
│ FastAPI + PostgreSQL 16 + pgvector + DuckDB :8002 │
│ 47 feeds · 8-phase orchestrator · Bayesian adjudication │
│ Evidence DAG (W3C PROV) · Zero-Trust Ingest PEP │
│ Hybrid RAG (HNSW + GIN + RRF + GPU reranker) │
│ LLM entity resolution · graph analytics · graph runtime │
│ Causal intelligence · GNN pipeline (GAT + T-GCN) │
│ LISA spatial autocorrelation · fusion heatmap │
│ Operations Agent (ORGA P1+P2) · 4-layer anti-hallucination │
│ Trust layer (constraint caps + corroboration) │
│ Redis cache · asyncio autopilot · 748 routes · 77+ MCP tools │
└───────────────────────────┬─────────────────────────────────────┘
┌──────────────────┼──────────────────┐
▼ ▼ ▼
GDELT · USGS · Ollama :11434 Pi :443/ingest
GDACS · NWS … qwen3 + RAG (offgrid-raspi)
47 feeds 7-model NIM delta sync
cascade Ed25519 + FedAvg
venv mode: Caddy layer absent; Vite serves :5176, FastAPI on :8002.
Agent reference → AGENTS.md · MCP setup → docs/MCP.md
Feed ingest (parallel asyncio.gather) Entity resolution (Splink + LLM)
│ │
▼ ▼
Claim atomizer ──→ E5 embeddings ──→ Constraint caps ──→ Corroboration
│ (pgvector HNSW) (geo/temporal/event) │
│ ▼
│ Two-sided Bayesian
│ (Beta sens/spec, F1)
│ │
▼ ▼
Fusion heatmap ──→ LISA (Local Moran's I) ──→ Situations board (76+)
│ │
▼ ▼
GNN pipeline (GAT + T-GCN) Briefing digest (24h)
│ │
▼ ▼
GNNExplainer ──→ FedAvg ──→ DP-SGD NVIDIA NIM 7-model cascade
→ Ollama fallback
3-node mesh for distributed intelligence collection across a LAN. Ed25519-authenticated, geo-sharded, with FedAvg gradient exchange.
┌──────────────────────────────────────────────────────────────────┐
│ Primary node (Docker + Caddy TLS) :443 │
│ 47 feeds · fusion · LLM briefing · FtM graph · PostgreSQL │
│ NVIDIA NIM 7-model cascade · Ollama fallback · GPU (RTX 3080 Ti) │
│ Evidence DAG · Bayesian adjudication · graph analytics │
│ GNN pipeline · causal intelligence · operations agent │
└───────────┬────────────────────────────────┬────────────────────┘
│ GET /api/node/pull │ POST /api/node/ingest
│ (briefing + entity delta) │ (HMAC-signed telemetry)
│ ETag/304 · SHA-256 · Ed25519 │ Ed25519 · 45s interval
▼ ▼
┌────────────────────────┐ ┌──────────────────────────────┐
│ Field node (venv) │ │ Sensor node (Pi 4) │
│ :8002 HTTP │ │ SSH-only · no backend │
│ Feeds · briefing │ ← SCP ── │ Edge autonomy (13 rules) │
│ LLM proxy · pull │ │ Sensors · offline portal │
│ Mesh push to Pi │ │ Mesh relay · LCD display │
└────────────────────────┘ └──────────────────────────────┘
| Node type | Role | Transport | Hardware |
|---|---|---|---|
| Primary | Feed aggregation, fusion, LLM briefing, entity resolution, graph analytics, evidence DAG, GNN training, operations agent | Docker + Caddy HTTPS :443 | Workstation, GPU optional |
| Field | Redundant feed collection, briefing pull, LLM proxy, push to sensor nodes | venv HTTP :8002 | Laptop or mini-PC |
| Sensor | Environmental sensors, offline portal, edge autonomy briefings, mesh relay | SSH SCP (no HTTP backend) | Raspberry Pi 4 |
Data flow:
- Primary generates 24h briefing → Field node pulls via
GET /api/node/pull(ETag/304, SHA-256 integrity, conflict detection, geo-sharded entity delta) - Sensor node pushes telemetry → Primary ingests via
POST /api/node/ingest(HMAC-signed, Ed25519) - Sensor node pulls briefing → edge autonomy fallback when hub unreachable >6h (13 rule categories, no LLM required)
- GNN gradients exchanged via
POST /api/mesh/gradient(FedAvg, ~10 MB diffs, DP-SGD) - All nodes register via
POST /api/mesh/heartbeat— status atGET /api/mesh/status
Each node type can run independently. The primary node is a complete workstation without field or sensor nodes.
| Component | Key file |
|---|---|
| Ed25519 Node Identity | mesh_identity.py |
| Epidemic Gossip | mesh_gossip.py |
| Signature Validation | predictions + briefing |
| Geo-Sharding (10×10 deg) | ftm_shard.py |
| Edge Autonomy (13 rules) | offgrid-raspi/edge_autonomy.py |
| Federated GNN (FedAvg) | mesh_federated.py |
| DP on Gradients | dp_sgd.py |
| Repo | Role | When you need it |
|---|---|---|
| intelshed (this repo) | Primary + field node — Cesium globe, 47 feeds, LLM briefing, entity graph, evidence DAG, mesh API | Workstation or laptop deployment |
| offgrid-raspi | Sensor node — portal, sensors, edge autonomy, offline services | Raspberry Pi edge deployment |
Pi repo is a git submodule at offgrid-raspi/. Field nodes clone intelshed and use the edge-node branch.
UI — EDGE dashboard: DATA → EDGE for live sensor node stats (CPU/RAM/disk/load/uptime, room DHT, services, mesh) and 24h sparklines. Header shows EDGE ONLINE when the sensor node has pushed within 5 minutes.
Docker mode (add -k for self-signed cert, -H "X-API-Key: <key>" if set):
| Check | Command |
|---|---|
| Health | curl -sk https://localhost/api/health/ping |
| Nodes | curl -sk -H "X-API-Key: <key>" https://localhost/api/nodes |
| Trust probes | curl -sk -H "X-API-Key: <key>" https://localhost/api/trust |
| Anomaly status | curl -sk -H "X-API-Key: <key>" https://localhost/api/anomalies/iso/status |
| Graph stats | curl -sk -H "X-API-Key: <key>" https://localhost/api/intel/graph/stats |
| Pi pull payload | curl -sk -H "X-Node-Token: <token>" https://localhost/api/node/pull |
| Mesh status | curl -sk -H "X-API-Key: <key>" https://localhost/api/mesh/status |
| Feature flags | curl -sk -H "X-API-Key: <key>" https://localhost/api/admin/flags |
venv mode: replace https://localhost with http://127.0.0.1:8002 and omit -k.
Independent benchmark v2 (2026-07-29) — 5/5 hard assertions passed, 5/5 promotion gates passed:
| Metric | Value | Gate |
|---|---|---|
| Corroboration rate | 10.1% | >10% |
| Trust separation (KS) | 0.667 | >0.5 |
| Novel rate | 84.1% | >5% |
| Embedding p95 | 109 ms | <500 ms |
| Anisotropy | 0.893 | — |
| Stability (CV, 3 runs) | 0.07 | — |
Do I need a GPU?
No. GPU acceleration (BGE reranker, NVIDIA NIM, GNN training) is optional. Without GPU, the system falls back to CPU embeddings and Ollama-only LLM inference. All 48 feeds work without GPU.
Do I need any paid API keys?
No. 37 of 48 feeds work without any API key. Only Cesium Ion (free tier) is required for terrain/imagery. NVIDIA NIM is optional (free tier available). See What works without keys.
Can I run this on a laptop?
Yes. The venv mode runs on any Python 3.12+ machine. Docker mode requires Docker Desktop. The full stack (backend + frontend) runs on 8 GB RAM without GPU. The Pi edge node runs on a Raspberry Pi 4 with 4 GB RAM.
Is this an attack tool?
No. IntelShed is for situational awareness and decision support. It aggregates publicly available information, correlates it, and generates briefings. It does not perform active scanning, exploitation, or any offensive operations.
How is this different from Maltego / Palantir?
IntelShed is fully open-source (MIT), self-hosted, and free. No commercial API is a hard dependency. It includes a 3D globe, 48 live feeds, an 8-phase multi-agent orchestrator, W3C PROV evidence DAG, and a 3-node mesh with federated GNN. Maltego and Palantir are commercial products with closed source.
- 1–8 of 47 feeds may be stale or warning at any given time — upstream APIs rate-limit, change endpoints, or go offline. The system degrades gracefully but stale feeds reduce briefing coverage.
- The 8-phase multi-agent orchestrator adds 15–35 s latency per query. It is opt-in via
WORLDBASE_CHAT_AGENTIC=1. - GPU acceleration (BGE reranker, NVIDIA NIM, GNN training) is optional. Without GPU, the system falls back to CPU embeddings and Ollama-only LLM inference.
- Dark web search requires a running Tor proxy (
intelshed-torcontainer or system Tor) for.onionengines. Clearnet engines work without Tor. - PostgreSQL is the primary FtM backend. DuckDB remains as a sidecar but some analytical queries may be slower on large graphs.
- Vibe Graphing (MEGA PLAN Phase 3) — code complete (
vibe_graphing.py, 241 LOC), tests pending. ftm_query.py(70 KB) is the last undecomposed god-object. All others were split into focused submodule packages (2026-07-20).
Contributions are welcome. See CONTRIBUTING.md for development setup, code style, testing, and pull request guidelines.
Please read SECURITY.md before reporting security vulnerabilities.
MIT — see LICENSE for full text.
Optional features (document intel ingest, external feeds, downloaded models) may pull in separate third-party licenses. GLiREL relation extraction is disabled by default (CC BY-NC-SA). See THIRD_PARTY_NOTICES.md and docs/INTEL_INGEST.md.