Skip to content

feat: integrate uncover multi-engine search, CDN/IP filtering, and consolidate Censys credentials#82

Merged
samugit83 merged 1 commit intosamugit83:masterfrom
L4stPL4Y3R:feature/osint-tools-integration
Mar 31, 2026
Merged

feat: integrate uncover multi-engine search, CDN/IP filtering, and consolidate Censys credentials#82
samugit83 merged 1 commit intosamugit83:masterfrom
L4stPL4Y3R:feature/osint-tools-integration

Conversation

@L4stPL4Y3R
Copy link
Copy Markdown
Contributor

@L4stPL4Y3R L4stPL4Y3R commented Mar 30, 2026

Summary

  • Integrate ProjectDiscovery uncover as a multi-engine OSINT search tool across the recon pipeline, AI agent, and webapp UI — supports Shodan, Censys, FOFA, ZoomEye, Netlas, CriminalIP, Quake, Hunter, PublicWWW, HunterHow, Google, Onyphe, and Driftnet engines
  • Add CDN/Cloudflare/CloudFront IP filtering (ip_filter.py) applied to all OSINT enrichment modules, preventing non-routable and CDN IPs from consuming API credits
  • Consolidate Censys credentials from 4 fields (API ID, API Secret, Platform Token, Org ID) to 2 fields (API Token + Organization ID), migrating censys_enrich.py from deprecated Search API v2 to Platform API v3
  • Harden Criminal IP enrichment with proper error classification (auth failure, credit exhaustion, rate limit), single-warning-then-stop behavior, and accurate skip counting

Type of Change

  • New feature
  • Refactor (no behavior change)

Component(s)

  • webapp (Next.js)
  • recon-orchestrator (Python)
  • agent (Python)

How to Test

  1. Start dev environment: docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d postgres neo4j recon-orchestrator kali-sandbox agent webapp
  2. Push schema: docker compose exec webapp npx prisma db push
  3. Open Settings > API Keys — verify only 2 Censys fields appear (API Token + Organization ID)
  4. Enter API keys for available OSINT tools and save
  5. Create a project, enable Uncover + OSINT modules in Project Settings > OSINT Enrichment
  6. Run a recon pipeline — verify uncover runs as GROUP 2b (before port scan), CDN IPs are filtered, and results appear in the graph
  7. Run tests: python3 -m pytest tests/test_censys_enrich.py tests/test_uncover_enrich.py tests/test_criminalip_enrich.py tests/test_graph_db_refactor.py -v

Checklist

  • I have tested this change locally with docker compose
  • I have not included real-world target data
  • My commits follow Conventional Commits
  • I have read and agree to the DISCLAIMER.md

Key Changes by File

New Files

File Purpose
recon/ip_filter.py CDN/non-routable IP filtering shared across all OSINT modules
recon/uncover_enrich.py Uncover Docker integration: provider-config generation, result parsing, pipeline merging
tests/test_uncover_enrich.py 20 unit tests covering config, dedup, extraction, queries, merging, expansion
webapp/prisma/migrations/20260329120000_consolidate_censys_keys/ Drop legacy censys_api_id and censys_api_secret columns

Modified Files (31 total)

  • Webapp: Prisma schema (10 new API key fields + uncover settings), API route, Settings UI (unified Censys fields), OsintEnrichmentSection (uncover toggle)
  • Pipeline: project_settings.py (key wiring), main.py (GROUP 2b uncover phase), all *_enrich.py modules (CDN filtering), censys_enrich.py (v2→v3 migration)
  • Agent: tools.py (UncoverToolManager + CensysToolManager v3), orchestrator.py (key wiring), prompts/tool_registry.py
  • Graph: osint_mixin.py (update_graph_from_uncover method)
  • Infrastructure: entrypoint.sh + container_manager.py (uncover image pull)
  • Tests: Updated for new key names, CDN filtering, auth error handling

…nsolidate Censys credentials

- Add ProjectDiscovery uncover integration across pipeline, agent, and UI
  with configurable max results and per-engine API key management
- Add CDN/Cloudflare/CloudFront IP filtering (ip_filter.py) to all OSINT
  enrichment modules (Shodan, Censys, FOFA, Netlas, OTX, VirusTotal, ZoomEye)
- Harden Criminal IP enrichment with proper error classification, credit
  exhaustion detection, and single-warning-then-stop behavior
- Consolidate 4 Censys credential fields into 2 (API Token + Org ID) and
  migrate censys_enrich.py from deprecated Search API v2 to Platform API v3
- Add 10 new API key fields in Global Settings for uncover engines (Quake,
  Hunter, PublicWWW, HunterHow, Google, Onyphe, Driftnet, Censys Platform)
- Wire uncover as GROUP 2b target expansion phase in recon pipeline (before
  port scan) with Neo4j graph storage via update_graph_from_uncover mixin
- Add UncoverToolManager to agentic layer with Docker-based execution
- Add comprehensive unit tests (test_uncover_enrich.py) and update existing
  test suites for new key names and filtering behavior

Made-with: Cursor
@samugit83 samugit83 merged commit 5453d6d into samugit83:master Mar 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants