Intel Scraper

A self-hosted RAG (retrieval-augmented generation) pipeline for studying cybersecurity. It scrapes news, reference docs, CTF writeups and ebooks, indexes them, and lets you ask questions that get answered using a local LLM, no cloud APIs involved.

Runs entirely on an old CPU-only homeserver (i5 6th gen, no GPU), so speed was a constant thing to work around.

Why I built this

I wanted a study tool for cybersecurity without paying for AI API access and without sending my data anywhere. This scrapes real source material (OWASP, PortSwigger, MITRE ATT&CK, security news feeds, and a few technical books) and only answers from that retrieved content, which cuts down on the model making things up compared to just asking it cold.

How it fits together

RSS feeds, reference docs, ebooks | v scraper.py / page_scraper.py / ebook_ingest.py | v SQLite (intel.db) | v embed.py (fastembed) | v Qdrant (vector store) | v query.py (retrieval + prompt building) | v Ollama (qwen2.5:1.5b, runs locally) | v CLI, web UI, or Discord bot

What it does

• Pulls from multiple sources: RSS feeds, static reference pages, CTF writeup listings, and EPUB/PDF ebooks
• Uses fastembed for embeddings instead of the full transformers stack, since that was way too slow to load on this CPU
• Filters out weak retrieval matches with a score threshold
• CLI with conversation memory, a way to save notes, and a basic quiz mode
• A small Streamlit web UI
• A Discord bot command (!ask) plus some basic server control commands
• Scrapes on a daily cron job so the dataset keeps growing

Stack

Python, SQLite, Qdrant, Ollama, fastembed, trafilatura, BeautifulSoup, Streamlit, discord.py

Getting it running

1. pip install -r requirements.txt
2. Copy .env.example to .env and fill it in
3. Start Qdrant: docker run -d -p 6333:6333 qdrant/qdrant
4. Pull a model: ollama pull qwen2.5:1.5b
5. Scrape: python scraper.py
6. Embed: python embed.py
7. Chat: python chat.py

Things I ran into

• sentence-transformers pulled in the full transformers/PyTorch stack, which took forever to import on this CPU every time. Switched to fastembed and it start up much faster.
• Uploading all embeddings to Qdrant in one request hit a payload size limit once the dataset grew past a few thousand chunks, so both the embedding step and the upload step needed to be batched.
• Just retrieving the top few matches without a score cutoff let in a lot of irrelevant chunks, mostly SEO listicle junk from some RSS feeds. Adding a minimum score threshold fixed that.