Datasoft KB Assistant

Scrape → Index → Chat: a local RAG over your Confluence knowledge base, with verified citations and zero hallucinated links.

Two Windows desktop apps that together turn the Datasoft Confluence knowledge base (FxOffice, SalesIQ, Web2, Web4) into a private, offline-first AI support assistant — packaged as standalone executables that any support agent can run without touching Python.

📸 Screenshot coming soon.

🎯 The Problem

Support agents fielding questions about FxOffice, SalesIQ, and other Datasoft products have to manually search a large Confluence knowledge base spread across multiple spaces and hunt through release-note tables to find accurate answers. That search is slow, the results are noisy, and nothing stops an agent from citing a page that doesn't actually say what they claim. Wrong citations erode trust; slow lookups frustrate customers.

💡 The Solution

The KB Scraper crawls every Confluence space with Playwright and converts the raw HTML — including complex release-note tables — into a clean local library/ of markdown and JSON. The KB Chatbot then indexes that library into ChromaDB, retrieves the most relevant chunks with semantic search and CrossEncoder reranking, and asks Claude to write a grounded answer. Every URL in the answer is verified against the retrieved context before it reaches the user: hallucinated links are stripped and replaced with verified suggestions. The whole pipeline runs locally — no cloud infra, no API keys required.

✨ Features

• Verified citations, zero hallucinated links — every cited URL is checked against retrieved context; links that weren't in the source are removed and replaced with real suggestions
• Full local RAG pipeline — embeddings (all-MiniLM-L6-v2), vector store (ChromaDB), and reranking (ms-marco-MiniLM-L-6-v2) all run on-device; ~1,400+ articles → ~3,000 searchable chunks
• No API key required — Claude is called through the Claude Code CLI subprocess (OAuth login); Haiku 4.5 / Sonnet 4.6 switching and per-conversation token/cost tracking included
• Multi-turn conversations — context-aware follow-ups with full conversation history
• Learn Mode — password-gated expert-correction workflow that writes fixes back into the library to improve future answers
• Low-confidence handling — when retrieval confidence is weak the bot surfaces related articles instead of guessing
• File and screenshot uploads — attach images or files to a question for additional context
• Incremental scraping — state tracking means re-runs only fetch new or changed pages
• Ships as standalone .exes — both apps bundle with PyInstaller; non-technical users just double-click
• Seriously tested — 15+ pytest modules covering chunking, ingestion, retrieval, citation verification, multi-turn chat, learn mode, attachments, settings, and the LLM provider (with a fake provider for fully offline tests)

🛠️ Tech Stack

Python 3.11 · Playwright · ChromaDB · sentence-transformers · CrossEncoder · Claude (Haiku 4.5 / Sonnet 4.6) · PySide6 · Tkinter · PyInstaller · pytest

Layer	Technology
Scraping	Playwright (headed Chrome), Confluence REST API
Parsing	Custom HTML parsers → markdown + structured JSON
Embeddings	sentence-transformers all-MiniLM-L6-v2
Vector store	ChromaDB (persistent, local)
Reranking	CrossEncoder ms-marco-MiniLM-L-6-v2
LLM	Claude Haiku 4.5 / Sonnet 4.6 via Claude Code CLI subprocess
GUI	PySide6 (Qt) for the chatbot · Tkinter for the scraper
Packaging	PyInstaller (standalone Windows executables)
Testing	pytest with golden-QA fixtures and a fake LLM provider

🚀 Quickstart

Prerequisites

• Python 3.11+
• Claude Code CLI installed and signed in (claude login) — used as the LLM backend; no API key needed
• A Confluence site to scrape (configure your URL in scraper/config.py)

Step 1 — Scrape your knowledge base

cd scraper
pip install -r requirements.txt
playwright install chromium

python run.py        # headless scrape
# or
python gui.py        # scraper with GUI

Set your Confluence URL and space definitions in scraper/config.py and scraper/kb_config.py (replace YOUR_CONFLUENCE_URL). Scraped content lands in library/.

Step 2 — Index and chat

cd Dev/kb_chatbot
pip install -r requirements.txt

python -m kb_chatbot.ingest   # builds the ChromaDB index from library/
python -m kb_chatbot.gui      # launches the PySide6 chatbot

Step 3 — Build standalone executables (optional)

build_exe.bat            # scraper  → dist/DatasoftKBScraper.exe
build_chatbot_exe.bat    # chatbot  → dist/DatasoftKBChatbot.exe

Run the tests

pytest tests/                    # scraper test suite
pytest Dev/kb_chatbot/tests/     # chatbot test suite (15+ modules)

🧠 How It Works

┌──────────────┐     ┌──────────────┐     ┌──────────────────────────────────┐
│  Confluence  │ ──▶ │   Scraper    │ ──▶ │  library/  (markdown + metadata) │
│  help site   │     │ (Playwright) │     └────────────────┬─────────────────┘
└──────────────┘     └──────────────┘                      │
                                                           ▼
                                          ┌──────────────────────────────────┐
                                          │  ingest.py → ChromaDB chunks     │
                                          └────────────────┬─────────────────┘
                                                           │
                                                           ▼
                                          ┌──────────────────────────────────┐
                                          │  Chatbot (PySide6 desktop app)   │
                                          │  retriever  →  CrossEncoder      │
                                          │  rerank  →  grounded prompt      │
                                          │  →  Claude  →  citation verify   │
                                          │  →  answer + clickable sources   │
                                          └──────────────────────────────────┘

The scraper uses Playwright to authenticate and crawl Confluence, then custom HTML parsers turn every article and release-note table into clean markdown stored in library/. On first launch the chatbot's ingest.py chunks that library with overlap-aware splitting, embeds each chunk with all-MiniLM-L6-v2, and persists the vectors in a local ChromaDB collection. At query time the chatbot retrieves the top-k candidates by cosine similarity, reranks them with a CrossEncoder for precision, builds a grounded prompt from the winners, calls Claude via the Claude Code CLI subprocess, and then runs a citation-verification pass that discards any URL not present in the retrieved context before the answer is shown to the user.

Configuration Notes

• All Confluence URLs in this repo are placeholders (YOUR_CONFLUENCE_URL) — point them at your own site
• The Learn Mode default password is a placeholder in Dev/kb_chatbot/settings.py — change it before deploying (only its SHA-256 hash is persisted)
• Scraped content (library/), vector indexes, and chat state are gitignored — they're regenerable and stay on your machine

Project Structure

knowledge-base-assistant/
├── scraper/                     # Confluence crawler
│   ├── core.py                  #   Playwright browser + state tracking
│   ├── engine.py / kb_engine.py #   Crawl orchestration
│   ├── discovery.py             #   Space/page discovery via REST API
│   ├── config.py / kb_config.py #   Site URL, products, space definitions
│   ├── parsers/                 #   HTML → markdown/JSON parsers
│   ├── writers/                 #   Library output writers
│   └── gui.py                   #   Scraper GUI (Tkinter)
├── Dev/kb_chatbot/              # RAG chatbot
│   ├── ingest.py / chunker.py   #   Library → ChromaDB chunks
│   ├── retriever.py             #   Vector search + CrossEncoder reranking
│   ├── citations.py             #   Citation extraction + verification
│   ├── prompt.py                #   Grounded prompt construction
│   ├── chat/                    #   Orchestrator, session, learn-mode writer
│   ├── llm/                     #   Claude Code provider + fake provider
│   ├── gui.py                   #   PySide6 desktop UI
│   └── tests/                   #   15+ test modules + fixtures
├── tests/                       # Scraper test suite
├── DatasoftKBScraper.spec       # PyInstaller config (scraper)
└── DatasoftKBChatbot.spec       # PyInstaller config (chatbot)

📄 License

MIT © Abdul Raqeeb Khatri