v2.0.0 - llms.txt-based Discovery

What's New in v2.0.0

This release is a major overhaul that switches to using Mintlify's llms.txt file as the authoritative source for documentation page discovery.

Key Features

• llms.txt-based discovery - Automatically discovers all 105+ documentation pages from the official index
• Clear progress indicators - See exactly which page is being scraped and progress percentage
• Timeout handling - 30-second timeout per page prevents hanging
• New index command - Preview documentation structure without scraping
• New --no-embeddings flag - Skip semantic search setup for faster scrapes
• Version flag - --version shows current version

Improvements

• Simplified dependencies (6 core packages vs 14 before)
• Semantic search is now optional
• Cleaned up codebase, removed 1,000+ lines of unused code
• Fixed SQLite datetime deprecation warnings
• Updated all supporting files and documentation
• Removed obsolete files (update_scheduler.py, install-base44-tool.sh)

Breaking Changes

• Minimum Python version is now 3.10+
• Semantic search dependencies must be installed separately if needed

Quick Start

git clone https://github.com/uricorn/base44-docs-tool.git
cd base44-docs-tool
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
playwright install chromium
python3 base44_docs_scraper.py scrape --no-embeddings

Stats

• 105 pages discovered and scraped
• 107,000+ words indexed
• 11 documentation sections
• ~4 minutes for full scrape