v1.2.0: 15 Major Features - 58% Size Reduction

Highlights

docpull v1.2.0 delivers 15 major features that dramatically improve documentation fetching efficiency. Real-world testing shows 58% size reduction (31 MB → 13 MB) when processing 1,914 documentation files.

New Features

Phase 1: Core Optimization

• Language Filtering: Auto-detect and filter by programming language
• Deduplication: SHA-256 based duplicate detection with flexible keep strategies
• Auto-Index Generation: Tree view, TOC, category-based, and statistics indexes
• Size Limits: Enforce per-file and total size constraints
• Multi-Source Configuration: YAML-based configuration for multiple documentation sources

Phase 2: Advanced Processing

• Selective Crawling: Include/exclude patterns for precise control
• Content Filtering: Remove unwanted sections from documentation
• Format Conversion: Output in Markdown, TOON, JSON, or SQLite
• Smart Naming: 4 naming strategies (full, short, flat, hierarchical)

Phase 3: Efficiency

• Metadata Extraction: Automatic metadata collection and JSON storage
• Update Detection: Skip unchanged files based on checksums
• Incremental Mode: Update only changed documentation

Phase 4: Integration

• Hooks/Plugins: Decorator-based plugin system for custom processing
• Git Integration: Automatic commits with templated messages
• Archive Mode: Create compressed archives (tar.gz, tar.bz2, tar.xz, zip)

Technical Details

• 20 new modules (3,886 lines of code)
• Full backward compatibility with v1.1.0
• All features integrated into CLI
• 145+ unit tests
• Zero syntax errors, zero linting issues

Installation

pip install --upgrade docpull

Quick Example

# Fetch Python docs with optimization
docpull https://docs.python.org/3/ ./python-docs \
  --language python \
  --deduplicate \
  --create-index \
  --max-total-size 20MB

# Multi-source with YAML config
docpull --sources-file sources.yaml

See the CHANGELOG for complete details.