Skip to content

v1.3.0: Rich Metadata Extraction & Simplified Profiles

Choose a tag to compare

View all tags
@zacharyr0th zacharyr0th released this 20 Nov 19:30
· 220 commits to main since this release
v1.3.0
2e3fcc1

v1.3.0: Rich Metadata Extraction & Simplified Profiles

Highlights

docpull v1.3.0 adds rich structured metadata extraction for enhanced AI/RAG integration and simplifies the profile system by focusing on the excellent generic fetcher.

New Features

Rich Metadata Extraction

  • Structured Metadata: Extract Open Graph, JSON-LD, and microdata during fetch
  • Enhanced Frontmatter: Adds author, description, keywords, images, publish dates, and more
  • AI/RAG Ready: Richer context for embeddings and retrieval systems
  • Opt-in Feature: Enabled with --rich-metadata flag or rich_metadata: true in config
  • Powered by extruct: Uses the battle-tested extruct library for extraction

Simplified Profile System

  • Streamlined Architecture: Removed 7 built-in profiles (React, Next.js, D3, Plaid, Tailwind, Bun, Turborepo)
  • Kept Stripe: Retained as reference implementation for custom profiles
  • Generic Fetcher Excellence: Works excellently for all documentation sites
  • Reduced Complexity: Less maintenance burden, simpler codebase
  • Easy Customization: Users can create custom profiles as needed

Technical Details

New Dependencies

  • Added extruct>=0.15.0 for structured metadata extraction

New Files

  • docpull/metadata_extractor.py - Rich metadata extraction module
  • tests/test_metadata_extractor.py - Comprehensive test suite (13 tests)

Updated Files

  • docpull/fetchers/base.py - Integrated rich metadata extraction
  • docpull/fetchers/generic_async.py - Added use_rich_metadata parameter
  • docpull/config.py - Added rich_metadata configuration option
  • docpull/sources_config.py - Added rich_metadata field
  • docpull/cli.py - Added --rich-metadata CLI flag
  • docpull/profiles/__init__.py - Simplified to single Stripe profile

Removed Files

  • 7 profile files (react.py, nextjs.py, d3.py, plaid.py, tailwind.py, bun.py, turborepo.py)
  • 7 fetcher implementation files (same names)

Version & Testing

  • Bumped version from 1.2.1 to 1.3.0
  • All 107 tests passing ✅
  • Zero mypy type errors ✅
  • All lint checks passing ✅

Example Usage

Rich Metadata Extraction

# Extract rich metadata during fetch
docpull https://docs.anthropic.com --rich-metadata

# Combine with other features
docpull https://stripe.com/docs --rich-metadata --create-index --language en

# Multi-source configuration
docpull --sources-file config.yaml

Enhanced Frontmatter Output

---
url: https://docs.example.com/guide
fetched: 2025-11-20
title: Getting Started Guide
description: Learn the basics of our platform
author: John Doe
keywords: [tutorial, guide, api]
image: https://docs.example.com/og-image.png
type: article
site_name: Example Docs
published_time: 2024-01-15T10:00:00Z
modified_time: 2024-01-20T15:30:00Z
---

Multi-Source Configuration with Rich Metadata

sources:
  anthropic:
    url: https://docs.anthropic.com
    rich_metadata: true  # Enable rich metadata extraction
    language: en
    create_index: true

  stripe:
    url: https://stripe.com/docs
    rich_metadata: true
    max_file_size: 200kb

Backward Compatibility

All existing workflows continue to work unchanged. Rich metadata extraction is opt-in, and the generic fetcher handles all documentation sites that previously used specific profiles.

Installation

pip install --upgrade docpull

Links

Stats: 30 files changed, +765/-867 lines

Assets 2
Loading