feat: structured metadata extraction (title, description, language, canonical URL, author, links, headings) #71
Description
What
Extract structured metadata from HTML pages and return it in FetchResponse. Agents currently have to re-parse markdown to get basic page info.
Fields to add
title— from<title>or<meta property="og:title">description— from<meta name="description">orog:descriptionlanguage— from<html lang="...">canonical_url— from<link rel="canonical">published_date/modified_date— from<meta>or<time>elements, JSON-LDauthor— from<meta name="author">or JSON-LDlinks— extracted list of[text, href]pairsheadings— outline/TOC as structured data
Why
Biggest bang for buck for agentic use — agents need this metadata universally and shouldn't have to re-parse converted content to get it.
Acceptance criteria
- New
metadatafield onFetchResponsewith above subfields - Extracted during HTML processing (no extra fetch)
- All fields optional
- Tests with wiremock covering extraction from realistic HTML