v3.0.3

Changelog

New Features

• Per-page website ingestion: loading a website into a knowledge base now lands one content row per page, with each row's id equal to the content_id its vectors carry, so pages can be listed, refreshed, and deleted individually. The parent row becomes the site row — named after the host, aggregating status ("9 of 10 pages loaded; failed: …"), and cascading deletes to every page and its vectors. Re-ingest is digest-driven: unchanged pages skip embedding entirely, changed pages replace only their own vectors, failed pages retry, and pages that left the sitemap are pruned. (#9856)
• Folder and file ingestion with per-file rows: a directory insert lands as a folder row owning one child row per file (nested folders flattened), with the same contract websites got — byte-digest refresh (unchanged files skip the read and the embed), failure isolation (a file whose reader raises becomes a FAILED child row with the reason and never aborts the folder), prune of files deleted from the folder, and cascade delete. An emptied or unenumerable folder keeps every previous row instead of reading as mass removal. Single-file inserts are byte-for-byte unchanged. (#9858)
• SitemapReader: discovers a site's pages per the sitemap protocol — the URL itself, robots.txt Sitemap: lines, /sitemap.xml, /sitemap_index.xml, gzip and nested indexes — with canonical dedup and a max_pages cap. It is auto-selected for bare sitemap*.xml(.gz) URLs and registered in the reader factory, so it appears in the UI reader dropdown. Each page becomes one whole-page Document, chunked exactly once, and failed pages return as data rather than aborting the read. (#9856)
• PageFetcher: a fetch seam below the URL readers. HttpxPageFetcher fetches with bounded concurrency and a redirect guard; ParallelPageFetcher resolves Parallel's keyed SDK, then its keyless MCP endpoint, then plain httpx — honoring retry-after with exponential backoff, falling back per page, and recording per-page extractor and attempts provenance. (#9856)
• PDF sitemap entries become citable page rows: HttpxPageFetcher routes application/pdf responses — and %PDF- bytes served under a wrong content type — through PDFReader. A missing pypdf surfaces as a per-page error naming agno[pdf] rather than an exception through the read. (#9858)
• KnowledgeManagementTools: the write-side operator toolkit for knowledge bases — ingest_url, ingest_text, ingest_path (file or folder), list_content (grouped by site and folder), ingest_status, and remove_content (requires confirmation by default), with sync and async variants under the same tool names, JSON envelopes, and scope="shared"|"user". (#9856, #9858)
• AgentOS knowledge routes: GET /knowledge/content?parent_id= lists a site's or folder's rows with correct totals, and POST /knowledge/content/{id}/refresh re-runs a URL or path-sourced row's ingest in the background. (#9856, #9858)

Bug Fixes

• Multi-page URL ingests orphaned their vectors on delete: WebsiteReader crawls and LLMsTxtReader wrote per-URL vector groups whose content_id matched no contents row (#6054 follow-on), so deleting the row from the Knowledge page left every page's vectors behind. Per-page content rows now own those vector groups — existing groups are adopted without re-embedding — so both per-page and cascade delete remove them. (#9856)
• Aborted legacy-row promotion retried unguarded: when promoting a legacy multi-page row to per-page rows failed partway, the retry ran without its ownership guard and could land COMPLETED beside stale searchable legacy vectors. The ownership marker is now re-stamped across the attempt, so retries stay guarded until the stale vectors are cleared. (#9860)
• Cap-truncated reads pruned live pages: a sitemap read that hit max_pages with entries remaining — the exact shape of POST /refresh — reported complete discovery, so reconciliation deleted every beyond-cap page row and its vectors. Cap truncation and never-opened index shards now mark discovery incomplete, which suppresses pruning and is reflected in the site status. (#9860)
• Remote parent_id filtering: GET /knowledge/content?parent_id=… against a RemoteKnowledge silently dropped the filter and returned the full base as a filtered-looking response; it now returns 501, matching remote refresh. (#9860)
• LightRag multi-page URL reads: the LightRag URL path now refuses multi-page reads with a clear FAILED message instead of silently ingesting only the first page, and no longer permanently mutates the shared reader's chunk flag. (#9856)

Cookbooks

• New examples: cookbook/07_knowledge/01_getting_started/05_website_per_page.py (per-page website ingestion) and cookbook/91_tools/knowledge_management_tools.py (the management toolkit, including folder ingestion). (#9856, #9858)