fix: bound knowledge-base ingestion

[GautamSharma99]

Summary

Add resource limits and cancellation support to knowledge-base discovery and document extraction.

Previously, recursive discovery accumulated every supported path without bounding directory depth, entry count, or document count. Documents
were then read completely into memory without enforcing their reported size, aggregate input size, or extracted-output size. PDF extraction
also processed every page without a page limit.

A large or adversarial knowledge base could therefore consume excessive memory, CPU, and temporary disk space. Cancellation was only checked
outside directory traversal and could not interrupt PDF processing.

Changes

Bounded discovery

Directory traversal now uses an asynchronous directory iterator instead of loading each directory's complete entry list.

The following limits are enforced:

• 128 supported documents
• 4,096 discovered filesystem entries
• 16 nested directory levels

The abort signal is checked:

• before opening directories;
• while iterating directory entries;
• before and after document metadata operations;
• after completing each directory.

Duplicate source paths are canonicalized and skipped before repeated discovery.

Bounded input

Knowledge-base input is limited to:

• 8 MiB per document
• 32 MiB across all documents

File metadata is checked during discovery so oversized inputs are rejected before temporary output is created or document contents are read.

Documents are opened with the no-follow boundary where supported and read in bounded 64 KiB chunks. Limits are rechecked against the opened
file and during reading, preventing a file-growth race from bypassing the initial metadata check.

Abort signals are checked between reads.

Bounded extraction

Extracted text is limited to:

• 8 MiB per document
• 32 MiB across the knowledge base

The aggregate limit also bounds the amount of extracted text written to the temporary knowledge-base directory.

DOCX extraction checks the declared uncompressed size of word/document.xml before decompression and observes cancellation before and after
extraction.

Bounded and cancellable PDFs

PDF documents are limited to 512 pages.

PDF extraction now:

• checks cancellation during loading and between pages;
• tracks extracted UTF-8 bytes as pages are processed;
• destroys the PDF loading task or document when cancellation occurs;
• preserves the original abort reason instead of wrapping it as a malformed-PDF error.

Documentation

The README now documents all knowledge-base limits so CLI behavior and user expectations match the implementation.

Security impact

This prevents untrusted or unexpectedly large knowledge bases from causing unbounded:

• recursive filesystem traversal;
• document accumulation;
• file reads and memory allocation;
• PDF page processing;
• extracted-text memory usage;
• temporary disk usage.

It also makes scan cancellation effective during knowledge-base discovery and PDF extraction.

Tests

Added regression coverage for:

• excessive document count;
• excessive directory nesting;
• oversized individual inputs;
• aggregate input limits;
• oversized extracted documents;
• aggregate extracted-text limits;
• excessive PDF page counts;
• cancellation during recursive discovery;
• existing PDF, DOCX, symlink, permission, cleanup, and validation behavior.

Verification performed:

• pnpm run types
• pnpm run format
• pnpm run build
• focused knowledge-base and API tests
• pnpm run test

Full test result:

• 411 passed
• 5 expected platform/integration skips
• 0 failed

[GautamSharma99]

Hey @mldangelo-oai @ianw-oai , could someone take a look at the knowledge-base resource limits and cancellation changes? I’d especially appreciate feedback on whether the chosen limits feel reasonable for real-world usage. Thanks!

[ianw-oai]

Thanks for putting this together. I don’t think this addresses a real problem for us, and the added complexity isn’t worth it, so I’m going to close this.