Skip to content

Enhancement: URL fetcher has no timeout, redirect limit, or max body size #22

@RobertLD

Description

@RobertLD

The fetchAndConvert() function in src/core/url-fetcher.ts calls fetch() with no:

  1. Timeout — a slow server can hang the process forever
  2. Redirect limit — could follow infinite redirects
  3. Max body size — a 10GB HTML page would be downloaded entirely into memory
  4. robots.txt respect — should check before fetching

Should add AbortSignal.timeout(30_000), validate content-length header, and cap the body read.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestsecuritySecurity related

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions