@rwese/pi-webfetch

Webfetch extension for pi coding agent - fetches remote URLs with browser rendering.

Features

• HTML pages → Browser rendering via agent-browser → markdown
• Plain text → Returned as-is
• Binary files → Downloaded to temp directory
• Auto-fallback → Uses static fetch with warning if browser unavailable
• Hybrid extraction → Markdown when HTML has good text ratio, text fallback otherwise

Tools

webfetch

Standard fetch - tries browser first for HTML, auto-fallback.

webfetch --url "https://example.com"

webfetch-spa

Explicit browser rendering for JavaScript-heavy pages.

webfetch-spa --url "https://reddit.com/r/example"

Options:

• waitFor - "networkidle" (default) or "domcontentloaded"
• timeout - Timeout in ms (default: 30000)

download-file

Download a file to a specific destination.

download-file --url "https://example.com/file.pdf" --destination "/path/to/save/file.pdf"

Requires: agent-browser CLI

npm i -g agent-browser && agent-browser install

How It Works

webfetch automatically:

1. Probes Content-Type via HEAD request
2. Skips browser for binary types (PDF, ZIP, images, etc.)
3. Tries agent-browser for HTML pages
4. Extracts HTML → converts to markdown via turndown
5. Falls back to text extraction if HTML quality is poor
6. Falls back to static fetch with warning if browser unavailable

Installation

npm install @rwese/pi-webfetch

Usage

## Fetched: https://example.com
- **Status**: 200
- **Processed as**: spa
⚠️ **Content extracted from body (article/main not found)**
---
[Page content here]

API

import { fetchUrl, downloadFile } from "@rwese/pi-webfetch";

// Fetch URL
const result = await fetchUrl("https://example.com");
// result.content - Array of { type: "text", text: string }
// result.details - { url, contentType, status, processedAs, browserWarning, ... }

// Download file to specific path
const download = await downloadFile("https://example.com/file.pdf", "/path/to/save.pdf");
// download.success - boolean
// download.size - file size in bytes
// download.contentType - Content-Type header

Development

npm install
npm test        # Run tests
npm run validate  # Type check + lint + test