Skip to content

feat: implement webfetch library, CLI, MCP server, and Python bindings#1

Merged
chaliy merged 2 commits intomainfrom
claude/implement-spec-tests-ci-n32Cu
Jan 14, 2026
Merged

feat: implement webfetch library, CLI, MCP server, and Python bindings#1
chaliy merged 2 commits intomainfrom
claude/implement-spec-tests-ci-n32Cu

Conversation

@chaliy
Copy link
Contributor

@chaliy chaliy commented Jan 14, 2026

Summary

  • Implements the webfetch specification with library, CLI, MCP server, and Python bindings
  • Full test coverage with 29 unit tests and 19 integration tests
  • CI pipeline with check, fmt, clippy, test, build, and docs

What

Implements the webfetch spec (specs/initial.md) including:

Library (webfetch crate):

  • WebFetchRequest/WebFetchResponse types with JSON schema support
  • HTTP client with async fetch, timeout handling, and partial content
  • HTML to markdown/text conversion with entity decoding
  • Tool builder pattern with URL allow/block lists
  • Comprehensive unit tests

CLI (webfetch-cli crate):

  • Binary name: webfetch
  • Flags: --url, --method, --as-markdown, --as-text, --user-agent, --llmtxt

MCP Server (webfetch-mcp crate):

  • JSON-RPC 2.0 over stdio
  • Exposes webfetch tool with proper schemas

Python Bindings (webfetch-python crate):

  • PyO3-based bindings
  • WebFetchRequest, WebFetchResponse, WebFetchTool classes
  • Convenience fetch() function

CI:

  • GitHub Actions workflow for check, fmt, clippy, test, build, and docs

Why

Implements the spec defined in specs/initial.md to provide a reusable webfetch tool.

How

  • Rust workspace with multiple crates
  • reqwest for HTTP client
  • Built-in HTML conversion (no external dependencies)
  • PyO3 for Python bindings
  • MCP protocol over stdio

Risk

  • Low - This is a new implementation following the spec
  • All tests pass locally

Checklist

  • Unit tests are passed
  • Integration tests (smoke tests) are passed
  • Documentation is in code (llmtxt, docstrings)
  • Specs are up to date and not in conflict
  • Linting passes (cargo clippy)
  • Formatting passes (cargo fmt)

Implements the webfetch specification with:

Library (webfetch crate):
- WebFetchRequest/WebFetchResponse types with JSON schema support
- HTTP client with async fetch, timeout handling, and partial content
- HTML to markdown/text conversion with entity decoding
- Tool builder pattern with URL allow/block lists
- Comprehensive unit tests

CLI (webfetch-cli crate):
- Binary name: webfetch
- Flags: --url, --method, --as-markdown, --as-text, --user-agent, --llmtxt

MCP Server (webfetch-mcp crate):
- JSON-RPC 2.0 over stdio
- Exposes webfetch tool with proper schemas

Python Bindings (webfetch-python crate):
- PyO3-based bindings
- WebFetchRequest, WebFetchResponse, WebFetchTool classes
- Convenience fetch() function

CI:
- GitHub Actions workflow for check, fmt, clippy, test, build, and docs

All tests pass (29 unit tests, 19 integration tests).
@chatgpt-codex-connector
Copy link

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

- Add `webfetch mcp` subcommand to run MCP server
- Add `webfetch fetch` subcommand (optional, --url also works)
- Remove separate webfetch-mcp crate
- Update CI workflow to remove webfetch-mcp artifact

Usage:
  webfetch --url <URL>           # Fetch directly
  webfetch fetch --url <URL>     # Explicit fetch subcommand
  webfetch mcp                   # Run MCP server over stdio
@chaliy chaliy merged commit f80739a into main Jan 14, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants