Skip to content

Joyce0615/jsfetch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

jsfetch

Fetch and render JavaScript-heavy web pages from the command line. Returns the actual rendered content instead of the empty loading shell you get from a plain HTTP request.

Uses a headless Chromium browser via Playwright to fully execute JavaScript, then extracts the rendered text or HTML.

Problem

Many modern websites (SPAs built with React, Vue, Angular, etc.) return only a JS loading shell on a plain HTTP fetch:

$ curl -s https://jobs.ashbyhq.com/decagon/95a4707d-... | grep -o '<body>.*</body>'
<body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body>

jsfetch renders the page fully, then returns what a real browser would display.

Install

Requires uv.

git clone <repo-url> && cd jsfetch
uv sync                         # creates .venv and installs deps
uv run playwright install chromium   # one-time, ~150 MB

For MCP server support:

uv sync --extra mcp

Example

Fetching an Ashby job board page that requires JS rendering:

$ uv run jsfetch https://jobs.ashbyhq.com/decagon/95a4707d-1def-481a-bb78-38a4b22b6891
# Jobs

Job not found

The job you requested was not found.

View all open positions

The same page with JSON output to inspect status and metadata:

$ uv run jsfetch https://jobs.ashbyhq.com/decagon -f json | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f'Title:  {d[\"title\"]}')
print(f'Status: {d[\"status\"]}')
print(f'Text:   {d[\"text\"][:120]}...')
"
Title:  Decagon Jobs
Status: 200
Text:   Open Positions (106)
Filters:
Reset filters
Department
All Departments
Deployment Strategists (19)
Engineering (32)...

CLI Usage

# Rendered plain text (default)
uv run jsfetch https://jobs.ashbyhq.com/decagon

# Structured JSON output
uv run jsfetch https://jobs.ashbyhq.com/decagon -f json

# Raw rendered HTML
uv run jsfetch https://jobs.ashbyhq.com/decagon -f html

# Extract a specific element via CSS selector
uv run jsfetch https://example.com -s "div.main-content"

# Extra wait time for slow-rendering pages
uv run jsfetch https://example.com --wait-after 3000

# Faster load when full network idle isn't needed
uv run jsfetch https://example.com --wait-until domcontentloaded

Options

Flag Description Default
-f, --format Output format: text, html, json text
-s, --selector CSS selector to extract a specific element whole page
--wait-until Load state: networkidle, load, domcontentloaded networkidle
--timeout Max wait time in ms 30000
--wait-after Extra ms to wait after load state is reached 0

JSON output schema

{
  "url": "https://example.com",
  "final_url": "https://example.com/redirected",
  "title": "Page Title",
  "status": 200,
  "text": "Rendered plain text...",
  "html": "<!DOCTYPE html>..."
}

MCP Server

jsfetch can run as an MCP server, exposing a jsfetch tool that LLM agents can call when they encounter JS-rendered pages.

Add to your MCP config (Claude Code ~/.claude/settings.json, Claude Desktop claude_desktop_config.json, etc.):

{
  "mcpServers": {
    "jsfetch": {
      "command": "uv",
      "args": ["--directory", "<path-to-repo>", "run", "python", "-m", "jsfetch.mcp_server"]
    }
  }
}

MCP tool parameters

Parameter Type Description Default
url string URL to fetch and render (required)
selector string CSS selector to narrow extraction null
output_format string text, html, or json text
wait_until string networkidle, load, or domcontentloaded networkidle
timeout_ms int Max wait time in ms 30000
wait_after_ms int Extra delay after load state 0

Claude Code Skill

jsfetch includes a Claude Code Agent Skill that Claude can invoke automatically when it encounters JS-rendered pages.

# Personal skill (available in all projects)
cp -r skill ~/.claude/skills/jsfetch

# Or project-only
cp -r skill .claude/skills/jsfetch

Once installed, Claude automatically uses jsfetch when a normal WebFetch returns a JS loading shell. You can also invoke it directly with /jsfetch <url>.

Project Structure

jsfetch/
├── jsfetch/
│   ├── render.py       # Core: headless Chromium rendering via Playwright
│   ├── cli.py          # CLI entry point (argparse)
│   └── mcp_server.py   # MCP server wrapper (FastMCP)
├── skill/
│   └── SKILL.md        # Claude Code Agent Skill definition
├── pyproject.toml
└── README.md

Requirements

  • uv
  • Python 3.10+ (uv handles this automatically)
  • Chromium (uv run playwright install chromium)
  • Optional: mcp[cli] for MCP server mode (uv sync --extra mcp)

About

Fetch and render JavaScript-heavy web pages using a headless browser. CLI + MCP server + Claude Code skill.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages