Skip to content

v4.2.1 — batch_scrape backwards-compat + CLI batch fix

Choose a tag to compare

@mysleekdesigns mysleekdesigns released this 18 May 04:35
· 57 commits to main since this release

Highlights

  • Retracts the v4.0.0 "breaking change" to batch_scrape defaults. A deep audit revealed it was a phantom at the MCP surface: server.js:544 always defaulted formats to ['json'], but the internal BatchScrapeSchema defaulted to ['markdown']. Because params are validated at both layers and the MCP layer wins, MCP clients (Claude Code, Cursor, the CLI) were never broken. Only direct programmatic callers of BatchScrapeTool.execute() ever saw the mismatch. v4.2.1 aligns the inner schema to ['json'] to close the latent gap. No migration needed for any caller — upgrades from 3.5.1 are clean.
  • Fixes the CLI batch command--format, --concurrency, --max-retries flags were being silently dropped because the command passed output_format / concurrency / max_retries to the tool, but the schema expects formats / maxConcurrency / jobOptions.maxRetries. Zod's strip-mode silently dropped the unknown keys.

What changed

  • src/tools/advanced/batchScrape/schema.js — inner default ['markdown']['json'] (matches MCP registration)
  • src/cli/commands/batch.js — map CLI flags to actual schema keys
  • tests/unit/tools/advanced/batchScrape.test.js — 4 new regression/contract tests
  • CHANGELOG.md — v4.0.0 entry retracted with postmortem; v4.2.1 entry added
  • IMPROVEMENT_ROADMAP_V4.md, PRD.md — annotated with the retraction

Install / upgrade

```bash
npm install -g crawlforge-mcp-server@4.2.1
```

Verification

  • 11/11 batchScrape unit tests pass (10 existing + 1 new contract test)
  • 3 new schema-default pin tests
  • 6/6 CLI integration tests pass
  • No regressions in 131-test D5.2 unit-test suite

Roadmap status

All phases of IMPROVEMENT_ROADMAP_V4.md complete: D1 ✓ D2 ✓ D3 ✓ D4 ✓ D5 ✓.