Skip to content

v1.0.44

Choose a tag to compare

View all tags
@github-actions github-actions released this 16 Dec 12:40
· 215 commits to main since this release
v1.0.44
a00182b

What's Changed

Changes since v1.0.7:

🚀 Features

  • feat(seo): add complete evidence blocks to all SEO rules
  • feat(cli): add shorthand keywords and category filter to seo command
  • feat(presets): add 6 new service presets with proper timeouts
  • feat(docs): redesign interactive playground with CLI parser and SEO analyzer
  • feat(ci): add install.sh script and include it in releases
  • feat(cli): add standalone binary generation support
  • feat(mcp): rename server to 'recker', add prompts support, and fix doc search
  • feat(cli): add format=json option to spider command
  • feat(cli): show help automatically when required arguments are missing
  • feat(cli): add version awareness
  • feat(yaml): add native YAML support (RFC 9512)
  • feat(core): integrate YAML and CSV into client and response
  • feat(csv): add native CSV support with streaming (RFC 4180)
  • feat(errors): add centralized error classification system
  • feat(shell): unify AI chat experience with interactive mode
  • feat(spider): integrate sitemap and robots.txt for URL discovery
  • feat(cli): improve SEO output with OpenGraph and Twitter Card details
  • feat(cli): expand CLI functionality and shell interface
  • feat(seo): expand SEO analysis with new rules and structured data
  • feat: add AI-first features and spider CLI command
  • feat(tests): drastically improve mock server and AI retry test performance
  • feat(mcp): register scrape and security tools
  • feat(mcp): add SEO tools for AI-assisted development
  • feat(seo): add detailed request timing metrics
  • feat(seo): add TTFB timing and OpenGraph display to shell
  • feat(seo): add SEO Spider for site-wide analysis
  • feat(scrape): add spider web crawler module
  • feat(seo): add CSP required directives rule
  • feat(seo): add Lighthouse a11y, best practices, and security rules
  • feat(seo): add PWA, Social Media, and Internal Linking rules
  • feat(seo): add 50+ new SEO rules across 5 categories
  • feat(seo): add i18n rules and scoring weight system
  • feat(seo): add rules engine with 70+ checks and detailed evidence
  • feat: add SEO analyzer and display presets in columns
  • feat(cli): add progress spinners for search operations
  • feat(testing): add full HTTP method support and webhook server
  • feat(benchmarks): add recker (bare) client to benchmarks
  • feat(mcp): add CLI flags to disable tool categories
  • feat(bare): add zero-overhead bare client for maximum performance
  • feat(plugins): add certificate pinning plugin
  • feat(mcp): add dynamic tool loading and network tools
  • feat(ai): add Google and Ollama providers with vector store
  • feat(benchmark): add averaged runner and API simulation benchmarks
  • feat(cli): add quiet mode and output file options
  • feat(testing): add mock servers for DNS, WHOIS, Telnet, FTP
  • feat(cli): add serve command for mock servers
  • feat(testing): add mock servers for all protocols
  • feat(hls): rewrite HLS plugin with fluent API
  • feat(client): make baseUrl optional
  • feat(presets): add cloud providers and social media presets
  • feat(presets): add Mailgun and Sinch presets
  • feat: add sparkline, system-metrics utils and improve error handling - Add sparkline.ts for CLI visualizations - Add system-metrics.ts for CPU/memory monitoring - Move ip-intel.ts to mcp/ directory - Add geoip-loader.ts for IP geolocation - Update tests for new error types - Fix various test improvements
  • feat(errors): add 16 specialized error types with standardized structure Add comprehensive error hierarchy with consistent retriable flags and suggestions: - StateError, ProtocolError, ConfigurationError, ValidationError - StreamError, DownloadError, ConnectionError, AuthenticationError - NotFoundError, UnsupportedError, ParseError, QueueCancelledError - MaxSizeExceededError, AbortError, NetworkError, TimeoutError Each error type includes: - Specific metadata fields for debugging - Protocol-specific suggestions for common issues - Consistent retriable property for retry logic
  • feat(tls): unify shell and CLI TLS output with SANs, pubkey, and extKeyUsage
  • feat(cli): add scroll history to shell with Page Up/Down and mouse scroll
  • feat(cli): add fullscreen documentation search with split panel - Add SearchPanel component with vertical split layout - Left panel: navigable search results list - Right panel: markdown preview with syntax highlighting - Keyboard navigation: arrows, page up/down, escape - Natural language search: 'how to retry?' triggers search - Add 'rek docs' CLI command for direct search - Integrate with shell '?' command
  • feat(load): improve error tracking with status codes - Add ErrorEntry type with status, message, count, lastSeen - Track errors with deduplication and count - Format common error messages (ECONNREFUSED, ETIMEDOUT, etc) - Add human-readable status text (404 -> Not Found) - Show real-time error list in dashboard during load test - Add status code badges with colors (500 red, 400 yellow) - Add background colors to colors utility (bgRed, bgGreen, etc) - Add LoadStats tests (13 tests)
  • feat(mcp): lazy download embeddings from GitHub Releases - Add embeddings-loader.ts for lazy download of embeddings - Cache embeddings in ~/.cache/recker/embeddings-{version}.json - Upload embeddings.json as GitHub Release asset in CI - Remove embeddings.json from npm package (3.5MB savings) - Add 16 tests for embeddings loader
  • feat(mcp): improve hybrid search with better keyword matching
  • feat(shell): add documentation search commands - Add ? / search command for hybrid fuzzy+semantic doc search - Add suggest command for implementation suggestions - Add example command for code examples - Natural language: input ending with ? triggers search - Lazy-loaded HybridSearch with 5min idle auto-unload for memory efficiency
  • feat(mcp): add hybrid search with semantic embeddings - Add HybridSearch combining Fuse.js (fuzzy) + pre-computed embeddings (semantic) - Add 5 MCP tools: search_docs, get_doc, code_examples, api_schema, suggest - Add fastembed for build-time embedding generation (68 docs, 384 dimensions) - Add user simulation tests (35 tests covering real-world queries) - Add math utilities: cosine similarity, levenshtein, RRF - Improve MemoryStorage with compression and memory limits - Add plugin documentation (retry, cache, auth, logger, etc.)
  • feat(mcp): add multi-transport support (stdio, http, sse) MCP Server now supports three transport modes: - stdio (default): For Claude Code and CLI integrations - Reads JSON-RPC from stdin, writes to stdout - Debug logs go to stderr - http: Simple HTTP POST endpoint - POST / for JSON-RPC requests - CORS enabled - sse: HTTP + Server-Sent Events - POST / for JSON-RPC requests - GET /sse for real-time notifications - GET /health for health checks Usage: rek mcp # stdio (for Claude Code) rek mcp -t http # HTTP server rek mcp -t sse -p 8080 # SSE server on custom port
  • feat(mcp): add MCP server for documentation access - Add MCPServer class with JSON-RPC 2.0 protocol support - Implement 2 focused tools: - search_docs: Search documentation by keyword with category filter - get_doc: Get full content of a specific doc file - Add CLI command: rek mcp to start the server - Index all markdown files from docs/ folder - Support for CORS and debug mode - Add comprehensive tests (22 tests) Usage: rek mcp # Start on port 3100 rek mcp -p 8080 # Custom port rek mcp --debug # Enable logging
  • feat(cli): add stdin pipe support and .env loading - Add stdin pipe support for request body (cat body.json | rek post api.com) - Add -e/--env flag to load .env files from current directory or custom path - Add 'env' command to interactive shell for loading .env files - Update 'vars' command to show both session and env variables - Fix URL prefix detection (httpbin.org was incorrectly matched as 'http') - Add proper Content-Type detection for piped string bodies - Mask sensitive env vars (keys, secrets, passwords, tokens) in vars output
  • feat(shell): $images now accepts optional selector - $images [selector]: scope image search to specific element - Without selector: searches entire page (including meta, link) - With selector: searches only within that element
  • feat(shell): add $beautify command to format minified JS/CSS - $beautify : download and format minified code - $beautify:save [file]: save formatted code to disk - Built-in JS/CSS parser with proper string/comment handling - Adds indentation, newlines after { } ; for readability - No external dependencies
  • feat(shell): add $unmap command to extract original sources from sourcemaps - $unmap : download and parse sourcemap, list all original files - $unmap:view : view source content by index (with syntax highlight) - $unmap:save [dir]: extract all sources to disk (removes webpack:// prefix) - No external dependencies - uses native JSON parsing
  • feat(shell): add $sourcemaps command for sourcemap discovery - Finds inline JS/CSS with sourceMappingURL comments - Infers potential .map files from external scripts/stylesheets - Shows confirmed vs inferred sourcemaps separately
  • feat(shell): add comprehensive asset extraction ($images, $scripts, $css) - Enhanced $images to find ALL images: , , CSS background-image, favicons, and meta og:image/twitter:image - Added $scripts command for JavaScript discovery (external + inline) - Added $css command for stylesheet discovery (external + inline) - Fixed flaky timing test with 5ms tolerance for timer imprecision
  • feat(shell): display OpenGraph and meta data in scrap command When loading a page with scrap, now shows: - Meta description (truncated to 100 chars) - OpenGraph data: site name, title, type, description, image, url Makes the shell experience richer for inspecting pages.
  • feat(cli): make cheerio a required dependency for better UX - Move cheerio from optional peer dependency to regular dependency - Web scraping commands now work out-of-the-box with npx recker - Still gracefully handle edge cases in shell initialization - Update help to show scraping availability status
  • feat(cli): network commands use baseUrl, change load default to realistic ## Network Commands Integration - whois, tls, dns, rdap, ping now use baseUrl when no argument provided - Example: url example.com then dns runs dns example.com ## Load Testing - Change default mode from 'throughput' to 'realistic'
  • feat(cli): Ctrl+C now exits the shell Previously Ctrl+C would clear the line and show a new prompt. Now both Ctrl+C and Ctrl+D exit the shell.
  • feat(cli): integrate scrap with base URL - Allow scrap without arguments when baseUrl is set - Scrapes the base URL directly - Relative paths still work: scrap /path
  • feat(cli): add interactive web scraping to shell ## New Commands - scrap <url> - Fetch and parse HTML document - $ <selector> - Query and count elements with CSS selector - $text <selector> - Extract text content from elements - $attr <name> <selector> - Extract attribute values - $html <selector> - Get inner HTML of first match - $links [selector] - List all links (href + text) - $images [selector] - List all images (src + alt) - $table <selector> - Extract tables as structured data ## Documentation - Added comprehensive Web Scraping section to shell docs - Command reference table - Workflow examples (product scraping, tables)
  • feat(cli): add network tools and fix cardinal dependency ## CLI Network Tools - Add whois command: domain/IP lookup with parsed fields - Add tls/ssl command: TLS certificate inspection with validity check - Add dns command: full DNS resolution (A, AAAA, MX, NS, SPF, DMARC, CAA) - Add rdap command: modern WHOIS with structured JSON - Add ping command: TCP connectivity check - Update help text with new Network Tools section ## Fix Cardinal Dependency - Make cardinal truly optional (graceful fallback to no highlighting) - Fixes issue where users had to manually install cardinal@2.0.0

🐛 Bug Fixes

  • fix: wrong version
  • fix(test): increase timeout for flaky spider crawl test
  • fix(deps): replace cheerio with css-select for lighter HTML parsing
  • fix(browser): add missing modules to browser build config
  • fix(ci): add missing source files and lint to pre-commit
  • fix(browser): correct documentation and add modern APIs
  • fix(test): correct HTML parser test expectations
  • fix(ci): use positional args for CLI version command
  • fix(cli): replace picocolors with internal colors utility
  • fix(test): respect AbortSignal in HLS cancel test mock
  • fix(cli): show specific error types in spider status
  • fix(hls): pass abort signal to all HTTP requests for proper cancellation
  • fix: improve hybrid search score combination and embeddings loading
  • fix: missing params for spider command
  • fix(shell): prevent crashes on Page Up/Down scroll navigation
  • fix(spider): improve error display in crawl results
  • fix(test): wait for WebSocket connections before closing
  • fix(cli): restore readline prompt after doc search closes
  • fix(cli): disable mouse reporting to prevent escape sequence leakage
  • fix(mcp): enable semantic search with runtime query embeddings
  • feat(bare): add zero-overhead bare client for maximum performance
  • fix(test): use RFC 5737 TEST-NET-1 for unreachable host test
  • fix(ci): use npm ecosystem for dependabot
  • fix(search): improve precision and fix UI navigation bug
  • fix(ci): use pnpm ecosystem for dependabot
  • fix(bench): improve error messages for connection pool exhaustion Add friendly error messages for: - 'expression evaluated to a falsy value' -> 'Connection pool exhausted' - AssertionError -> 'Internal assertion failed' These errors occur under high load when undici's connection pool encounters internal state inconsistencies.
  • fix(shell): use Q key instead of Escape to exit scroll mode
  • fix(shell): allow scroll keys to work while in scroll mode
  • fix(shell): consume all mouse events to prevent garbage chars
  • fix(cli): intercept mouse scroll events before readline processes them
  • fix(mcp): ensure index is ready before handling requests - buildIndex() was called in constructor but not awaited - Added ensureIndexReady() to guarantee index is built - Call ensureIndexReady() in start() before accepting requests - Fixes 'HybridSearch not initialized' error in concurrent tests
  • fix(security): remove docsify-cli and its vulnerable dependencies - Replaced docsify-cli with serve for local docs preview - Removes vulnerabilities in marked (CVE-2021-21306, CVE-2022-21680, CVE-2022-21681) - Removes vulnerability in got (CVE-2022-33987) - Removed package-lock.json (using pnpm only) - Updated $scripts and $css in scrap tip message
  • fix(shell): add automatic retry and better error handling for WHOIS - WHOIS command now retries up to 3 times with exponential backoff - Better error messages with error codes and suggestions - Handle socket close events for rate limiting detection - Show detailed error info when all retries fail
  • fix(shell): improve WHOIS/RDAP domain extraction and output - Add getRootDomain() helper to extract root domain from URL (e.g., tetis.io from www.tetis.io) - WHOIS and RDAP commands now use root domain for lookups - Improved WHOIS output: shows all fields when no important ones found - Falls back to raw response when no parsed data available

📚 Documentation

  • docs(mcp): update documentation with all 18 tools
  • docs: fix navbar paths and add lightning animation styling
  • docs: update CLI docs to use key=value syntax
  • docs(cli): add comprehensive help text for all major commands
  • docs: add CLI and MCP overview pages to getting started
  • docs: add YAML and CSV plugins to documentation
  • docs: reorganize scraping docs and enhance SEO spider documentation
  • docs: add dedicated SEO documentation section
  • docs: update SEO documentation with AI Search category and validators
  • docs: restructure and update documentation
  • docs: comprehensive documentation update for SEO and Spider features
  • docs(seo): document HTTP vs browser data source requirements
  • docs: add SEO analyzer documentation
  • docs(benchmarks): update with fresh 16-library benchmark results
  • docs(benchmarks): update with 18-library benchmark results
  • docs(mini): add comprehensive mini client documentation
  • feat(benchmarks): add recker (bare) client to benchmarks
  • docs(benchmarks): update with complete 22-client comparison results
  • feat(mcp): add CLI flags to disable tool categories
  • feat(bare): add zero-overhead bare client for maximum performance
  • docs: comprehensive documentation update
  • docs: update plugin names to use *Plugin suffix
  • docs(mcp): complete tools documentation with rek_ prefix
  • docs(mcp): add network tools documentation
  • docs: update documentation for new features
  • docs: add quiet mode and output file documentation
  • docs: remove duplicate hero image from _media folder
  • docs: add hero lightning background to cover page
  • docs: update documentation with RDAP, GeoIP, and CLI protocols
  • docs: update documentation with unified API patterns - Update README.md with zero-config examples - Add usage styles to all protocol docs (AI, WebSocket, DNS, WHOIS) - Complete API reference with all interfaces - Add tests for unified API, circuit-breaker, dedup, user-agent - Add recker.ts with lazy-loaded singleton instances - Coverage now at 90%+
  • docs: replace logo with lightning emoji
  • docs(cli): add read-modify-write pipe examples Add examples showing how to chain requests to fetch data, transform with jq, and send updates via PATCH/PUT
  • docs(mcp): add Claude Code one-liner installation command
  • docs: translate all documentation to English - Translate 9 plugin docs (overview, memory-cache, retry, circuit-breaker, cache, dedup, auth, logger, cookie-jar) - Translate 4 HTTP specialty docs (specialties index, SOAP, JSON-RPC, OData) - All documentation is now in English for international accessibility
  • docs: update benchmark results with latest measurements
  • docs: separate MCP Client and Server documentation - Split 05-mcp.md into two focused documents: - 05-mcp-client.md: Connect to external MCP servers - 06-mcp-server.md: Serve docs to AI assistants - Add ready-to-use configs for Claude Code, Cursor, Windsurf, Codex, Gemini, and Grok - Update home page with MCP Server highlight - Update sidebar navigation
  • docs: add MCP Server documentation - Document all transport modes (stdio, http, sse) - Add Claude Code integration examples - Include configuration options and usage examples - Add JSON-RPC methods reference - Restructure doc to cover both Client and Server
  • docs: add npx@latest instructions and fix broken links - Add "Quick Start (No Install)" section with npx recker@latest examples - Add npx instructions to shell docs and quick-start - Fix broken links in getting-started (concepts.md, api/client.md) - Point Next Steps to existing docs (http/, reference/)
  • docs: fix broken links, add type definitions, improve examples ## Broken Links Fixed - Fix all guides/* references to new structure (http/, protocols/, etc.) - Update README.md, docs/README.md, _navbar.md, _404.md, features.md - Fix api/README.md, migration.md, examples/README.md cross-references ## Type Definitions Added - Add WhoisResult and WhoisOptions interfaces to WHOIS docs - Add ReckerResponse interface with all methods and fields - Add ProgressEvent and SSEEvent types to responses doc ## Examples Improved - Replace Promise.all with client.batch() in recipes and WHOIS docs - Simplify chunked processing example in concurrency docs - Add stats tracking to batch examples ## Hero Tagline Updated - New tagline: "Fast as infrastructure demands. AI-ready from the first byte. Observable down to the millisecond. Resilient when everything else fails."

✅ Tests

  • test(presets): add cloud provider preset tests
  • test: improve test infrastructure and add mock server tests
  • test: add DNS propagation tests and improve embeddings-loader coverage
  • test: improve code coverage to 89.76% Add tests for UDP response streaming, cache network-first strategy, GraphQL GET method, retry hooks, XML edge cases, and more.
  • test: improve code coverage to 90%+ - Add comprehensive AI client tests (retry logic, metrics, error handling) - Add OpenAI provider error handling tests (context length, auth, overloaded) - Add Anthropic provider error handling tests (rate limit, overloaded) - Add tests for colors utility, optional-require, UDP exports - Update vitest.config.ts to exclude runtime-dependent files from coverage - Update docs with stdin pipe and .env loading features Coverage improved from 73.85% to 90.41%

🔧 Maintenance

  • chore: add suggs files to gitignore
  • chore: remove all cheerio and commander references
  • chore(release): include embeddings.json in published package
  • chore(bench): add recker-browser to comparison and update docs
  • chore(spider): update default depth=5 and maxPages=100
  • chore(spider): change default maxDepth from 3 to 4
  • chore: fix shebang and few fixes
  • chore: update lockfile after ora removal
  • chore(husky): skip pre-commit tests for version bumps
  • chore: remove simple-get and tiny-json-http from benchmarks
  • chore: remove deprecated HTTP client libraries from benchmarks
  • feat(bare): add zero-overhead bare client for maximum performance
  • chore: ignoring agents definitions
  • chore: misc improvements and fixes
  • chore: reduce npm package size by removing declaration maps
  • ci: bump actions/setup-node from 4 to 6 (#2)
  • ci: bump actions/checkout from 4 to 6 (#1)
  • ci: add dependabot config and fix docs script - Replace docsify-cli with serve (removes got vulnerability) - Add dependabot.yml for automated dependency updates - Group minor/patch updates, individual PRs for major versions
  • ci: remove duplicate docs workflow GitHub Pages already handles deployment via pages-build-deployment
  • ci: skip next release on version bump commits
  • ci: add job summary for releases
  • ci: rename alpha to next for npm dist-tag
  • chore: add recker bin alias and fix docs script
  • ci: fix contributors using GitHub API for correct usernames - Use GitHub Compare API to get actual GitHub usernames - Prevents incorrect @ mentions (e.g., @Filipe instead of @forattini-dev) - Fallback to author names without @ if API fails
  • ci: fix duplicate workflow runs - Remove redundant publish.yml (now handled by ci.yml) - Skip alpha release on version bump commits to avoid duplicate runs - Tags still trigger stable release correctly
  • ci: add GitHub Packages publishing and enhanced releases - Publish to both NPM and GitHub Packages (@forattini-dev/recker) - Generate changelog from conventional commits (feat, fix, docs, test, ci) - Create beautiful GitHub Release with: - 🚀 Features section - 🐛 Bug Fixes section - 📚 Documentation section - ✅ Tests section - 🔧 Maintenance section - 👥 Contributors list - 📦 Installation instructions
  • ci: fix workflows to use pnpm instead of npm - Replace npm ci with pnpm install --frozen-lockfile - Add pnpm/action-setup@v4 with version 9 - Enable pnpm cache in actions/setup-node - Update publish.yml to use pnpm v9 (was v8)
  • chore: move test hook from pre-push to pre-commit Tests and build now run before commit instead of before push. This provides faster feedback during development.
  • chore: removed unecessary docs

👥 Contributors

Full Changelog: v1.0.7...v1.0.44

📦 Installation

# NPM
npm install recker@v1.0.44

# PNPM
pnpm add recker@v1.0.44

# GitHub Packages
npm install @forattini-dev/recker@v1.0.44 --registry=https://npm.pkg.github.com

💻 Standalone Binaries

No Node.js required! Download the binary for your OS below.

Contributors

filipeforattini, dependabot, and 2 other contributors
Assets 7
Loading