Skip to content

Releases

AICrawl v1.0.0

AICrawl v1.0.0 Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 04 Nov 10:32
v1.0.0
83d2d56

Full Changelog: v0.0.1-experimental.2...v1.0.0

Assets 10
Loading

AICrawl v0.0.1-experimental.2

AICrawl v0.0.1-experimental.2 Pre-release
Pre-release

Choose a tag to compare

View all tags
@github-actions github-actions released this 04 Nov 05:36
v0.0.1-experimental.2
4641c38

Full Changelog: v0.0.1-experimental.1...v0.0.1-experimental.2

Loading

AICrawl v0.0.1-experimental.1

AICrawl v0.0.1-experimental.1 Pre-release
Pre-release

Choose a tag to compare

View all tags
@Ahaduzzamankhan Ahaduzzamankhan released this 04 Nov 04:49
v0.0.1-experimental.1
41471c3

A single Windows x64 executable that gives humans and AI agents four ways to crawl permitted websites and extract structured data: CLI, embedded dashboard, REST API and MCP server.

Download

Asset Size Purpose
aicrawl-windows-x64.exe ~49 MB Self-contained executable
aicrawl-windows-x64.exe.sha256 < 1 KB SHA-256 checksum

No installer and no separate dashboard assets. Download the executable and run it.

# Verify the checksum (compare with the .sha256 asset)
Get-FileHash .\aicrawl-windows-x64.exe -Algorithm SHA256

# Check the build and start all local services
.\aicrawl-windows-x64.exe version
.\aicrawl-windows-x64.exe start

Highlights

  • One binary, four surfaces. CLI, React dashboard, REST API and MCP server share the same engine.
  • Deterministic extraction out of the box. No model and no API keys required; optionally configure an OpenAI-compatible endpoint or local Ollama.
  • Agent-ready MCP. Seven tools over stdio or streamable HTTP, with page content wrapped as untrusted data.
  • Policy-respecting crawler. robots.txt enforcement, SSRF/DNS validation on every redirect hop, domain allow/deny lists, rate limits, crawl budgets, size caps and cancellation.
  • Local-first storage. SQLite with WAL, migrations, jobs, pages, results, caches and settings.
  • Encrypted provider keys. ChaCha20-Poly1305 at rest; API keys stored as SHA-256 hashes and shown once.
  • Optional JavaScript rendering. Playwright bridge for SPAs when you need it.

Requirements

  • Windows 10 or 11, x64.
  • Optional rendering: Node.js 20+ and npx playwright install chromium.
  • No Visual Studio runtime is required for the prebuilt binary.

Known limitations

  • JavaScript-heavy pages need the optional Playwright bridge.
  • search_website searches one fetched page; use crawl_website for site-wide coverage.
  • Semantic extraction quality depends on the external provider you configure.
  • AICrawl does not bypass authentication, paywalls or CAPTCHAs.

Security

Please report vulnerabilities privately through the security policy. Do not open a public issue.

Experimental build: interfaces, CLI flags and storage may change before a stable release. MIT licensed.

Full changelog: CHANGELOG.md

Full Changelog: https://github.com/Ahaduzzamankhan/aicrawl/commits/v0.0.1-experimental.1

Loading