crawly: the literature assistant (v2.2.1)

🕷️ crawly: the literature assistant — Release v2.2.1

crawly (crawly: the literature assistant) is an end-to-end academic research assistant for discovering scientific papers across 8+ official Open Access databases (OpenAlex, Unpaywall, Semantic Scholar, arXiv, Europe PMC, Crossref, DOAJ, HAL, OpenAIRE), downloading full-text PDFs within a strict Legal Open-Access framework by default, exporting BibTeX citations, and generating AI-powered literature reviews using local self-hosted LLMs (Ollama, LM Studio, vLLM) or cloud APIs (OpenAI, Gemini, Anthropic, OpenRouter, Groq, DeepSeek).

📦 Zero-Setup Download Options (No pip install Required)

• crawly-universal-windows-linux.zip: Universal cross-platform bundle containing crawly (Linux/WSL/macOS launcher), crawly.bat (Windows launcher), crawly.pyz (self-contained archive with pypdf, rich, requests, beautifulsoup4), and annotated config.json.
• crawly-linux-x86_64: Native standalone Linux / WSL executable binary.
• crawly-windows-x86_64.exe: Native standalone Windows executable binary.
• config.json: External configuration file with default values and documentation for all AI providers and academic endpoints.

🐍 PyPI Package

• pip install crawly-ai: Install directly from PyPI into any Python 3.9+ environment.

🛡️ Quality & Compliance

• 100% Automated Test & Code Coverage Verified (2261 / 2261 statements).
• Strict Legal Open-Access Default with mandatory user risk-acceptance prompts before any optional third-party fallback.

Full Changelog: v2.2.0...v2.2.1

crawly: the literature assistant (v2.2.0)

🕷️ crawly: the literature assistant — Release v2.2.0

crawly (crawly: the literature assistant) is an end-to-end academic research assistant for discovering scientific papers across 8+ official Open Access databases (OpenAlex, Unpaywall, Semantic Scholar, arXiv, Europe PMC, Crossref, DOAJ, HAL, OpenAIRE), downloading full-text PDFs within a strict Legal Open-Access framework by default, exporting BibTeX citations, and generating AI-powered literature reviews using local self-hosted LLMs (Ollama, LM Studio, vLLM) or cloud APIs (OpenAI, Gemini, Anthropic, OpenRouter, Groq, DeepSeek).

📦 Zero-Setup Download Options (No pip install Required)

• crawly-universal-windows-linux.zip: Universal cross-platform bundle containing crawly (Linux/WSL/macOS launcher), crawly.bat (Windows launcher), crawly.pyz (self-contained archive with pypdf, rich, requests, beautifulsoup4), and annotated config.json.
• crawly-linux-x86_64: Native standalone Linux / WSL executable binary.
• crawly-windows-x86_64.exe: Native standalone Windows executable binary.
• config.json: External configuration file with default values and documentation for all AI providers and academic endpoints.

🐍 PyPI Package

• pip install crawly-ai: Install directly from PyPI into any Python 3.9+ environment.

🛡️ Quality & Compliance

• 100% Automated Test & Code Coverage Verified (2261 / 2261 statements).
• Strict Legal Open-Access Default with mandatory user risk-acceptance prompts before any optional third-party fallback.

Full Changelog: v2.1.3...v2.2.0

crawly: the literature assistant (v2.1.2)

🕷️ crawly: the literature assistant — Release v2.1.2

crawly (crawly: the literature assistant) is an end-to-end academic research assistant for discovering scientific papers across 8+ official Open Access databases (OpenAlex, Unpaywall, Semantic Scholar, arXiv, Europe PMC, Crossref, DOAJ, HAL, OpenAIRE), downloading full-text PDFs within a strict Legal Open-Access framework by default, exporting BibTeX citations, and generating AI-powered literature reviews using local self-hosted LLMs (Ollama, LM Studio, vLLM) or cloud APIs (OpenAI, Gemini, Anthropic, OpenRouter, Groq, DeepSeek).

📦 Zero-Setup Download Options (No pip install Required)

• crawly-universal-windows-linux.zip: Universal cross-platform bundle containing crawly (Linux/WSL/macOS launcher), crawly.bat (Windows launcher), crawly.pyz (self-contained archive with pypdf, rich, requests, beautifulsoup4), and annotated config.json.
• crawly-linux-x86_64: Native standalone Linux / WSL executable binary.
• crawly-windows-x86_64.exe: Native standalone Windows executable binary.
• config.json: External configuration file with default values and documentation for all AI providers and academic endpoints.

🛡️ Quality & Compliance

• 100% Automated Test & Code Coverage Verified (1883 / 1883 statements).
• Strict Legal Open-Access Default with mandatory user risk-acceptance prompts before any optional third-party fallback.

Full Changelog: v2.1.1...v2.1.2

crawly: the literature assistant (v2.1.1)

🕷️ crawly: the literature assistant — Release v2.1.1

crawly (crawly: the literature assistant) is an end-to-end academic research assistant for discovering scientific papers across 8+ official Open Access databases (OpenAlex, Unpaywall, Semantic Scholar, arXiv, Europe PMC, Crossref, DOAJ, HAL, OpenAIRE), downloading full-text PDFs within a strict Legal Open-Access framework by default, exporting BibTeX citations, and generating AI-powered literature reviews using local self-hosted LLMs (Ollama, LM Studio, vLLM) or cloud APIs (OpenAI, Gemini, Anthropic, OpenRouter, Groq, DeepSeek).

📦 Zero-Setup Download Options (No pip install Required)

• crawly-universal-windows-linux.zip: Universal cross-platform bundle containing crawly (Linux/WSL/macOS launcher), crawly.bat (Windows launcher), crawly.pyz (self-contained archive with pypdf, rich, requests, beautifulsoup4), and annotated config.json.
• crawly-linux-x86_64: Native standalone Linux / WSL executable binary.
• crawly-windows-x86_64.exe: Native standalone Windows executable binary.
• config.json: External configuration file with default values and documentation for all AI providers and academic endpoints.

🛡️ Quality & Compliance

• 100% Automated Test & Code Coverage Verified (1883 / 1883 statements).
• Strict Legal Open-Access Default with mandatory user risk-acceptance prompts before any optional third-party fallback.

Full Changelog: v2.1.0...v2.1.1

crawly: the literature assistant (v2.1.0)

🕷️ crawly: the literature assistant — Release v2.1.0

crawly (crawly: the literature assistant) is an end-to-end academic research assistant for discovering scientific papers across 8+ official Open Access databases (OpenAlex, Unpaywall, Semantic Scholar, arXiv, Europe PMC, Crossref, DOAJ, HAL, OpenAIRE), downloading full-text PDFs within a strict Legal Open-Access framework by default, exporting BibTeX citations, and generating AI-powered literature reviews using local self-hosted LLMs (Ollama, LM Studio, vLLM) or cloud APIs (OpenAI, Gemini, Anthropic, OpenRouter, Groq, DeepSeek).

📦 Zero-Setup Download Options (No pip install Required)

• crawly-universal-windows-linux.zip: Universal cross-platform bundle containing crawly (Linux/WSL/macOS launcher), crawly.bat (Windows launcher), crawly.pyz (self-contained archive with pypdf, rich, requests, beautifulsoup4), and annotated config.json.
• crawly-linux-x86_64: Native standalone Linux / WSL executable binary.
• crawly-windows-x86_64.exe: Native standalone Windows executable binary.
• config.json: External configuration file with default values and documentation for all AI providers and academic endpoints.

🛡️ Quality & Compliance

• 100% Automated Test & Code Coverage Verified (1883 / 1883 statements).
• Strict Legal Open-Access Default with mandatory user risk-acceptance prompts before any optional third-party fallback.

Full Changelog: v2.0.0...v2.1.0

crawly: the literature assistant (v2.0.0) — Multi-Source Open-Access Search, PDF Downloader & AI Literature Review

🕷️ crawly: the literature assistant — v2.0.0 Major Release

crawly (crawly: the literature assistant) v2.0.0 transforms the original Google Scholar & Sci-Hub utility into a unified, legal-first, AI-powered academic research assistant that works out-of-the-box on both Windows and Linux / WSL without requiring end users to install Python dependencies.

• Repository: https://github.com/lokeshsk/crawly
• Interactive Documentation & Command Builder: https://lokeshsk.github.io/crawly/
• Maintainer: Lokesh Kumar (crawly.theliteratureassistant@gmail.com)

---

✨ What's New in v2.0.0

1. 🌐 8+ Legal Open-Access Academic Databases (Default)

Discover and deduplicate research papers across official open scholarly APIs out-of-the-box:

• OpenAlex
• Unpaywall (DOI-based Legal Open-Access resolver)
• Semantic Scholar
• arXiv
• Europe PMC / PubMed Central
• Crossref
• DOAJ (Directory of Open Access Journals)
• HAL Open Archive
• OpenAIRE Research Graph
• (Optional) Google Scholar & third-party mirror fallback — disabled by default and guarded by an explicit Legal Warning & Non-Liability Risk Acceptance prompt.

2. 📄 Multi-Stage Open-Access PDF Downloader (downloader.py / crawly download)

• Automatically resolves and downloads verified full-text PDFs (%PDF- header validation) from publisher Open-Access links, Unpaywall, arXiv, Europe PMC, OpenAlex, Semantic Scholar, DOAJ, HAL, and OpenAIRE.
• Exports clean CSV datasets and BibTeX (.bib) citation files ready for Overleaf, Zotero, Mendeley, and JabRef.

3. 🧠 3-Mode AI Literature Review Engine (Local & Cloud LLMs)

Powered by pypdf full-text extraction and configurable via config.json:

• Mode 1 — Per-Paper Structured Summaries: Extracts Objective, Methodology, Key Findings, and Limitations for each paper into AI_Paper_Summaries_<dataset>.md and populates the AI_Summary CSV column.
• Mode 2 — Cross-Paper Literature Review Synthesis: Generates a comparative Markdown table and thematic synthesis across all papers in AI_Literature_Review_<dataset>.md.
• Mode 3 — Citation-Grounded RAG Q&A: Interactive and CLI question-answering over your downloaded PDFs with [Paper #N, Page P] citations.
• Supported AI Providers:
  • Self-Hosted / Local: Ollama (http://localhost:11434), LM Studio (http://localhost:1234/v1), vLLM / llama.cpp
  • Cloud APIs: OpenAI, Google Gemini, Anthropic Claude, OpenRouter, Groq, DeepSeek
  • Offline Fallback: Built-in Extractive NLP & BM25 Engine (extractive) requiring zero internet or API keys.

4. ⚙️ External config.json & Modern Progress Bars

• Zero hardcoded settings: all AI providers, models, base URLs, API keys, temperatures, timeouts, and endpoints live in config.json with built-in _documentation labels.
• Rich terminal UI dashboard with multi-column progress bars (spinner, progress bar, percentage, M/N counter, elapsed time, and ETA).

5. 🤖 100% Automated Test & Code Coverage Quality Gate (crawly-bot)

• Complete unit test suite in tests/ achieving 100% code coverage (1,883 / 1,883 statements across crawly.py, downloader.py, and crawly_core/*).
• Automated crawly-bot GitHub Actions workflow blocks any Pull Request that fails a test or drops below 100% coverage and instructs the contributor on how to fix it.

---

📦 Zero-Setup Download Options (No pip install Required)

Release Asset	Platform	How to Run
crawly-universal-windows-linux.zip	Windows & Linux / WSL / macOS	Unzip and run crawly.bat on Windows or ./crawly on Linux/WSL
crawly-linux-x86_64	Linux / WSL (64-bit)	chmod +x crawly-linux-x86_64 && ./crawly-linux-x86_64
crawly-windows-x86_64.exe	Windows (64-bit)	Double-click or run .\crawly-windows-x86_64.exe in PowerShell/CMD
crawly.pyz	Any OS with Python 3.9+	python crawly.pyz (bundles pypdf, rich, requests, bs4 inside)
config.json	All Platforms	Place next to the executable to customize AI models, keys, and settings

---

⚠️ Legal Warning & Non-Liability Disclaimer

The developers and contributors of crawly (crawly: the literature assistant) do NOT recommend, endorse, or encourage the use of Sci-Hub, shadow libraries, or automated scraping of websites that prohibit automated access. By default, crawly operates strictly within a legal framework using official Open Access APIs. Any optional feature beyond official Open Access APIs is disabled by default ("use_scihub_fallback": false) and used strictly at the user's own sole risk and legal responsibility, with zero developer liability.