Skip to content

crawly: the literature assistant (v2.0.0) — Multi-Source Open-Access Search, PDF Downloader & AI Literature Review

Choose a tag to compare

View all tags
@lokeshsk lokeshsk released this 13 Oct 05:05
· 18 commits to main since this release
v2.0.0
c171c19

🕷️ crawly: the literature assistant — v2.0.0 Major Release

crawly (crawly: the literature assistant) v2.0.0 transforms the original Google Scholar & Sci-Hub utility into a unified, legal-first, AI-powered academic research assistant that works out-of-the-box on both Windows and Linux / WSL without requiring end users to install Python dependencies.

✨ What's New in v2.0.0

1. 🌐 8+ Legal Open-Access Academic Databases (Default)

Discover and deduplicate research papers across official open scholarly APIs out-of-the-box:

  • OpenAlex
  • Unpaywall (DOI-based Legal Open-Access resolver)
  • Semantic Scholar
  • arXiv
  • Europe PMC / PubMed Central
  • Crossref
  • DOAJ (Directory of Open Access Journals)
  • HAL Open Archive
  • OpenAIRE Research Graph
  • (Optional) Google Scholar & third-party mirror fallback — disabled by default and guarded by an explicit Legal Warning & Non-Liability Risk Acceptance prompt.

2. 📄 Multi-Stage Open-Access PDF Downloader (downloader.py / crawly download)

  • Automatically resolves and downloads verified full-text PDFs (%PDF- header validation) from publisher Open-Access links, Unpaywall, arXiv, Europe PMC, OpenAlex, Semantic Scholar, DOAJ, HAL, and OpenAIRE.
  • Exports clean CSV datasets and BibTeX (.bib) citation files ready for Overleaf, Zotero, Mendeley, and JabRef.

3. 🧠 3-Mode AI Literature Review Engine (Local & Cloud LLMs)

Powered by pypdf full-text extraction and configurable via config.json:

  • Mode 1 — Per-Paper Structured Summaries: Extracts Objective, Methodology, Key Findings, and Limitations for each paper into AI_Paper_Summaries_<dataset>.md and populates the AI_Summary CSV column.
  • Mode 2 — Cross-Paper Literature Review Synthesis: Generates a comparative Markdown table and thematic synthesis across all papers in AI_Literature_Review_<dataset>.md.
  • Mode 3 — Citation-Grounded RAG Q&A: Interactive and CLI question-answering over your downloaded PDFs with [Paper #N, Page P] citations.
  • Supported AI Providers:
    • Self-Hosted / Local: Ollama (http://localhost:11434), LM Studio (http://localhost:1234/v1), vLLM / llama.cpp
    • Cloud APIs: OpenAI, Google Gemini, Anthropic Claude, OpenRouter, Groq, DeepSeek
    • Offline Fallback: Built-in Extractive NLP & BM25 Engine (extractive) requiring zero internet or API keys.

4. ⚙️ External config.json & Modern Progress Bars

  • Zero hardcoded settings: all AI providers, models, base URLs, API keys, temperatures, timeouts, and endpoints live in config.json with built-in _documentation labels.
  • Rich terminal UI dashboard with multi-column progress bars (spinner, progress bar, percentage, M/N counter, elapsed time, and ETA).

5. 🤖 100% Automated Test & Code Coverage Quality Gate (crawly-bot)

  • Complete unit test suite in tests/ achieving 100% code coverage (1,883 / 1,883 statements across crawly.py, downloader.py, and crawly_core/*).
  • Automated crawly-bot GitHub Actions workflow blocks any Pull Request that fails a test or drops below 100% coverage and instructs the contributor on how to fix it.

📦 Zero-Setup Download Options (No pip install Required)

Release Asset Platform How to Run
crawly-universal-windows-linux.zip Windows & Linux / WSL / macOS Unzip and run crawly.bat on Windows or ./crawly on Linux/WSL
crawly-linux-x86_64 Linux / WSL (64-bit) chmod +x crawly-linux-x86_64 && ./crawly-linux-x86_64
crawly-windows-x86_64.exe Windows (64-bit) Double-click or run .\crawly-windows-x86_64.exe in PowerShell/CMD
crawly.pyz Any OS with Python 3.9+ python crawly.pyz (bundles pypdf, rich, requests, bs4 inside)
config.json All Platforms Place next to the executable to customize AI models, keys, and settings

⚠️ Legal Warning & Non-Liability Disclaimer

The developers and contributors of crawly (crawly: the literature assistant) do NOT recommend, endorse, or encourage the use of Sci-Hub, shadow libraries, or automated scraping of websites that prohibit automated access. By default, crawly operates strictly within a legal framework using official Open Access APIs. Any optional feature beyond official Open Access APIs is disabled by default ("use_scihub_fallback": false) and used strictly at the user's own sole risk and legal responsibility, with zero developer liability.

Assets 2
Loading