Automated scrapers for ScienceDirect and INFORMS PubsOnLine — search papers by keyword, author, or journal, and batch-download PDFs using your institutional access.

Files ending in _en.py are full English-interface versions; the others are Chinese-interface versions. Logic is identical.

• Multiple search modes: keyword, journal browse, journal+keyword, author, ISSN, advanced (combine any criteria)
• Bulk PDF download with institutional access (university SSO / CARSI)
• Anti-bot measures: Chrome TLS fingerprint spoofing via curl_cffi, stealth JS injection, automatic rate-limit handling
• ScienceDirect: Chrome DevTools Protocol (CDP) captures PDF bytes directly — no Playwright required
• INFORMS: simple direct HTTP download; Atypon does not enforce JS challenges on PDF endpoints
• Output formats: CSV / JSON / XLSX
• Interactive wizard mode (run with no arguments)

Step 1 — Install dependencies

pip install curl_cffi websocket-client browser-cookie3 openpyxl

For INFORMS scraper, also install:

pip install beautifulsoup4 lxml

Step 2 — Run

# ScienceDirect (interactive wizard)
python3 sd_scraper_en.py

# INFORMS (interactive wizard)
python3 informs_scraper_en.py

macOS only (Chrome path is hardcoded to /Applications/Google Chrome.app).
On Linux/Windows, set CHROME_BIN in the scraper class to your Chrome executable path.

Full dependency list: requirements.txt

# Interactive wizard (recommended for first use)
python sd_scraper_en.py

# Keyword search — save metadata as XLSX
python sd_scraper_en.py -m keyword -q "machine learning" -n 100 --browser-cookies

# Keyword search + download PDFs
python sd_scraper_en.py -m keyword -q "deep learning" -n 50 --browser-cookies --download-pdfs

# Browse a journal (most recent first)
python sd_scraper_en.py -m journal -j "Energy" -n 200 --browser-cookies --sort date

# Keyword search within a specific journal
python sd_scraper_en.py -m journal_keyword -j "Renewable Energy" -q "solar cell" -n 50 --browser-cookies

# Search by author
python sd_scraper_en.py -m author -a "Zhang Wei" -n 30 --browser-cookies

# Advanced search (combine criteria)
python sd_scraper_en.py -m advanced -q "deep learning" --date 2021-2024 --type REV -n 50 --browser-cookies --download-pdfs

# Interactive wizard
python informs_scraper_en.py

# Keyword search
python informs_scraper_en.py -m keyword -q "supply chain" -n 100 --browser-cookies

# Browse a journal
python informs_scraper_en.py -m journal -j mnsc -n 200 --browser-cookies

# Specific volume/issue TOC
python informs_scraper_en.py -m toc -j mnsc -v 71 -i 3 --browser-cookies

# Keyword search + download PDFs
python informs_scraper_en.py -m keyword -q "inventory" -n 50 --browser-cookies --download-pdf

# Best login method: pop up Chrome, log in manually
python informs_scraper_en.py -m keyword -q "machine learning" -n 50 --chrome-login --download-pdf

Cloudflare blocks direct HTTP requests to PDF endpoints. The scraper works around this using Chrome DevTools Protocol (CDP):

1. Script auto-launches Chrome in debug mode (copies your existing profile — no re-login needed)
2. Navigates to the article page, establishing cookie context
3. Intercepts the PDF response bytes via Network / Fetch DevTools events
4. Writes the PDF directly to disk — no Save dialog, no Playwright required

First-time setup: open Chrome, log in to ScienceDirect via your institution (SSO/CARSI), and open at least one article PDF to confirm access. The script handles everything else automatically.

# If Chrome isn't already logged in, use this first:
python sd_scraper_en.py --open-browser-login
# Then run your actual scrape:
python sd_scraper_en.py -m keyword -q "turbine" -n 50 --browser-cookies --download-pdfs

INFORMS (Atypon platform) does not enforce JS challenges on PDF endpoints — a valid session cookie is sufficient.

# Option 1: read cookies from Chrome (must be logged in already)
python informs_scraper_en.py -m keyword -q "inventory" -n 30 --browser-cookies --download-pdf

# Option 2: pop up Chrome, log in, then auto-extract cookies
python informs_scraper_en.py -m keyword -q "inventory" -n 30 --chrome-login --download-pdf

# Option 3: member credentials (direct login)
python informs_scraper_en.py -m keyword -q "inventory" -n 30 --member 123456 --password MyPwd --download-pdf

All output goes to ./results/ (ScienceDirect) or ./informs_result/ (INFORMS) by default.

results/
└── keyword_machine_learning_20250101_120000/
    ├── keyword_machine_learning_20250101_120000.xlsx   ← metadata
    └── pdfs/
        ├── 001_Zhang_2024_Deep learning for...pdf
        ├── 002_Li_2023_Transfer learning in...pdf
        └── ...

• Institutional access required for full-text PDF download. Open-access papers can be downloaded without login.
• Rate limiting: the scraper adds random delays between requests (default 2–5 s). Do not reduce these aggressively.
• macOS only in current form. The Chrome binary path (/Applications/Google Chrome.app/...) is hardcoded. Linux/Windows users: update CHROME_BIN in the class definition.
• Cookie expiry: session cookies expire (typically days to weeks). Re-login if you encounter 403 errors.
• Use responsibly and in accordance with your institution's and the publishers' terms of service.

MIT