Feature Proposal: Integration of Institutional Subscription Databases (Scopus, WoS, ScienceDirect)

[kimbap4077-afk]

Hi @Imbad0202,

First, thank you for creating and maintaining academic-research-skills. The multi-agent orchestration and automated verification layers (Semantic Scholar, OpenAlex, Crossref, arXiv) provide a fantastic framework for maintaining academic integrity and eliminating AI hallucinations.

I would like to initiate a discussion regarding a potential feature enhancement to expand database verification and retrieval for institutional users.

---

Feature Proposal: Integration of Institutional Subscription Databases (Scopus, Web of Science, ScienceDirect, EBSCO, PubMed)

Context & Core Objective
Many academic researchers operate under university networks with active subscriptions to proprietary databases (e.g., Scopus, Web of Science, ScienceDirect, EBSCO, PubMed) accessed via institutional VPN or EZproxy. Integrating these sources would allow the pipeline to bypass paywalls for full-text PDF analysis and conduct rigorous systematic reviews (PRISMA standard) across paywalled literature.

Proposed Implementation Architecture

• Modular Database Clients: Introduce dedicated API clients (e.g., scopus_client.py, wos_client.py, pubmed_client.py) following the existing pattern of arxiv_client.py.
• Authentication & Network Routing:
  • Institutional API Keys: Support developer keys via environment variables (e.g., SCOPUS_API_KEY, WOS_API_KEY).
  • Proxy / Institutional VPN: Enable HTTP request routing through institutional VPN endpoints or EZproxy URL formats (*.ezproxy.uni.edu).
• Pipeline Integration:
  • Stage 1 (deep-research): Dispatch parallel search queries across both open-access and subscription indexes.
  • Stage 2.5 (integrity_verification_agent): Extend verification logic to cross-reference database-specific identifiers (e.g., Scopus EID, WoS Accession Number) alongside DOIs.
• Opt-in & Graceful Fallback: Keep subscription databases strictly opt-in (ARS_ENABLE_SUBSCRIPTION_DBS=1). If institutional keys or VPN routing are unavailable, the system should gracefully fall back to Semantic Scholar, OpenAlex, and Crossref without interrupting execution.
• Rate Limiting Safety: Implement adaptive request throttling and backoff logic to prevent rate-limit violations and safeguard institutional IP addresses from temporary blocks.

---

I would appreciate your feedback on the technical feasibility of this proposal within the current architecture, or whether supporting institutional proxy routing and proprietary API clients is something you would consider for future releases.

Note: I am very new to GitHub and its ecosystem, so please bear with me if I am not fully familiar with all the standard contribution conventions.

[Imbad0202]

Thanks for the suggestion. I understand that many scholars rely on subscription databases like Scopus and Web of Science in their daily work.

Short answer. Retrieval through your institutional access is fully supported, and it deliberately happens on your side rather than inside the pipeline. The verification gate stays subscription-free. PubMed is the one piece from your list I would take an issue for.

Retrieval and full text. You search and download with your own library access, exactly as you do today. A small converter then writes what you kept into a file the pipeline knows how to read. Concretely, the Material Passport accepts user-retrieved sources through literature_corpus[] (input port and three reference adapters under scripts/adapters/ in v3.6.4, corpus-first consumers in v3.6.5). Entries carry bibliographic data plus a pointer back to your own files, and the pipeline can read locally saved PDFs from those pointers. The adapter contract in academic-pipeline/references/adapters/overview.md is written so users can add their own, and a Scopus or WoS export converter can live entirely on your side of that contract.

The reason the boundary sits there. Not everyone has these subscriptions, every institution licenses a different bundle, and paywalled full text carries copyright and license terms that belong to you and your library rather than to a public tool. If ARS held keys or routed through EZproxy, it would be taking on access and rights it cannot honor for every user. Keeping retrieval in your hands means each researcher works exactly within what their own institution licensed. The adapter overview records the same line for the Zotero Web API ("auth and rate-limit management would pull ARS into data-layer territory").

Verification. The citation-existence gate uses four open indexes, Semantic Scholar, OpenAlex, Crossref, and arXiv. No key or subscription is needed for verification to work, and results have to stay reproducible by anyone. A citation that verifies on a subscribed machine and comes back unresolvable on an unsubscribed one would break that, and the gate caches resolver results locally for 90 days, which sits badly with subscription API terms. The separate Chinese-literature client (#595) faced the same question for CNKI-registered DOIs and stayed off subscription access for the same reasons. So Scopus EIDs and WoS accession numbers as verification identifiers are not planned.

PubMed is the exception. E-utilities are free for everyone, and the #595 client touches PubMed for coverage confirmation on some Chinese medical citations. If that part matters to you, please open an issue for a standalone PubMed resolver. It could improve biomedical coverage, and I would be glad to look at it with you.

[kimbap4077-afk]

Thank you very much for the detailed explanation and clear guidance regarding the data layer boundaries and licensing constraints. That makes total sense.

I would really appreciate your help with the PubMed integration. A standalone PubMed resolver that supports full retrieval and verification would be incredibly valuable for our work with biomedical and medical literature.

As suggested, I will open a new issue for this request so we can discuss the implementation details further. Thanks again for your support and willingness to collaborate on this!