Rex_Google_Scholar_scraping

Description

This script retrieves all papers that cite a given paper (specified by its DOI) using the Semantic Scholar API and attempts to download their PDFs from arXiv. If a paper cannot be downloaded, the script records its details in a missing_papers.txt file and opens the corresponding URLs in a web browser for manual access.

Features

• Fetches citing papers for a given DOI from the Semantic Scholar API.
• Attempts to download PDFs of citing papers from arXiv.
• Saves successfully downloaded papers in the downloads/ directory.
• Logs missing papers and their URLs in missing_papers.txt.
• Provides an option to reset the downloads/ directory.

Prerequisites

• Python 3.x
• Required Python packages:
  • requests
  • arxiv
  • argparse

Installation

1. Clone this repository or download the script.
2. Install dependencies using pip:

   pip install requests arxiv argparse

Usage

Running the script

To run the script and input a DOI manually:

python script.py

The program will prompt you to enter the DOI of the target paper.

Reset the downloads directory

To delete all downloaded PDFs and reset the downloads/ directory:

python script.py --reset

Output

• Downloaded PDFs are saved in the downloads/ directory.
• If a paper cannot be downloaded, its details are logged in missing_papers.txt.

Notes

• If an API key for Semantic Scholar is available, it can be added to API_KEY in the script for better rate limits.
• The script includes a delay (time.sleep(2)) between download attempts to avoid excessive API requests.