A comprehensive toolkit for collecting and working with Semantic Scholar's research data through both the Datasets API and Graph API. This repository provides scripts and guides for downloading venue catalogs, paper datasets, and interacting with Semantic Scholar's APIs effectively.
- Python 3.7+
- Semantic Scholar API key (optional for basic usage, required for datasets and higher rate limits - get one at semanticscholar.org/product/api)
# Clone and setup
git clone <your-repo>
cd semantic-scholar-api
# Create virtual environment
python -m venv .venv
# Activate virtual environment
# Windows PowerShell:
.venv\Scripts\Activate.ps1
# Windows CMD:
.venv\Scripts\activate.bat
# Linux/Mac:
source .venv/bin/activate
# Install dependencies
pip install requests pandas tqdm# PowerShell
$env:S2_API_KEY = "your_api_key_here"
# Or edit your scripts directly
api_key = "your_api_key_here"Semantic Scholar provides two main APIs:
Purpose: Download complete datasets (papers, authors, venues, etc.) Best for: Bulk data analysis, research datasets, comprehensive studies Data size: Millions of records per dataset Rate limits: Requires API key for download links, 1 request/second rate limit
Purpose: Search and retrieve specific papers, authors, or venues Best for: Targeted searches, real-time queries, specific paper analysis Data size: Up to 1,000 results per query (10,000 with bulk search) Rate limits: 100 requests/5 minutes (without API key), 1 request/second with API key
| Dataset | Description | Size | Files |
|---|---|---|---|
papers |
Core paper metadata (title, authors, date, etc.) | 200M+ records | 30 files (~1.5GB each) |
authors |
Author information and affiliations | 75M+ records | 30 files (~100MB each) |
publication-venues |
Journal and conference details | 200K+ venues | 1 file |
citations |
Citation relationships between papers | 2.4B+ citations | 30 files (~8.5GB each) |
abstracts |
Paper abstracts | 100M+ records | 30 files (~1.8GB each) |
s2orc_v2 |
Full-text papers with structure | 16M+ records | Multiple files (~6GB each) |
embeddings-specter_v2 |
Paper embeddings (768-dim vectors) | 120M+ records | Multiple files (~28GB each) |
import requests
# Get all available release dates
response = requests.get("https://api.semanticscholar.org/datasets/v1/release/")
releases = response.json()
print("Available releases:", releases[-5:]) # Show last 5 releases# Get datasets in latest release
response = requests.get("https://api.semanticscholar.org/datasets/v1/release/latest")
data = response.json()
print("Available datasets:", data['datasets'])api_key = "your_api_key_here"
headers = {"x-api-key": api_key}
# Get download links for a specific dataset
dataset_name = "publication-venues" # or "papers", "authors", etc.
response = requests.get(
f"https://api.semanticscholar.org/datasets/v1/release/latest/dataset/{dataset_name}",
headers=headers
)
download_info = response.json()
print(f"Files to download: {len(download_info['files'])}")
for file_info in download_info['files']:
print(f"- {file_info['name']} ({file_info['size']} bytes)")def download_file(url, filename):
response = requests.get(url, stream=True)
total_size = int(response.headers.get('content-length', 0))
with open(filename, 'wb') as file:
downloaded = 0
for chunk in response.iter_content(chunk_size=8192):
if chunk:
file.write(chunk)
downloaded += len(chunk)
if total_size > 0:
percent = (downloaded / total_size) * 100
print(f"\rProgress: {percent:.1f}%", end='')
print(f"\nDownloaded: {filename}")
# Download each file
for file_info in download_info['files']:
download_file(file_info['url'], file_info['name'])python collect_venues.py --output data/venues.jsonl --stats-csv data/venue_stats.csv# Download first file only for testing
python -c "
import requests
api_key = 'your_key_here'
headers = {'x-api-key': api_key}
response = requests.get('https://api.semanticscholar.org/datasets/v1/release/latest/dataset/papers', headers=headers)
files = response.json()['files']
print(f'Found {len(files)} paper files. Download first file:')
print(files[0]['url'])
"import requests
api_key = "your_api_key_here"
headers = {"x-api-key": api_key}
# Search for papers
params = {
"query": "machine learning",
"fields": "title,authors,year,publicationVenue,openAccessPdf",
"limit": 100
}
response = requests.get(
"https://api.semanticscholar.org/graph/v1/paper/search",
params=params,
headers=headers
)
papers = response.json()
print(f"Found {papers['total']} papers")
for paper in papers['data'][:5]:
print(f"- {paper['title']} ({paper['year']})")params = {
"query": "neural networks",
"fields": "title,year,citationCount",
"year": "2020-2024", # Papers from 2020 to 2024
"limit": 50
}params = {
"query": "venue:NIPS", # Papers from NIPS conference
"fields": "title,year,authors",
"limit": 100
}params = {
"query": "author:Hinton",
"fields": "title,year,citationCount",
"limit": 50
}# Use bulk search for >1,000 results (up to 10,000 total)
url = "https://api.semanticscholar.org/graph/v1/paper/search/bulk"
params = {
"query": "deep learning",
"fields": "title,year,publicationVenue"
}
response = requests.get(url, params=params, headers=headers)
data = response.json()
print(f"Estimated total: {data['total']}")
all_papers = []
# Paginate through results using token
token = data.get('token')
all_papers.extend(data['data'])
while token:
# Get next page using token
params['token'] = token
response = requests.get(url, params=params, headers=headers)
data = response.json()
all_papers.extend(data['data'])
token = data.get('token') # Will be None when no more results
print(f"Retrieved {len(all_papers)} papers total")import pandas as pd
import json
# Load venue data
venues_df = pd.read_json('data/venues.jsonl', lines=True)
print(venues_df.head())
print(f"Total venues: {len(venues_df)}")
# Analyze venue types
venue_types = venues_df['type'].value_counts()
print("Venue types:", venue_types)# View first few records
head -n 5 data/venues.jsonl
# Count total records
wc -l data/venues.jsonl
# Filter journals only (requires jq)
jq 'select(.type == "journal")' data/venues.jsonl | head -5# Analyze paper publication years
with open('papers.jsonl', 'r') as f:
years = []
for line in f:
paper = json.loads(line)
if paper.get('year'):
years.append(paper['year'])
import matplotlib.pyplot as plt
plt.hist(years, bins=50)
plt.title('Papers by Publication Year')
plt.xlabel('Year')
plt.ylabel('Number of Papers')
plt.show()Problem: Script can't find venue datasets
Solution: Use exact dataset name publication-venues (with hyphen)
Problem: API key issues Solutions:
- Verify API key is correct
- Check API key has dataset access permissions
- Ensure API key is properly formatted in headers
Problem: Download links expire after ~1 hour Solution: Get fresh download links before downloading:
# Always get fresh links right before downloading
response = requests.get(dataset_url, headers=headers)
download_info = response.json()
# Then immediately download filesProblem: Too many API requests (100 requests per 5 minutes without API key) Solutions:
- Get an API key for higher limits
- Add delays between requests:
time.sleep(3)# 3+ seconds for unauthenticated - Use bulk endpoints for large queries
- Implement exponential backoff
import time
from datetime import datetime, timedelta
class RateLimiter:
def __init__(self, calls_per_minute=60):
self.calls_per_minute = calls_per_minute
self.calls = []
def wait_if_needed(self):
now = datetime.now()
# Remove calls older than 1 minute
self.calls = [call_time for call_time in self.calls
if now - call_time < timedelta(minutes=1)]
if len(self.calls) >= self.calls_per_minute:
sleep_time = 60 - (now - self.calls[0]).seconds
if sleep_time > 0:
time.sleep(sleep_time)
self.calls.append(now)
# Usage
# Use 20 calls/minute for unauthenticated, higher for authenticated
limiter = RateLimiter(calls_per_minute=20) # Conservative for unauthenticated
for query in queries:
limiter.wait_if_needed()
response = requests.get(api_url, params=query, headers=headers)import requests
import json
def download_venues():
api_key = "your_api_key_here"
headers = {"x-api-key": api_key}
# Get download info
response = requests.get(
"https://api.semanticscholar.org/datasets/v1/release/latest/dataset/publication-venues",
headers=headers
)
download_info = response.json()
# Download venue file
for file_info in download_info['files']:
file_response = requests.get(file_info['url'])
with open('venues.gz', 'wb') as f:
f.write(file_response.content)
print("Downloaded venues.gz")
download_venues()def search_recent_ai_papers():
api_key = "your_api_key_here"
headers = {"x-api-key": api_key}
params = {
"query": "artificial intelligence",
"fields": "title,authors,year,publicationVenue,citationCount,openAccessPdf",
"year": "2024-",
"limit": 100
}
response = requests.get(
"https://api.semanticscholar.org/graph/v1/paper/search",
params=params,
headers=headers
)
papers = response.json()
# Save to file
with open('recent_ai_papers.json', 'w') as f:
json.dump(papers, f, indent=2)
print(f"Found {len(papers['data'])} recent AI papers")
return papers['data']
papers = search_recent_ai_papers()def get_paper_citations(paper_id):
api_key = "your_api_key_here"
headers = {"x-api-key": api_key}
# Get papers that cite this paper
response = requests.get(
f"https://api.semanticscholar.org/graph/v1/paper/{paper_id}/citations",
params={"fields": "title,year,authors", "limit": 1000},
headers=headers
)
citations = response.json()
return citations['data']
# Example usage
paper_id = "204e3073870fae3d05bcbc2f6a8e263d9b72e776" # BERT paper
citing_papers = get_paper_citations(paper_id)
print(f"BERT has been cited by {len(citing_papers)} papers"){
"paperId": "string",
"title": "string",
"authors": [{"authorId": "string", "name": "string"}],
"year": 2024,
"publicationVenue": {"id": "string", "name": "string", "type": "journal"},
"citationCount": 42,
"openAccessPdf": {"url": "string"},
"abstract": "string"
}{
"id": "string",
"name": "string",
"type": "journal|conference",
"issn": "string",
"alternate_names": ["string"],
"url": "string"
}- Fork the repository
- Create a feature branch
- Add your improvements
- Submit a pull request
This project is licensed under the MIT License. Semantic Scholar data is licensed under ODC-BY.
- Graph API Base:
https://api.semanticscholar.org/graph/v1/ - Datasets API Base:
https://api.semanticscholar.org/datasets/v1/ - Recommendations API Base:
https://api.semanticscholar.org/recommendations/v1/
- Without API Key: 100 requests per 5-minute window
- With API Key: Higher limits (exact limits not publicly documented)
- Bulk Search: Can retrieve up to 10,000 results total per query
Dataset sizes and availability may change with each release. The numbers provided are based on the 2025-09-04 release and should be verified for current releases.
- Semantic Scholar API Documentation
- Graph API Documentation
- Datasets API Documentation
- Semantic Scholar FAQ
- Open Data Platform Paper
Need help? Check the Troubleshooting section or open an issue.