scholar

Utilities for tracking citations to your Google Scholar papers. GS has no API, so these tools scrape the website using a real browser and maintain a local JSON database.

Setup

conda activate test
pip install playwright
playwright install chromium

Scripts

scholar.py — scrape & update

Fetches your GS profile, finds all papers, and collects every paper that cites each of them. Saves results to scholar_db.json.

conda run -n test python scholar.py              # update DB (default: visible browser)
conda run -n test python scholar.py --full       # force re-scrape all citations
conda run -n test python scholar.py --headless   # run without visible browser (may get blocked)

First run fetches everything and is slow — plan for GS to CAPTCHA you. A Chrome window opens; solve any CAPTCHA you see and the script continues automatically. GS will eventually rate-limit you; the script saves after each paper so re-running resumes where it left off.

Subsequent runs skip papers whose citation count hasn't changed and whose citations are fully fetched, so they're much faster.

report.py — offline reporting

Reads scholar_db.json locally — no network access needed. Every run also writes errors.tsv flagging suspicious citation records (impossible years, missing years).

conda run -n test python report.py                        # DB stats: papers, citation counts, completeness
conda run -n test python report.py --since 2024           # citing papers published in 2024 or later, full detail
conda run -n test python report.py --since 2024 --nolist  # same, counts-only summary table
conda run -n test python report.py --hist                 # horizontal-bar histogram (no other output)
conda run -n test python report.py --hist --sortby momentum  # histogram ranked by momentum

The --since filter is by the citing paper's publication year, not the date it was scraped.

All reporting modes accept --sortby to control how papers are ranked:

Metric	Description
frequency (default)	Total GS citation count
recency	Year of the most recent stored citation
distance	Most-recent-cite year minus pub year (raw longevity, one late cite dominates)
centroid	Mean cite year minus pub year (where citation mass sits in time)
longevity	Most-recent minus first-cite year (how long in circulation)
latebloomer	Median cite year minus pub year (rewards delayed recognition)
momentum	Recency-weighted mean cite year minus pub year (recent citations count more)

For any metric other than frequency, the computed score is displayed alongside the citation count in every report mode (as a dedicated Score column in table views, inline in full --since listings, and as an extra field in histogram rows). For frequency, the count is the score so no extra column is shown.

--hist prints a horizontal-bar histogram — one row per paper, bars (█) proportional to GS citation count, papers ranked top→bottom by the chosen --sortby metric. When --sortby is not frequency, the score is printed after the count. No error-check output is produced.

check_pubdates.py — data-quality checks

Runs three independent checks against scholar_db.json and prints the results. No network access needed.

conda run -n test python check_pubdates.py            # all three checks to stdout
conda run -n test python check_pubdates.py --tsv      # also write check_pubdates.tsv (inversions only)
conda run -n test python check_pubdates.py --all      # show every offending citation, not just the earliest

Check 1 — Incomplete scans: papers whose citation fetch was never finished (citations_complete = false). Sorted by shortfall (GS count − stored count) so the most under-scraped papers appear first. Re-run scholar.py to resume these.

Check 2 — Count mismatches: papers marked citations_complete = true but where the number of stored citations doesn't match GS's citation_count. Delta is signed (+ = more stored than GS reports, − = fewer). Sorted by |delta|. Usually means GS's count drifted or the scrape silently dropped a page.

Check 3 — Date inversions: papers whose recorded pub year is after one or more of their stored citations — logically impossible, usually a wrong GS year for your paper. Shows the earliest offending citation and a suggested corrected pub year. Sorted by inversion magnitude. Fix by editing the "year" field in scholar_db.json for the affected paper.

Configuration

One constant to set in report.py:

EARLIEST_PUB_YEAR = 1977   # your actual earliest publication year

This is used to detect impossible citation years (a paper claiming to cite you before you published anything). Don't derive it from the DB — GS may be missing your older papers.

Database (scholar_db.json)

{
  "author": { "name": ..., "scholar_id": ... },
  "papers": {
    "<paper_id>": {
      "title": ..., "year": ..., "citation_count": ...,
      "citations_complete": true/false,   ← false = interrupted, re-run to finish
      "last_checked": "<ISO timestamp>"
    }
  },
  "citations": {
    "<paper_id>": [
      { "title": ..., "authors": ..., "year": ..., "venue": ..., "url": ..., "first_seen": ... }
    ]
  }
}

citations_complete: false means GS cut off the scrape for that paper. Re-running scholar.py will resume those automatically.

Data quality notes

• GS year metadata is unreliable — years on both your papers and citing papers are sometimes wrong.
• GS occasionally mixes references from a paper into the list of papers citing it.
• The errors.tsv written on each report.py run captures missing years and years that predate EARLIEST_PUB_YEAR.
• GS uses non-breaking spaces (\xa0) in its author/venue/year info lines — the scraper normalises these before parsing.
• Scholar ID is hardcoded as weAbyM4AAAAJ in scholar.py.