Granthvani Calendar Data Pipeline

Scrapes Hindu calendar data for 2026 from drikpanchang.com and stores it in a SQLite database and JSON file for use in the Granthvani app.

Data collected

Category	Fields
Panchang	Tithi, Nakshatra, Yoga, Karana, Paksha, Lunar month, Samvat
Timings	Sunrise, Sunset, Moonrise, Moonset
Muhurat	Abhijit, Brahma, Godhuli, Vijaya, and other auspicious/inauspicious periods
Choghadiya	8 day + 8 night slots with name, type, and time range
Festivals	139 Hindu festivals across the year
Vrats & Upvas	Ekadashi, Pradosh, Chaturthi, Purnima, Amavasya, Shivaratri, Durgashtami, Satyanarayana, Chandra Darshan
Planetary positions	Weekly — planet, sign, longitude, nakshatra, pada, speed, retrograde flag

Files

database.py          — SQLite schema and DB helper functions
scraper_panchang.py  — Daily panchang + choghadiya scraper
scraper_festivals.py — Festivals, vrats, and planetary positions scraper
run_scrape.py        — Main orchestration (4 phases)
resume_scrape.py     — Auto-resumes after an IP ban lifts
fetch_data.py        — Query helpers for reading data out of the DB

Generated (gitignored):

granthvani_calendar.db   — SQLite database
granthvani_calendar.json — Full year export as JSON
scrape_2026.log          — Scrape run log

Setup

pip install requests beautifulsoup4

Usage

Full scrape (first run)

python run_scrape.py --year 2026 --delay 5 --planet-step 7

Flag	Default	Description
--year	2026	Year to scrape
--delay	5.0	Base seconds between requests
--planet-step	7	Scrape planetary positions every N days
--skip-festivals	off	Skip Phase 2 (festivals)
--skip-vrats	off	Skip Phase 3 (vrats)
--skip-planets	off	Skip Phase 4 (planets)

The scraper auto-skips dates already in the database, so re-running resumes where it left off.

Resume after IP ban

python resume_scrape.py

Polls every 5 minutes until the site responds, then restarts with conservative settings.

Rate limiting

The scraper uses several measures to avoid bans:

• 5s base delay with ±50% random jitter between requests
• User-Agent rotation across 6 browser profiles
• Homepage warm-up to establish cookies before scraping
• 200–300s cooldown + session rotation every 10 days
• reCAPTCHA detection — backs off 2–4 minutes if a CAPTCHA page is served
• Exponential backoff (60s × attempt) on HTTP 403/429

Database schema

panchang            — One row per date; all tithi/nakshatra/muhurat fields
choghadiya          — 16 rows per date (8 day + 8 night slots)
festivals           — One row per festival occurrence
vrats               — One row per vrat occurrence
planetary_positions — One row per planet per scraped date