Substack to EPUB Converter

A Python tool for scraping Substack premium articles and converting them to EPUB format. This tool uses Playwright for authentication and Requests for efficient scraping.

Features

• Premium Authentication: Automatically login to Substack premium accounts
• Article Scraping: Scrape both free and premium articles
• EPUB Generation: Convert articles to well-formatted EPUB files
• Rich Formatting: Preserves formatting, images, and styling
• Batch Processing: Handle large collections with multiple EPUB files
• Efficient: Uses hybrid approach (Playwright + Requests)
• Local Caching: Caches images and attachments locally to avoid repeated downloads
• Smart Attachment Handling: Converts large attachments to download links to reduce EPUB size
• Enhanced Premium Content Detection: Automatically detects and handles premium-only articles
• PDF Attachment Support: Downloads and caches PDF attachments with proper metadata
• Improved Error Handling: Robust error handling and fallback mechanisms

Requirements

• Python 3.12 or higher
• Premium subscription to the target Substack site

Installation

1. Clone the repository:

git clone <repository-url>
cd Substack2Epub

2. Install dependencies:

uv sync

3. Install Playwright browsers:

uv run playwright install chromium

4. Create configuration file:

cp .env.example .env

5. Edit .env file with your credentials:

SUBSTACK_SITE_URL=https://your-substack-site.substack.com
SUBSTACK_EMAIL=your-email@example.com
SUBSTACK_PASSWORD=your-password

Usage

Basic Usage

# Convert all articles to EPUB
uv run main.py

# Limit to 50 articles
uv run main.py --limit 50

# Specify output file
uv run main.py --output my_articles.epub

Command Line Options

• --site-url: Override Substack site URL
• --email: Override email from command line
• --password: Override password from command line
• --limit: Limit number of articles to fetch from archive
• --max-articles: Maximum articles to include in final EPUB
• --output, -o: Output EPUB file path

Examples

# Convert with command line credentials
uv run main.py --site-url https://example.substack.com --email user@example.com --password mypass

# Fetch latest 100 articles but only include 30 in EPUB
uv run main.py --limit 100 --max-articles 30

Architecture

The tool uses a hybrid approach:

1. Playwright: Handles JavaScript-based login and extracts session cookies
2. Requests: Uses authenticated session for efficient API calls
3. BeautifulSoup: Parses HTML content when API is not available
4. EbookLib: Generates well-formatted EPUB files
5. Cache Manager: Handles local caching of images and attachments
6. PlaywrightEnhancedScraper: Enhanced scraper with Playwright fallback for premium content

Caching System

The tool includes a sophisticated caching system to improve performance and reduce bandwidth usage:

Image Caching

• Images are downloaded and cached locally in cache/images/
• Subsequent runs use cached images instead of re-downloading
• Cache is managed using URL hashing for consistent file naming
• Images are still embedded in EPUB files for offline reading

Attachment Caching

• Large attachments (PDF, DOC, ZIP, etc.) are cached locally in cache/attachments/
• Attachments are converted to download links in EPUB to reduce file size
• Cache information is displayed in the EPUB showing file size and cache status
• Supports multiple attachment types including documents, archives, and media files

Cache Management

• Cache index is stored in cache/cache_index.json
• Automatic cache directory creation
• Cache statistics tracking
• Easy cache clearing functionality

Benefits

• Faster subsequent runs: No need to re-download images and attachments
• Reduced bandwidth: Downloads files only once
• Smaller EPUB files: Large attachments are stored locally, not embedded
• Offline access: All cached files are available locally
• Smart storage: Uses URL hashing to avoid duplicate downloads

File Structure

Substack2Epub/
├── main.py              # Main application
├── auth.py              # Authentication module
├── scraper.py           # Article scraping module
├── playwright_scraper.py # Enhanced scraper with Playwright support
├── epub_generator.py    # EPUB generation module
├── cache_manager.py     # Local caching module
├── cache/               # Local cache directory (auto-created)
│   ├── images/          # Cached images
│   ├── attachments/     # Cached attachments
│   └── cache_index.json # Cache index file
├── pyproject.toml       # Python project configuration
├── uv.lock              # Dependency lock file
├── .env.example        # Environment configuration template
├── CLAUDE.md           # Claude Code instructions
└── README.md           # This file

Security

• Your credentials are stored locally in .env file
• Session cookies are extracted and used securely
• No data is sent to external servers
• The tool only accesses your own subscribed content

Troubleshooting

Login Issues

• Verify your credentials in .env file
• Ensure you have an active premium subscription
• Check if the site URL is correct
• Try deleting cookie files and re-authenticating

Scraping Issues

• Some sites may have rate limiting
• Try with smaller --limit values
• Check your internet connection
• Premium content detection may require multiple attempts
• Check cache directory permissions if caching fails

EPUB Generation Issues

• Ensure output directory is writable
• Check disk space for large collections
• Try with fewer articles
• Verify attachment caching is working properly

Cache Issues

• Delete cache/ directory to reset cache
• Check available disk space for cache
• Verify write permissions in cache directory
• Cache index corruption can be resolved by deleting cache/cache_index.json

Contributing

Feel free to submit issues and enhancement requests.

License

This project is for personal use only. Please respect the terms of service of the Substack platform.

Disclaimer

This tool is intended for personal archival purposes. Please ensure you have the right to access and download the content. Use responsibly and respect content creators' rights.