Scryfall Indexer with Python Bindings

A high-performance Scryfall Magic: The Gathering card indexer written in Rust with Python bindings. This allows you to call blazingly fast Rust functions from your Python code while maintaining the performance benefits of Rust.

Features

• High Performance: Core indexing and search written in Rust
• Python Integration: Easy-to-use Python API via PyO3 bindings
• Fuzzy Search: Advanced search with n-grams, metaphone matching, and Levenshtein distance
• Autocomplete: Fast prefix-based autocomplete suggestions
• Redis Backend: Efficient storage and retrieval using Redis
• Parallel Processing: Multi-threaded indexing using Rayon
• Progress Tracking: Real-time progress bars during indexing

Prerequisites

• Rust: Install from rustup.rs
• Python: 3.8 or higher
• Redis: Running Redis server (default: localhost:9999)
• maturin: Python package for building Rust extensions

Installation

1. Clone the repository:

   git clone <your-repo-url>
   cd scryfall

2. Install maturin:

   pip install maturin

3. Build the Python module:

   # For development (builds in debug mode, installs in current environment)
   maturin develop
   
   # For production (builds optimized release)
   maturin build --release

4. Install Redis (if not already installed):

   # Ubuntu/Debian
   sudo apt install redis-server
   
   # macOS
   brew install redis
   
   # Start Redis
   redis-server

Python API

Core Functions

import scryfall_indexer

# Download and index all Scryfall data
result = scryfall_indexer.download_and_index(redis_url="redis://127.0.0.1:9999")
print(result)  # "Successfully indexed 123456 cards with 789 sets"

# Search for cards with fuzzy matching
cards = scryfall_indexer.search_cards("Lightning Bolt", max_results=10)
for card in cards:
    print(f"{card['name']} - {card['oracle_id']}")

# Get autocomplete suggestions
suggestions = scryfall_indexer.get_autocomplete("light", max_results=5)
print(suggestions)  # ["Lightning Bolt", "Lightning Strike", ...]

# Get detailed card information
card = scryfall_indexer.get_card_by_oracle_id("oracle_id_here")
print(f"Name: {card['name']}")
print(f"Sets: {card['sets']}")

# Get indexing statistics
stats = scryfall_indexer.get_stats()
print(f"Cards: {stats['card_count']}, Sets: {stats['set_count']}")

Function Reference

download_and_index(redis_url=None)

Downloads the complete Scryfall database and builds search indexes.

• Parameters:
  • redis_url (optional): Redis connection string (default: "redis://127.0.0.1:9999")
• Returns: Success message string
• Time: ~5-10 minutes for full dataset

search_cards(query, max_results=None, redis_url=None)

Searches for cards using fuzzy matching algorithms.

• Parameters:
  • query: Search query string
  • max_results (optional): Maximum results to return (default: 20)
  • redis_url (optional): Redis connection string
• Returns: List of card dictionaries

get_card_by_oracle_id(oracle_id, redis_url=None)

Retrieves detailed information for a specific card.

• Parameters:
  • oracle_id: Scryfall Oracle ID
  • redis_url (optional): Redis connection string
• Returns: Card dictionary with full details

get_autocomplete(prefix, max_results=None, redis_url=None)

Gets autocomplete suggestions for a given prefix.

• Parameters:
  • prefix: Text prefix to complete
  • max_results (optional): Maximum suggestions (default: 10)
  • redis_url (optional): Redis connection string
• Returns: List of card name strings

get_stats(redis_url=None)

Returns statistics about the indexed data.

• Parameters:
  • redis_url (optional): Redis connection string
• Returns: Dictionary with card_count, set_count, and last_update

Usage Examples

Basic Search

import scryfall_indexer

# Search for Lightning Bolt
results = scryfall_indexer.search_cards("Lightning Bolt")
for card in results:
    print(f"{card['name']} ({len(card['sets'])} printings)")

Fuzzy Search

# Even with typos, fuzzy search will find matches
results = scryfall_indexer.search_cards("lghtnng bolt")  # Typo
print(f"Found {len(results)} cards despite typos")

Building a Web API

from flask import Flask, jsonify, request
import scryfall_indexer

app = Flask(__name__)

@app.route('/search')
def search():
    query = request.args.get('q', '')
    limit = int(request.args.get('limit', 20))
    
    if not query:
        return jsonify({'error': 'Query parameter required'}), 400
    
    try:
        results = scryfall_indexer.search_cards(query, limit)
        return jsonify({'results': results})
    except Exception as e:
        return jsonify({'error': str(e)}), 500

@app.route('/autocomplete')
def autocomplete():
    prefix = request.args.get('prefix', '')
    limit = int(request.args.get('limit', 10))
    
    suggestions = scryfall_indexer.get_autocomplete(prefix, limit)
    return jsonify({'suggestions': suggestions})

if __name__ == '__main__':
    app.run(debug=True)

Performance Characteristics

• Indexing: ~5-10 minutes for complete Scryfall dataset (~250k cards)
• Search: Sub-millisecond response times for most queries
• Memory: ~2-4GB Redis memory usage for full dataset
• Fuzzy Search: Handles typos, partial matches, and phonetic similarity

Development

Project Structure

scryfall/
├── src/
│   ├── main.rs          # Core Rust implementation
│   └── lib.rs           # Python bindings (PyO3)
├── Cargo.toml           # Rust dependencies
├── pyproject.toml       # Python packaging
└── example.py           # Python usage examples

Building for Development

# Install in development mode (rebuilds on changes)
maturin develop

# Run the example
python example.py

# Run original Rust binary
cargo run

Building for Production

# Build optimized wheel
maturin build --release

# Install the wheel
pip install target/wheels/scryfall_indexer-*.whl

Troubleshooting

Module Import Error

ImportError: No module named 'scryfall_indexer'

Solution: Run maturin develop to build and install the module.

Redis Connection Error

Error: Redis connection failed

Solution: Ensure Redis is running on the specified port:

redis-server --port 9999

Build Errors

If you encounter build errors, ensure you have:

• Latest Rust toolchain: rustup update
• Python development headers: python3-dev (Ubuntu) or Xcode (macOS)
• maturin: pip install -U maturin

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test with maturin develop
5. Submit a pull request

Acknowledgments

• Scryfall for providing the comprehensive MTG API
• PyO3 for excellent Rust-Python integration
• maturin for seamless building and packaging