Advanced Python Code Crawler

A sophisticated web crawler specifically designed to extract, analyze, and explain Python code from web sources using AI-powered analysis.

🚀 Features

• Intelligent Python Code Extraction: Automatically identifies and extracts Python code from various web sources
• AI-Powered Code Analysis: Uses LLM APIs (OpenAI/Anthropic) to generate detailed explanations for each code snippet
• Modular Architecture: Clean separation between crawling, processing, and output components
• Configurable Pipeline: End-to-end configuration for URLs, extraction rules, and processing options
• Smart Code Separation: Intelligently separates large code files into logical units (functions, classes)
• Multiple Output Formats: Saves code and explanations in organized file pairs
• Robust Error Handling: Comprehensive logging and error recovery
• Rate Limiting: Respectful crawling with configurable delays

📋 Requirements

• Python 3.8+
• API key for LLM service (OpenAI or Anthropic)

🛠️ Installation

# Clone the repository
git clone https://github.com/MagicOwO/crawler.git
cd crawler

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Set up environment variables
cp env.example .env
# Edit .env and add your API keys

⚙️ Configuration

The crawler is configured through config.yaml:

# URLs to crawl
urls:
  - "https://github.com/python/cpython/tree/main/Lib"
  - "https://docs.python.org/3/library/"
  - "https://realpython.com/"

# Data extraction settings
extraction:
  data_type: "python_code"
  python_code:
    min_lines: 5
    max_lines: 500
    code_block_patterns:
      - "```python"
      - "<code class=\"python\">"

# LLM processing
post_processing:
  enabled: true
  processors:
    - type: "llm_explanation"
      config:
        provider: "openai"  # or "anthropic"
        model: "gpt-3.5-turbo"
        api_key_env: "OPENAI_API_KEY"

# Output settings
output:
  base_directory: "data"
  file_naming:
    script_prefix: "python_script_"
    description_prefix: "description_"

🎯 Usage

Basic Usage

# Run the full pipeline
python main.py

# Use custom configuration
python main.py --config my_config.yaml

Advanced Usage

# Validate configuration
python main.py --validate-config

# Run only crawling (no LLM processing)
python main.py --crawl-only

# Set logging level
python main.py --log-level DEBUG

Environment Variables

Create a .env file with your API keys:

OPENAI_API_KEY=your_openai_api_key_here
# OR
ANTHROPIC_API_KEY=your_anthropic_api_key_here

📁 Output Structure

The crawler generates organized output:

data/
├── python_script_1.txt      # Extracted Python code
├── description_1.txt        # AI-generated explanation
├── python_script_2.txt
├── description_2.txt
├── ...
└── crawl_summary.txt        # Processing summary

Each script file includes:

• Source URL and metadata
• Clean, validated Python code
• Line count and extraction details

Each description file contains:

• Detailed code explanation
• Purpose and functionality
• Key components analysis
• Usage examples and patterns

🏗️ Architecture

src/
├── crawler/
│   ├── base_crawler.py           # Base crawling functionality
│   └── python_code_crawler.py    # Python-specific extraction
├── processors/
│   ├── base_processor.py         # Base processing interface
│   └── llm_explanation_processor.py  # LLM integration
├── output/
│   └── file_output_manager.py    # File output handling
└── orchestrator.py              # Main pipeline coordinator

🔧 Extending the Crawler

Adding New Data Types

1. Create a new crawler class inheriting from BaseCrawler
2. Implement the extract_content method
3. Register in the orchestrator

Adding New Processors

1. Create a processor class inheriting from BaseProcessor
2. Implement the process method
3. Add configuration support

Custom Output Formats

Extend FileOutputManager or create new output handlers.

📊 Example Results

Input: Python function from a tutorial website Output:

• python_script_1.txt: Clean, validated Python code
• description_1.txt: Detailed explanation including purpose, parameters, return values, and usage examples

🚨 Important Notes

• API Costs: LLM processing incurs API costs. Monitor your usage.
• Rate Limiting: Respect website rate limits and robots.txt
• Legal Compliance: Ensure you have permission to crawl target websites
• Code Validation: The crawler validates Python syntax but doesn't execute code

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

📄 License

MIT License - see LICENSE file for details.

🆘 Troubleshooting

Common Issues

"No API key found"

• Ensure your .env file contains the correct API key
• Check the environment variable name matches your config

"No Python code extracted"

• Verify the target URLs contain Python code
• Check the extraction patterns in config.yaml
• Try running with --log-level DEBUG for detailed logs

"Configuration validation failed"

• Run python main.py --validate-config for specific errors
• Check your config.yaml syntax
• Ensure all required fields are present