WebAPI Generator

Automatically generate APIs from websites by analyzing HTTP traffic using AI.

Overview

This tool captures HTTP requests from a website, analyzes them using OpenAI, and generates a standardized API with documentation. It bridges the gap between web scraping and API development by automatically inferring API structures from HTTP traffic.

Features

• Website Scraping & Request Recording

  • Crawl websites using wget to extract API requests
  • Intercept live HTTP traffic using a proxy server
  • Analyze static website files to detect API endpoints

• AI-Powered Analysis

  • Use OpenAI to analyze and categorize HTTP requests
  • Group similar requests into logical API endpoints
  • Infer parameter types and response structures

• API Specification Generation

  • Convert analyzed requests into OpenAPI 3.0 specifications
  • Generate comprehensive API documentation
  • Create client SDKs in multiple programming languages

Installation

# Clone the repository
git clone https://github.com/yourusername/webapi-generator.git
cd webapi-generator

# Create a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install OpenAPI Generator (optional, for SDK generation)
npm install @openapitools/openapi-generator-cli -g

Configuration

Create a .env file in the project root directory with the following content:

OPENAI_API_KEY=your_openai_api_key_here

Usage

Proxy Mode

Capture HTTP requests using a proxy server:

python -m src.main proxy --port 8080 --web-port 8081

1. Set up your browser to use the proxy (typically at http://localhost:8080)
2. Browse the target website to capture requests
3. Press Ctrl+C to stop the proxy when finished

Crawl Mode

Automatically crawl a website and extract API requests:

python -m src.main crawl --url https://example.com --depth 3 --generate-sdks

This will:

• Download the website using wget
• Extract potential API requests from HTML and JavaScript files
• Analyze the requests using OpenAI
• Generate an OpenAPI specification
• Optionally create client SDKs

Analyze Mode

Analyze previously captured requests:

python -m src.main analyze --input-file captured_requests.json --api-title "Example API" --generate-sdks

Use this mode when you already have a JSON file containing HTTP requests.

Web UI Mode

Run the web UI for a more user-friendly experience:

python -m src.main web --host 127.0.0.1 --port 8000

Then open your browser at http://127.0.0.1:8000 to access the web interface, which provides:

• File upload for request analysis
• Website crawler
• API specification generator
• SDK generation options

Output

The tool generates the following outputs in the specified output directory:

• OpenAPI Specification: YAML or JSON file describing the API
• Client SDKs: Code libraries for various programming languages (Python, JavaScript, Java, Go, C#)
• Analyzed API Structure: JSON file containing the detailed API analysis
• Extracted Requests: JSON file containing the captured HTTP requests

How It Works

1. Request Capture: Collect HTTP requests through proxy or crawling
2. AI Analysis: Process requests with OpenAI to identify patterns and structures
3. Endpoint Mapping: Group similar requests into logical API endpoints
4. Parameter Inference: Detect path, query, and body parameters with their types
5. Response Modeling: Extract response structures and status codes
6. OpenAPI Generation: Convert the analyzed structure into a standard specification
7. SDK Generation: Create client libraries based on the specification

Project Structure

📂 webapi-generator/
 ├── README.md
 ├── requirements.txt
 ├── src/
 │   ├── __init__.py
 │   ├── proxy_capture.py   # Proxy-based request capture
 │   ├── run_proxy.py       # Script to run the proxy
 │   ├── ai_analyzer.py     # AI-powered request analysis
 │   ├── openapi_generator.py  # OpenAPI specification generation
 │   ├── crawler.py         # Website crawler
 │   ├── sdk_generator.py   # Client SDK generation
 │   ├── web_ui.py          # Web interface
 │   └── main.py            # CLI interface
 └── .gitignore

Limitations

• The quality of the generated API depends on the coverage of captured requests
• Not all authentication methods may be correctly detected
• Complex request/response structures might require manual refinement
• Large websites may require significant OpenAI API usage

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License