GhostFetch Proxy

An independent, stateless microservice that acts as an HTTP relay to spoof browser TLS fingerprints and bypass aggressive Web Application Firewalls (WAFs).

Created by @atriganguly

Repository | Live Demo | Documentation

---

---

Table of Contents

1. Executive Summary
2. Problem Statement & Solution
3. Target Audience & Use Cases
4. System Architecture
5. Core Engineering Mechanics
6. Technology Stack
7. Environment Configuration
8. Installation & Quick Start
9. Operational Execution Modes
10. Data Lifecycle & Output Schema
11. Deployment & Infrastructure
12. Troubleshooting & Diagnostics
13. AI Agent Execution Boundaries
14. Support & Contributions
15. License

---

Executive Summary

GhostFetch Proxy is a high-performance, stateless HTTP microservice built to eliminate execution bottlenecks caused by aggressive WAFs, enforce scraping system stability, and automate critical data workflows without triggering anti-bot heuristics.

The system maintains operational visibility through structured JSON telemetry, offering a low-maintenance containerized infrastructure that scales efficiently while reducing operational overhead.

---

Problem Statement & Solution

The Problem

Automated systems frequently fail to extract data from modern web targets due to aggressive fingerprinting.

• System Volatility: WAFs (e.g., Cloudflare, DataDome) instantly block requests that utilize standard Python HTTP libraries (Requests, HTTPX, urllib) by analyzing TLS handshakes (JA3 hashes) and HTTP/2 frames.
• High Infrastructure Overhead: Relying on expensive residential proxy networks to bypass IP bans does not solve TLS-level blocking.
• Telemetry Gaps: Standard scraping scripts fail silently when presented with Captcha challenges or 403 Forbidden errors, leaving data engineers blind.

The Solution

GhostFetch Proxy resolves structural instability by introducing an independent, containerized TLS spoofing relay.

• Deterministic Execution: Translates standard API payloads into mathematically perfect browser footprints (e.g., Google Chrome 110).
• Cost & Overhead Reduction: Operates entirely within standard Docker containers without external SaaS dependencies.
• Audit-Ready Logging: Captures structured HTTP outputs, normalizing binary and text payloads for complete operational transparency.

---

Target Audience & Use Cases

• Technical Leadership: Provides clear visibility into system reliability and data collection infrastructure costs.
• Software & QA Engineers: Delivers a modular architecture with isolated subsystems for rapid testing and API integration.
• Data & System Operations: Guarantees reliable data collection, structured JSON exports, and predictable execution workflows against highly secured targets.

---

System Architecture

The application uses a decoupled architecture to isolate presentation, orchestration, execution, and persistent storage layers.

+-------------------+      +-------------------+      +-------------------+
|  Client App       | ---> |  GhostFetch Proxy | ---> |  Target Server    |
|  (Scraper/Engine) |      |  (FastAPI Relay)  |      |  (Cloudflare/WAF) |
+-------------------+      +-------------------+      +-------------------+
                                     |                          |
                                     v                          v
                           +-------------------+      +-------------------+
                           | Normalized Output |      | Spoofed TLS Layer |
                           | (JSON Response)   |      | (curl_cffi Engine)|
                           +-------------------+      +-------------------+

Component Breakdown

• Orchestration Layer: FastAPI handles request routing, payload validation via Pydantic, and asynchronous concurrency.
• Execution Core: curl_cffi processes the outbound requests, manipulating the TLS client hello and HTTP/2 settings to match real browsers.
• Persistence & Telemetry Layer: Normalizes inbound target responses, automatically decoding text or encoding binary assets (like images) into Base64.

---

Core Engineering Mechanics

To maintain system resilience under heavy loads or platform constraints, the application incorporates key engineering patterns:

1. Stateful Continuation & Relay

To prevent hitting execution limits, the engine operates completely statelessly. All cookies, headers, and targets are passed per-request, meaning the proxy requires zero local disk storage and can be scaled horizontally infinitely.

2. Deterministic State Locking

Execution paths are locked using strict Pydantic models that evaluate incoming network traffic and parameter validity before authorizing the downstream curl_cffi relay step.

3. Memory-Bounded Batch Processing

Data items are buffered in memory and instantly converted. If a target returns binary data (e.g., an image or PDF), the proxy dynamically detects the MIME type and encodes the payload into Base64 to prevent JSON serialization crashes.

4. Graceful Error & Failure Isolation

External API calls and network requests are wrapped in strict error boundaries, converting upstream target timeouts or proxy-level crashes into localized HTTP 502/500 JSON responses that do not interrupt the primary FastAPI loop.

---

Technology Stack

Category	Technology	Operational Purpose
Core Engine	Python 3.10+	Primary runtime environment and business logic execution.
API Framework	FastAPI	Asynchronous request handling, routing, and endpoint exposure.
Automation / Driver	curl_cffi	Low-level system interaction, network calls, and TLS impersonation.
Data Layer	Pydantic	Strict payload validation and JSON serialization.

---

Environment Configuration

System settings are managed independently of application logic through environment variables or configuration files.

Configuration Parameters Matrix

Variable Name	Type	Default Value	Description
GHOSTFETCH_HOST	String	0.0.0.0	Binding address for the Uvicorn server.
GHOSTFETCH_PORT	Integer	8080	Exposed port for the API microservice.

---

Installation & Quick Start

Prerequisites

• Python 3.10+ installed and configured in your system environment.
• Pip available on the host machine.

Step-by-Step Setup

1. Clone the Repository

   git clone [https://github.com/atriganguly/ghostfetch-proxy.git](https://github.com/atriganguly/ghostfetch-proxy.git)
   cd ghostfetch-proxy

2. Configure Environment Parameters

   cp .env.sample .env

3. Install Dependencies

   pip install -r requirements.txt

4. Launch the Core Engine

   python main.py

   The Swagger UI will be available at http://localhost:8080/docs.

---

Operational Execution Modes

The engine can be initialized under distinct operational profiles depending on performance and resource requirements:

• Single Target Relay: Send a POST request to /api/v1/fetch with a target URL and method.
• Authorized Relay: Inject custom headers and cookies arrays into the JSON payload to pass authentication state through the proxy.
• Payload Relay: Forward complex REST operations by passing a payload dictionary to execute spoofed POST/PUT commands.

---

Data Lifecycle & Output Schema

System telemetry, raw outputs, and execution logs are serialized asynchronously to JSON to prevent I/O blocking.

Primary Output Schema (/api/v1/fetch Response)

Field Name	Data Type	Field Description
status_code	Integer	The exact HTTP status returned by the target server.
headers	Dictionary	Raw response headers generated by the target server.
content_type	String	Extracted MIME type from the target response.
is_binary	Boolean	True if the body was converted to Base64 (e.g., images).
body	String	The raw HTML/JSON text, or Base64 encoded binary data.

---

Deployment & Infrastructure

Detached Server Execution

To run the primary application persistently on a remote Linux instance (e.g., Ubuntu/Debian):

nohup python main.py > /dev/null 2>&1 &

Docker Containerization

Deploy the application inside an isolated container using the provided Dockerfile setup:

FROM python:3.10-slim

ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8080

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

Build and execute the image:

docker build -t ghostfetch-proxy:latest .
docker run -d -p 8080:8080 ghostfetch-proxy:latest

---

Troubleshooting & Diagnostics

• Issue: Target system returns rate-limiting errors (HTTP 403 / 503).

  • Cause: WAFs like Cloudflare occasionally serve 503 challenges even to legitimate browsers during high load.
  • Resolution: This is a target-side issue. Check the body field of the proxy response for Captcha flags and implement retry logic in your client app.

• Issue: The proxy returns HTTP 502 Bad Gateway.

  • Cause: The curl_cffi engine failed to resolve the target URL (e.g., DNS failure, timeout).
  • Resolution: Verify the target URL format in the payload and increase the timeout parameter in your request.

• Issue: Cannot decode JSON response in client.

  • Cause: You requested a binary asset (like a PNG) and the proxy Base64 encoded it.
  • Resolution: Check the is_binary flag in the response. If True, run a Base64 decode on the body field.

---

AI Agent Execution Boundaries

Autonomous LLMs, coding agents, and automated patch routines operating on this codebase must adhere to these structural boundaries:

1. Maintain strict decoupling between configuration parameters and core execution engines.
2. Do not introduce arbitrary file writes or state tracking. The proxy must remain stateless.
3. Keep all binary/text detection heuristics safely wrapped in error boundaries.

---

Support & Contributions

This project is actively maintained to deliver reliable, open-source automation and execution infrastructure.

• Bug Reports & Feature Suggestions: https://github.com/atriganguly/support/
• Direct Enquiries: Contact @atriganguly for technical questions, contributions, or pull request reviews.

---

License

Distributed under the GNU General Public License v3.0.