Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

160 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

HTTP Request Client

A high-performance HTTP/2 client library written in C, with TLS 1.3 session resumption, HPACK header compression, and Browser-like TLS fingerprinting.

Features

  • HTTP/2 — full implementation: concurrent stream multiplexing over a shared connection (per-connection reader thread), HPACK dynamic table, flow control, SETTINGS/PING ACK, GOAWAY
  • TLS 1.3 — session resumption with pre_shared_key (NewSessionTicket callback)
  • TLS fingerprint — GREASE, ECH, ALPS, cert compression (Brotli), signature algorithms alignment
  • Compression — gzip, deflate, Brotli, Zstd response decompression
  • Proxy — HTTPS CONNECT tunnel with authorization
  • Session pool — thread-safe connection reuse with configurable expiration (up to 1024 concurrent sessions); concurrent same-host requests share one multiplexed connection
  • Cross-language — native bindings for Node.js (N-API), Python (cffi), Java (JNI)

Architecture

┌─────────────────────────────────────────────────┐
│                 Language Bindings               │
│     Node.js (N-API) │ Python (cffi) │ Java (JNI)│
├─────────────────────────────────────────────────┤
│             libhttp2client (shared lib)         │
├─────────────────────────────────────────────────┤
│ Http2Client → Basket → Session → RequestHandler │
│                        → ResponseHandler        │
│                        → SocketHandler          │
├─────────────────────────────────────────────────┤
│ SSLHandler (BoringSSL) │ CompressHandler        │
│ BrowserHandler (Chrome)│ UrlParser / File       │
├─────────────────────────────────────────────────┤
│ BoringSSL │ Brotli │ Zstd │ Jansson │ zlib      │
└─────────────────────────────────────────────────┘

Project Structure

├── include/            # Public headers
│   ├── Http2Client.h   # C API entry point
│   ├── Basket.h        # Request/Response/Session data structures
│   ├── SSLHandler.h    # TLS layer
│   ├── Session.h       # Connection session pool
│   ├── Compat.h        # POSIX <-> Winsock2 networking shim
│   └── ...
├── src/                # C source
│   ├── Http2Client.c   # init / request / cleanup
│   ├── Session.c       # Session pool + TLS session cache
│   ├── SSLHandler.c    # Browser-like TLS configuration
│   ├── RequestHandler.c # HTTP/2 HEADERS + DATA frames
│   ├── ResponseHandler.c # Frame parsing, HPACK decoding
│   ├── CompressHandler.c # gzip/deflate/brotli/zstd
│   └── ...
├── tests/              # C test programs
├── third_party/        # Git submodules
│   ├── boringssl/      # TLS 1.3
│   ├── brotli/         # Brotli compression
│   ├── zstd/           # Zstandard compression
│   ├── jansson/        # JSON parsing
│   └── zlog/           # Logging (optional)
├── nodejs/             # Node.js N-API binding
├── python/             # Python cffi binding
└── java/               # Java JNI binding

Prerequisites

  • CMake >= 3.29
  • C17 compiler (Clang / GCC)
  • Git (third-party dependencies are cloned automatically at configure time)
  • Windows: MSYS2 with the MinGW-w64 toolchain (see below). This codebase relies on POSIX APIs (pthread, <stdatomic.h>, BSD sockets), which MinGW-w64 provides; native MSVC is not supported.

zlib is used for gzip/deflate. On macOS/Linux the system zlib is used automatically; if it is missing it is downloaded and built from source.

Building

macOS

cmake -B build
cmake --build build -j$(sysctl -n hw.ncpu)

Linux

cmake -B build
cmake --build build -j$(nproc)

Windows (MSYS2 / MinGW-w64)

Native MSVC is not supported. Build inside the MSYS2 MinGW64 shell, whose GCC ships the POSIX headers (pthread.h, stdatomic.h, sockets) this project needs.

  1. Install MSYS2, then open the "MSYS2 MinGW64" shell and install the toolchain (NASM is required by BoringSSL):
    pacman -S --needed \
      mingw-w64-x86_64-toolchain \
      mingw-w64-x86_64-cmake \
      mingw-w64-x86_64-nasm \
      git
  2. Configure and build:
    cmake -B build -G "Ninja"
    cmake --build build -j$(nproc)
    (Use -G "MinGW Makefiles" if Ninja is not installed.)

If configuration fails with No CMAKE_ASM_NASM_COMPILER could be found, install NASM (pacman -S mingw-w64-x86_64-nasm) or pass -DOPENSSL_NO_ASM=1. See Troubleshooting.

Networking compatibility layer

The networking code is written against the POSIX BSD socket API. On Windows this is bridged to Winsock2 by include/Compat.h, so no source changes are needed per platform:

POSIX Windows (via Compat.h)
<sys/socket.h>, <netdb.h>, <arpa/inet.h> <winsock2.h>, <ws2tcpip.h>
close(fd) closeSocket(fd)closesocket
fcntl(fd, ..., O_NONBLOCK) setSocketNonBlocking / setSocketBlockingioctlsocket(FIONBIO)
errno / EINPROGRESS / ECONNREFUSED SOCKET_LAST_ERROR / SOCKET_EINPROGRESS / SOCKET_ECONNREFUSED
usleep sleepMicrosecondsSleep

WSAStartup / WSACleanup are invoked automatically inside initialiseEnv() / cleanupEnv(), and the Winsock library (ws2_32) is linked automatically on Windows. No extra setup is required.

This produces:

  • lib/shared/libhttp2client.dylib (macOS) / .so (Linux) / .dll (Windows) — shared library for language bindings
  • test_GET / test_POST — C test executables

Build Output

Artifact Path
Shared library lib/shared/libhttp2client.{dylib,so,dll}
Static library lib/static/libhttp2client.a
Test executables bin/test_GET, bin/test_POST

BoringSSL Patches

Matching a real browser's TLS ClientHello byte-for-byte requires a few capabilities that stock BoringSSL does not expose. Rather than fork the submodule, the project keeps small, self-contained patches under patches/boringssl/ and applies them to the pristine checkout at build time.

How they are applied — the apply_third_party_patches() function in CMakeLists.txt globs patches/boringssl/*.patch, applies them in filename order (0001-, 0002-, …) with git apply, and skips any patch that git apply --reverse --check reports as already applied. This runs automatically at cmake configure time, is idempotent (re-running never double-applies), and needs no manual step.

Patch Purpose
0001-tls13-configurable-cipher-order.patch Adds SSL_set_tls13_cipher_prefs(ssl, str), letting the caller advertise the TLS 1.3 cipher suites in an explicit order. Stock BoringSSL fixes the TLS 1.3 cipher order based on AES-hardware detection, which leaks the host's CPU capabilities into the fingerprint. The patch also re-enables three legacy 3DES suites (ECDHE-ECDSA-DES-CBC3-SHA, ECDHE-RSA-DES-CBC3-SHA, DES-CBC3-SHA) so the advertised cipher list can match the reference browser exactly.
0002-allow-duplicate-verify-sigalgs.patch Lets the client-advertised signature_algorithms list contain duplicate entries (e.g. rsa_pss_rsae_sha384 twice), which some browsers emit and which BoringSSL otherwise rejects. Only the verify (ClientHello) preferences allow duplicates; signing preferences still reject them, since a repeated entry there is a genuine configuration error.

The browser-specific values these patches consume (cipher list, signature algorithms, extension toggles) live in the per-profile BrowserFingerprint structs in src/BrowserHandler.c and are applied in src/SSLHandler.c.

C API

#include "Http2Client.h"

// 1. Initialize (call once at startup)
void initialiseEnv(void);

// 2. Send request (two-step, thread-safe)
//    handleRequest is a BLOCKING/synchronous call: it does not return until the
//    response arrives, the timeout elapses, or an error occurs. Concurrency is
//    achieved by calling it from multiple threads -- it may be called
//    concurrently, and same-host requests share one HTTP/2 connection and are
//    multiplexed on separate streams.
//    Step 1: get result pointer and length
//    requestJSONString: JSON config (see format below)
//    outLen: output parameter for response JSON length
//    Returns: malloc'd response JSON string, or NULL on error
char* handleRequest(const char *requestJSONString, int *outLen);

//    Step 2: copy content to your buffer (frees the source pointer)
//    basketStr: pointer returned by handleRequest
//    dest:      caller-allocated buffer (at least outLen + 1 bytes)
void getBasketContent(char *basketStr, char *dest);

// 3. Cleanup (call once at shutdown)
void cleanupEnv(void);

Blocking by design. handleRequest is synchronous — the calling thread blocks until the response is fully read, the timeout fires, or an error is returned. There is no async/callback variant in the C core. To issue requests concurrently, call handleRequest from multiple threads (it is thread-safe); same-host requests are then multiplexed over a single shared HTTP/2 connection. The language bindings build their concurrency on top of this: e.g. the Node.js binding runs each blocking call on a libuv worker thread (raise UV_THREADPOOL_SIZE for high concurrency — see nodejs/README.MD), and Python/Java use their own thread pools.

Example

#include "Http2Client.h"
#include <stdlib.h>

int main() {
    initialiseEnv();

    const char *request = "{"\n"
        "  \"method\": \"GET\","\n"
        "  \"url\": \"https://tls.peet.ws/api/all\","\n"
        "  \"connectTimeoutInMilliseconds\": 3000,"\n"
        "  \"responseReadingTimeoutInMilliseconds\": 30000,"\n"
        "  \"decompress\": 0,"\n"
        "  \"log\": 1,"\n"
        "  \"headers\": {"\n"
        "    \"host\": \"tls.peet.ws\","\n"
        "    \"user-agent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36\","\n"
        "    \"sec-ch-ua\": \"\\\"Not:A-Brand\\\";v=\\\"99\\\", \\\"Google Chrome\\\";v=\\\"145\\\", \\\"Chromium\\\";v=\\\"145\\\"\","\n"
        "    \"sec-ch-ua-mobile\": \"?0\","\n"
        "    \"accept\": \"*/*\","\n"
        "    \"sec-fetch-site\": \"same-origin\","\n"
        "    \"sec-fetch-mode\": \"cors\","\n"
        "    \"sec-fetch-dest\": \"script\","\n"
        "    \"accept-encoding\": \"gzip, deflate, br, zstd\","\n"
        "    \"accept-language\": \"en-US,en;q=0.9\","\n"
        "    \"priority\": \"u=1\""\n"
        "  },"\n"
        "  \"proxy\": {"\n"
        "    \"scheme\": \"https\","\n"
        "    \"host\": \"127.0.0.1\","\n"
        "    \"port\": \"24801\","\n"
        "    \"authorization\": \"Basic dXNlcm5hbWU6cGFzc3dvcmQ=\""\n"
        "  },"\n"
        "  \"session\": { \"expirationInMilliseconds\": 300000 }"\n"
        "}";

    // Step 1: get result pointer and length
    int len = 0;
    char *result = handleRequest(request, &len);
    if (result != NULL && len > 0) {
        // Step 2: allocate exact buffer, copy content (frees result internally)
        char *buf = malloc(len + 1);
        getBasketContent(result, buf);
        printf("%s\n", buf);
        free(buf);
    }

    cleanupEnv();
    return 0;
}

Request JSON Format

Field Type Description
url string Target URL (required)
method string HTTP method: GET, POST
headers object Request headers
payload object Request body (for POST)
connectTimeoutInMilliseconds number TCP + TLS connect timeout
responseReadingTimeoutInMilliseconds number Response reading timeout
decompress number Decompression flags: 0 (none), 1 (gzip), 2 (deflate), 4 (br), 8 (zstd), or combinations (e.g. 15 = all)
log number Enable logging: 0 (off), 1 (on)
proxy object Proxy: { scheme, host, port, authorization? }
session object Session: { expirationInMilliseconds, clientHelloId? }

session.clientHelloId

Optional uTLS-style identifier that pins the TLS/HTTP/2 wire fingerprint to emulate. When omitted, the fingerprint follows the request's User-Agent, and an unrecognized User-Agent falls back to hellochrome_auto.

clientHelloId Emulated profile
hellochrome_auto Desktop Chrome — currently emulated version
hellochrome_150 Desktop Chrome 150 (version-pinned)
hellocrios_auto Chrome on iOS (CriOS) — currently emulated version
hellocrios_150 Chrome on iOS (CriOS) 150 (version-pinned)

_auto always tracks the latest emulated version, while _<version> pins that specific profile. Matching is case-insensitive.

Testing

# C tests (from build directory)
./bin/test_GET
./bin/test_POST

# Node.js
cd nodejs && npm install && npm test

# Python
cd python && bash build.sh

# Java
cd java && bash build.sh

Troubleshooting

Windows: No CMAKE_ASM_NASM_COMPILER could be found

BoringSSL compiles its optimized crypto routines from .asm sources, which requires the NASM assembler. CMake aborts when nasm is not on your PATH. Since Windows builds use the MSYS2 / MinGW-w64 toolchain (see Building), choose one of the following fixes:

Option A — Install NASM (recommended, keeps assembly optimizations)

  1. In the MSYS2 MinGW64 shell, install NASM:
    pacman -S --needed mingw-w64-x86_64-nasm
  2. Verify it is on your PATH:
    nasm --version
  3. Delete the CMake cache and re-configure so the compiler is re-detected:
    rm -rf build
    cmake -B build -G "Ninja"
    cmake --build build

If CMake still cannot find it, point it explicitly:

cmake -B build -G "Ninja" -DCMAKE_ASM_NASM_COMPILER="$(which nasm)"

Option B — Disable assembly optimizations (no NASM needed)

Build BoringSSL in pure-C mode. This is slightly slower but avoids the assembler dependency entirely:

cmake -B build -G "Ninja" -DOPENSSL_NO_ASM=1
cmake --build build

Tech Stack

Component Library Purpose
TLS 1.3 BoringSSL TLS handshake, session resumption, cert compression
HTTP/2 Custom implementation Frames, HPACK, stream multiplexing, flow control
JSON Jansson Request/response JSON serialization
Brotli Brotli Response decompression + TLS cert compression
Zstd Zstd Response decompression
zlib zlib gzip / deflate decompression

Using the Release Artifacts

Every GitHub Release ships prebuilt binaries so you do not need to compile the C library (BoringSSL, etc.) yourself. Each binding is published following its language's standard package-manager layout, so you install/use it the way you would any other package.

Component Release artifact(s) Standard form
C library http2client-<ver>-all.tar.gz tarball with linux/ macos/ win/ subdirs (lib + include/)
Node.js http2-client-nodejs-<ver>.tgz npm package with prebuilds/<plat>-x64/http2addon.node
Python http2_client-<ver>-<plat>.whl (×3) platform wheels, self-contained native lib inside
Java http2-client-java-<ver>.jar single cross-platform fat JAR (classes + native/linux|macos|win)

<plat> is linux / macos / win. A checksums-<ver>.sha256 file is published alongside — verify before use:

sha256sum -c "checksums-${VER}.sha256"

The C library (libhttp2client) must be loadable at runtime (see per-binding notes). The simplest approach is to keep it next to the binding, or add its directory to the loader path:

Platform Loader path env / flag
Linux LD_LIBRARY_PATH=/path/to/libdir
macOS DYLD_LIBRARY_PATH=/path/to/libdir (or install_name_tool the rpath)
Windows add the directory to PATH

The per-binding install/use instructions live in each language's own README:

License

Apache-2.0

About

An HTTP Request Client, simulate browsers' TLS fingerprints and behaviour, including HTTP/2 features, response payload decompression, proxy supported, NodeJS/Python/Java supported

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages