Skip to content

httpclient: duplicate Host header and case-insensitive header handling #116

Description

@Oaklight

Problem

When users pass pre-constructed headers (e.g. for AWS SigV4 signing), the httpclient module generates duplicate or conflicting headers:

  1. _build_raw_http_request always inserts Host: on top of whatever is in the headers dict. If the user already provides host or Host, the request ends up with two Host headers → MinIO/S3 returns 400 Bad Request.

  2. No case-insensitive header merging. _prepare_request sets User-Agent, Accept-Encoding as defaults, then does req_headers.update(headers or {}). If the user passes user-agent (lowercase), both User-Agent and user-agent end up in the dict since Python dicts are case-sensitive.

Expected behavior (aligned with httpx/requests)

Both httpx and requests handle this correctly:

  • httpx: Uses case-insensitive header storage internally. Host is auto-generated from URL but user-provided Host/host overrides it. Default headers (user-agent, accept-encoding, etc.) can be overridden by user headers. No duplicates.
  • requests: Session defaults are merged with user headers; user wins. urllib3 handles Host.

Suggested fix

  1. _build_raw_http_request: Check if req_headers already contains Host (case-insensitive) before adding the auto-generated one. If the user provided it, use theirs.

  2. _prepare_request: When merging user headers with defaults, use case-insensitive matching so user-agent overrides User-Agent, host overrides Host, etc. Could normalize all keys to lowercase internally (httpx approach), or do a case-insensitive key check before inserting defaults.

Context

Discovered while building the s3 module's async client (AsyncS3Client). SigV4 signing requires precise control over all HTTP headers — the signature is computed over canonical headers (lowercased), and any extra or duplicate headers cause signature verification to fail on the server side.

The sync path using http.client.HTTPConnection.request() works because http.client does case-insensitive Host deduplication internally. The async path using _build_raw_http_request does not.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Priority 2 - Medium-High

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions