Problem
When users pass pre-constructed headers (e.g. for AWS SigV4 signing), the httpclient module generates duplicate or conflicting headers:
-
_build_raw_http_request always inserts Host: on top of whatever is in the headers dict. If the user already provides host or Host, the request ends up with two Host headers → MinIO/S3 returns 400 Bad Request.
-
No case-insensitive header merging. _prepare_request sets User-Agent, Accept-Encoding as defaults, then does req_headers.update(headers or {}). If the user passes user-agent (lowercase), both User-Agent and user-agent end up in the dict since Python dicts are case-sensitive.
Expected behavior (aligned with httpx/requests)
Both httpx and requests handle this correctly:
- httpx: Uses case-insensitive header storage internally.
Host is auto-generated from URL but user-provided Host/host overrides it. Default headers (user-agent, accept-encoding, etc.) can be overridden by user headers. No duplicates.
- requests: Session defaults are merged with user headers; user wins. urllib3 handles Host.
Suggested fix
-
_build_raw_http_request: Check if req_headers already contains Host (case-insensitive) before adding the auto-generated one. If the user provided it, use theirs.
-
_prepare_request: When merging user headers with defaults, use case-insensitive matching so user-agent overrides User-Agent, host overrides Host, etc. Could normalize all keys to lowercase internally (httpx approach), or do a case-insensitive key check before inserting defaults.
Context
Discovered while building the s3 module's async client (AsyncS3Client). SigV4 signing requires precise control over all HTTP headers — the signature is computed over canonical headers (lowercased), and any extra or duplicate headers cause signature verification to fail on the server side.
The sync path using http.client.HTTPConnection.request() works because http.client does case-insensitive Host deduplication internally. The async path using _build_raw_http_request does not.
Problem
When users pass pre-constructed headers (e.g. for AWS SigV4 signing), the httpclient module generates duplicate or conflicting headers:
_build_raw_http_requestalways insertsHost:on top of whatever is in the headers dict. If the user already provideshostorHost, the request ends up with two Host headers → MinIO/S3 returns 400 Bad Request.No case-insensitive header merging.
_prepare_requestsetsUser-Agent,Accept-Encodingas defaults, then doesreq_headers.update(headers or {}). If the user passesuser-agent(lowercase), bothUser-Agentanduser-agentend up in the dict since Python dicts are case-sensitive.Expected behavior (aligned with httpx/requests)
Both httpx and requests handle this correctly:
Hostis auto-generated from URL but user-providedHost/hostoverrides it. Default headers (user-agent,accept-encoding, etc.) can be overridden by user headers. No duplicates.Suggested fix
_build_raw_http_request: Check ifreq_headersalready containsHost(case-insensitive) before adding the auto-generated one. If the user provided it, use theirs._prepare_request: When merging user headers with defaults, use case-insensitive matching souser-agentoverridesUser-Agent,hostoverridesHost, etc. Could normalize all keys to lowercase internally (httpx approach), or do a case-insensitive key check before inserting defaults.Context
Discovered while building the
s3module's async client (AsyncS3Client). SigV4 signing requires precise control over all HTTP headers — the signature is computed over canonical headers (lowercased), and any extra or duplicate headers cause signature verification to fail on the server side.The sync path using
http.client.HTTPConnection.request()works becausehttp.clientdoes case-insensitive Host deduplication internally. The async path using_build_raw_http_requestdoes not.