Skip to content

Switch benchmarks to real HTTP roundtrips against a localhost WSGI server#959

Closed
Kludex wants to merge 2 commits into
mainfrom
high-level-benchmark-suite
Closed

Switch benchmarks to real HTTP roundtrips against a localhost WSGI server#959
Kludex wants to merge 2 commits into
mainfrom
high-level-benchmark-suite

Conversation

@Kludex
Copy link
Copy Markdown
Member

@Kludex Kludex commented May 17, 2026

Summary

  • The previous suite mixed pure-Python microbenchmarks (URL parsing, header construction) with MockTransport-based "client" benchmarks that short-circuited every transport-layer code path. Optimisations to the network backends (e.g. perf: use memoryview in write() to avoid copies #954) could not be detected because MockTransport never touches httpcore2._backends.
  • This PR replaces the suite with six benchmarks that exercise the user-facing path: a tiny in-process WSGI server runs on a random localhost port, and the benchmarks drive a real httpx2.Client against it. The whole stack runs - Client, Transport, connection pool, real sockets - so any optimisation along that path now produces a measurable signal.

Benchmarks

  • test_bench_url_parse_and_join - pure CPU, URL parse + join regression canary.
  • test_bench_request_build_json - request construction + JSON encode.
  • test_bench_client_get_json - GET a moderate JSON response from the real server.
  • test_bench_client_post_large - POST 4MB through the full client (exercises socket write).
  • test_bench_client_stream_download - stream 4MB from the server (exercises socket read).
  • test_bench_client_keepalive_burst - 64 sequential GETs over a kept-alive connection.

Test plan

AI Disclaimer

This PR was developed with the assistance of either Claude or Codex. I've reviewed and verified the changes.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ae46818f2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/test_benchmark.py
Comment on lines +26 to +27
class _SilentHandler(WSGIRequestHandler):
def log_message(self, format: str, *args: Any) -> None:
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Enable HTTP/1.1 for keep-alive benchmark server

_SilentHandler inherits WSGIRequestHandler without overriding protocol_version, so the benchmark server responds as HTTP/1.0 (connection closed after each response). In this setup, test_bench_client_keepalive_burst cannot actually exercise a kept-alive connection, and the other client benchmarks also include reconnect overhead rather than stable connection reuse, which invalidates the transport/pooling signal this commit is trying to measure.

Useful? React with 👍 / 👎.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 17, 2026

Merging this PR will degrade performance by 92.57%

❌ 1 regressed benchmark
🆕 5 new benchmarks
⏩ 6 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 WallTime test_bench_request_build_json N/A 204.6 ms N/A
🆕 WallTime test_bench_url_parse_and_join N/A 165.8 ms N/A
🆕 WallTime test_bench_client_post_large N/A 149.4 ms N/A
🆕 WallTime test_bench_client_get_json N/A 344.5 ms N/A
WallTime test_bench_client_stream_download 28.2 ms 379.3 ms -92.57%
🆕 WallTime test_bench_client_keepalive_burst N/A 235.6 ms N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing high-level-benchmark-suite (a1d3bd5) with main (8d5f182)

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@Kludex Kludex closed this May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant