Cookie jar becomes performance bottleneck when sharing client across diverse requests

[konstin]

In a small scraper I run, I noticed that the network process was continuously near 100% CPU use, even though it should have been network bound. During profiling, I noticed that the bottleneck is the cookie jar, highlighted in red:

SVG: https://gist.github.com/konstin/a85b7ba02fd8db8400533a7cffc09fea

Following https://pydantic.dev/docs/httpx2/guides/async/#opening-and-closing-clients, I've been using one client for the scraper. It seems that the CookieJar in there gets copied for every request. The call graph for that:

• My tool calls AsyncClient.stream()
• which calls self.build_request()
• which calls self._merge_cookies(cookies)
• which calls Cookies(self.cookies)
• which calls Cookies.__init__()
• which creates a new jar and inserts every cookie

As the cookies accumulate, this copy becomes slow.

[Kludex]

It seems there's a performance issue in the the cookies library itself. I'll investigate more tomorrow.

I have this PR open for the time being: #1285

[konstin]

This is from codex and it only targets a part of the overhead, but it suggested this fairly small fix to skip the merge step when not setting custom cookies:

 def _merge_cookies(self, cookies=None):
+    if cookies is None:
+        # Request copies Cookies before generating the header, so no merge
+        # snapshot is needed when there are no per-request cookies.
+        return self.cookies
     if cookies or self.cookies:
         merged_cookies = Cookies(self.cookies)
         merged_cookies.update(cookies)
         return merged_cookies
     return cookies

- cookies = Cookies(self.cookies)
  return Request(
      method=method,
      url=url,
      headers=headers,
-     cookies=cookies,
+     cookies=self.cookies,
      stream=stream,
      extensions=request.extensions,
  )

Path	Parent CPU samples	Prototype
Copies in _merge_cookies and redirects	12.6%	Solved here
Copy inside Request	10.5%	Not solved here
Cookie/domain selection	17.7%	Not solved here
Expiry scanning	6.7%	Not solved here

[Kludex]

The above was already on main (some minutes after you created this discussion: fc6ff89). 😅

But we still have some performance issues there that the PR mentioned is trying to solve.

[Kludex]

I've simplified the implementation. Now it's easy to understand for humans as well. 🙏

I've filled python/cpython#160027 on the CPython's side.

---

The performance improvement is now available in httpx2 2.16.0.

@konstin Thanks for your work on uv! ❤️

[konstin]

Awesome, thank you for the quick fix and the release! I confirmed that this solves the slowdown for me.