Skip to content

Conversation

@kylebarron
Copy link
Member

@kylebarron kylebarron commented Dec 1, 2025

The HTTPConnector trait in object_store allows for users to define arbitrary HTTP client providers. This allows us to expose a Python API that users can implement to override how HTTP requests are handled.

This gives total visibility into the HTTP requests that obstore is making.

This PR also defines an HTTP client backed by aiohttp.

Todo

  • Update the type hinting of HTTP headers, because there could be multiple values per http header. Note that perhaps this will cause problems with reconstructing pyo3-object_stores across libs because it’ll now be returning an iterator of tuples, not a dict.
  • Handle consuming iterator of tuples as well in impl<'py> FromPyObject<'_, 'py> for PyHeaderMap {
  • Consistent casing of HTTP vs Http
  • Add client_factory parameters for gcs and azure
  • method to close underlying aiohttp client session. Perhaps we should add weakrefs from the client factory onto each created client, so that we can call close_all() from the connector?

Is it possible to use yield, to allow Python to manage when to close the connection?

        async with self.session.request(method, url, headers=headers) as resp:
            version = _get_http_version_from_response(resp)
            yield _AiohttpResponse(
                status=resp.status,
                version=version,
                headers=resp.headers.items(),
                body=resp.content,
            )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants