Why does stream not return a file-like object (this breaks compatibility) #2296

jacksbox · 2022-07-04T11:52:13Z

jacksbox
Jul 4, 2022

In my company we use mostly httpx in our projects to do sync and async calls to our and other apis, but recently we ran into an issue, which (imo) could (and maybe should?) be solved:

httpx.stream (in contrast to e.g. urlib and therefor requests etc. as well) does not return a file-like object - and that in turn breaks other packages which expect a file-like object, e.g. s3 stream upload via boto3 expects a file-like object, with the normal read(amount) method.

Sure, I could write a proxy object to do this, but as this is quit a low level functionality it feels like that it's something this package should directly provide.

Answered by My-Responsitories

Aug 26, 2023

Is the following code what you want

with httpx.stream("GET", "https://example.com/") as response:
    raw_stream = response.extensions["network_stream"]
    print(raw_stream.read(2))
# b'\x1f\x8b'

print(requests.get("https://example.com/", stream=True).raw.read(2))
# b'\x1f\x8b'

View full answer

tomchristie · 2022-07-04T12:25:02Z

tomchristie
Jul 4, 2022
Maintainer

I don't quite understand the question. Perhaps providing an example of what you mean would help work through this?

The stream method requires a context manager and returns a response within that context-managed block...

with client.stream("GET", "https://www.example.com") as response:
    ...

You can use response.read(amount) or any of the other regular methods on the response within that block.

10 replies

tomchristie Aug 21, 2023
Maintainer

I'll belatedly follow up on this.

The .raw property on a requests.Response is almost never what you actually want, as it returns the raw bytewise content of the response which will often be compressed. Do we have an equivelent to that in httpx, yes. The .stream property is equivalent. Does it have a file-like interface? No, it has a stream-like interface.

jacksbox Aug 21, 2023
Author

@tomchristie I opened this issue as I was working on part of a service which copied a remote file to a s3 bucket. The reason for this issue was I did not want to save the file first on the server and then upload it to s3 but instead stream it directly to there.

boto3 supports this via the following method https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/upload_fileobj.html
It requires a file like object, with the read(size) method available.
requests provides this whereas httpx.stream does/did not provide this (at the time of opening this issue)

tomchristie Aug 21, 2023
Maintainer

requests provides this

So, requests doesn't have a file-like interface on the response.

It does have a file-like interface on the .raw property but that probably wouldn't be what you'd want for interfacing with that upload method, since it is an interface onto the possibly encoded (gzip or whatever) content of the response.

>>> import requests
>>> r = requests.get("https://www.example.com")
>>> r.content
b'<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset="utf-8" />\n    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />\n    <meta name="viewport" content="width=device-width, initial-scale=1" />\n    <style type="text/css">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href="https://www.iana.org/domains/example">More information...</a></p>\n</div>\n</body>\n</html>\n'

>>> r = requests.get("https://www.example.com", stream=True)
>>> r.raw.read(4096)
b'\x1f\x8b\x08\x00\xc2\x15\xa8]\x00\x03}TMs\xdb \x10\xbd\xfbWl\xd5K2#$\'i\x1a\x8f-i\xfa\x99i\x0fi\x0fi\x0f=\x12\xb1\xb2\x98\x08P\x01\xc9\xf6t\xf2\xdf\xbbB\x8e#7\x99\x9a\x91\x81]x\xbb\xef\xb1\x90\xbd\x12\xa6\xf4\xbb\x16\xa1\xf6\xaa)f\xd9c\x87\\\x143\xa0_\xe6\xa5o\xb0\xf8\xbc\xe5\xaam\x10>\x19\xc5\xa5\xce\xd2\xd1:\x1b\x97(\xf4\x1c\xca\x9a[\x87>\x8f:_\xb1E\x04i1q\xd6\xde\xb7\x0c\x7fw\xb2\xcf\xa3\x8fF{\xd4\x9e\ra#(\xc7Y\x1ey\xdc\xfat\x08\xbf:@\xbd\x84\xa4\xb9\xc2<\xea%nZc\xfdd\xffF\n_\xe7\x02{Y"\x0b\x93\x18\xa4\x96^\xf2\x86\xb9\x927\x98\x9f=A9\xbf#2C\x06\xfb\xc0\xa5s\xd1\xe8\xbb3b\x07\x7f\xc20Lyy\xbf\xb6\xa6\xd3\x82\x95\xa61v\t\xaf\xab9\xb5\xf3\xd5a\x89\xe2v-\xf5\x12\xe6O\xa6\x96\x0b!\xf5\xfa\xc8VQ\xa6\xac\xe2J6\xbb%0\xde\x92\x9c\xcc\xed\x9cG\x15\xc3\xd8\xb3N\xc6\xf0\xa1\x91\xfa\xfe\x86\x97\xb7\xc1tM\x9bb\x88nqm\x10~~\x8dh\xfc\xbdE\r\xb7\\\xbba\xf2\x05\x9b\x1e\xbd,9|\xc3\x0e\xc9r0\xc4\xf0\xde\x12w\xc2\xa6\xa5\xcc\xa1\x95\xd5S.a\xf0\x10\xfe\x85\xec\'t\x83pKx;\x9f\xb7\xdb\xe7\x0c/Q\x01\xef\xbcy\x81\xe89\xaa\xd5\x7fE\x13\xd4&\x19\xdc\x19+\xd02\xcb\x85\xec\x1c\xe9\x94\\\x1e\x01\x98-s5\x17fC\xc8\xed\x16.\xe8\xbb\xa2o\x18\xdb\xf5\x1d?\x99\xc7\xa1%\xf3\xf3\xd3\xd5\x84\x0c_\x0e\xea\xc5\xd4\xf7\xd2I\x8fbB\xed1\x93\x8b\xc5\x9b\xc5b\x92\xc9p\xfeL`i,\xf7\xd2\x10Km4NA\xdf)\x14\x92\xc3\x89\xe2[\xb6\xd7\xe7j\xd0\xe7t\x02~\xac\xe2QU\xfc\xa3\xd8D\xe5c\xc7\xc3$d\x96\x86\n-\xc2Ye\xe9x\x1dg\xd9P\x9bt;)\xd8\xbe\x8e\xeb\xb3g7\x93L\xa3\xaf-~\xd4\xd2\x81\x08v\xa0Qe,t\x0ea\x985M\xe7\xfc@\xb8G\xc0\x11\xc1\r\x0ez\x0e:E\xf7\xc9%\xf0\xcbtDb\x17\xb6xB\x1a\xabe\x8f\xa6\xa1!y\t\xa0\xb3Ht|m:\x0f\xad\x95\x14\xa24t\xb4R\x071\x81\xe6\xdc\xddS\x85\x84\xe8-Z%\x9d#G\x92\xa5\xed!\xcf\x8c\x1e\x08\x8bU\x1e\r\xcf\x84[\xa6\xe9f\xb3I$\xd7<1v\x9d\x8e!]\xbaO3*n\x8c\x1dH\x10\xa0\nA\x92\x84\xd0x\x11\x10\xb34\x88\x93\xa5{\xa9\xd2\xf1A\xfb\x0b(\xeb|o\xe8\x04\x00\x00'

I can see what you're asking for here, although it's not something that requests provides.

The closest you'd be able to do as things currently stand would be to write a lightweight adapter class which provides a .read(n) method which calls into the response .iter_bytes(n).

Maybe something along these lines...

class StreamingToFileSyncAdapter:
    def __init__(self, response):
        self.streaming_source = response.iter_bytes()
        self.buffer = b""
        self.buffer_offset = 0

    def read(self, num_bytes):
        while len(self.buffer) - self.buffer_offset < num_bytes:
            try:
                chunk = next(self.streaming_source)
                self.buffer += chunk
            except StopIteration:
                break

        if len(self.buffer) - self.buffer_offset >= num_bytes:
            data = self.buffer[self.buffer_offset:self.buffer_offset + num_bytes]
            self.buffer_offset += num_bytes
            return data
        else:
            data = self.buffer[self.buffer_offset:]
            self.buffer = b""
            self.buffer_offset = 0
            return data

jacksbox Aug 22, 2023
Author

It does have a file-like interface on the .raw property but that probably wouldn't be what you'd want for interfacing with that upload method, since it is an interface onto the possibly encoded (gzip or whatever) content of the response.

In my case it was exactly what I wanted (I knew what I would get from the remote source) :) In the end I solved it quite similar to your code example.

But as requests provides this functionality out of the box (ok, on the .raw prop of the response) and httpx aims to be broadly compatible with the requests I thought it would be a nice addition. (even as you note this difference on the docs)

tomchristie Aug 23, 2023
Maintainer

So, the crux of this is that httpx and httpcore use a byte-iterator API as the primitive for the request/response body, rather than using a file API as the primitive for the request/response body. Both are valid API choices.

Questions for the class: What implications does this design choice have as a result? If you wanted to change this aspect of the API, how would you go about implementing that change?

mscanlon-exos · 2023-08-17T08:55:25Z

mscanlon-exos
Aug 17, 2023

@tomchristie Was this feature request/ question dropped? Seems to me there is still no work around for this

2 replies

zanieb Aug 20, 2023
Collaborator

I think the next step here is for someone to provide a simple code example that demonstrates how they are using this feature with requests so we can understand the desired API.

tomchristie Aug 21, 2023
Maintainer

Seems to me there is still no work around for this

Because it's not clear what the question is. No work around for what exactly?

My-Responsitories · 2023-08-26T01:33:22Z

My-Responsitories
Aug 26, 2023

Is the following code what you want

with httpx.stream("GET", "https://example.com/") as response:
    raw_stream = response.extensions["network_stream"]
    print(raw_stream.read(2))
# b'\x1f\x8b'

print(requests.get("https://example.com/", stream=True).raw.read(2))
# b'\x1f\x8b'

0 replies

luked99 · 2024-06-09T19:12:15Z

luked99
Jun 9, 2024

I think the next step here is for someone to provide a simple code example that demonstrates how they are using this feature with requests so we can understand the desired API.

I hit a similar problem but with tarfile while trying to move some code from requests to httpx. This is what my requests code looks like:

import tarfile
import requests

url = "http://localhost:9999/test.tar.gz"

with requests.get(url, stream=True) as r:
    r.raise_for_status()
    with tarfile.open(fileobj=r.raw, mode="r|*") as tf:
        tf.extractall(path="out")

I did try the "network_stream" approach but I couldn't get it to work reliably. Possibly I was doing it wrong:

import tarfile
import httpx

url = "http://localhost:9999/test.tar.gz"

with httpx.stream("GET", url) as r:
    raw_stream = r.extensions["network_stream"]
    r.raise_for_status()
    with tarfile.open(fileobj=raw_stream, mode="r|*") as tf:
        tf.extractall(path="httpx-out")

I want to avoid writing the tarfiles to a temporary file as they are a few GiB in size.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Why does stream not return a file-like object (this breaks compatibility) #2296

{{title}}

Replies: 4 comments 12 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Why does stream not return a file-like object (this breaks compatibility) #2296

jacksbox Jul 4, 2022

Replies: 4 comments · 12 replies

tomchristie Jul 4, 2022 Maintainer

tomchristie Aug 21, 2023 Maintainer

jacksbox Aug 21, 2023 Author

tomchristie Aug 21, 2023 Maintainer

jacksbox Aug 22, 2023 Author

tomchristie Aug 23, 2023 Maintainer

mscanlon-exos Aug 17, 2023

zanieb Aug 20, 2023 Collaborator

tomchristie Aug 21, 2023 Maintainer

My-Responsitories Aug 26, 2023

luked99 Jun 9, 2024

jacksbox
Jul 4, 2022

Replies: 4 comments 12 replies

tomchristie
Jul 4, 2022
Maintainer

tomchristie Aug 21, 2023
Maintainer

jacksbox Aug 21, 2023
Author

tomchristie Aug 21, 2023
Maintainer

jacksbox Aug 22, 2023
Author

tomchristie Aug 23, 2023
Maintainer

mscanlon-exos
Aug 17, 2023

zanieb Aug 20, 2023
Collaborator

tomchristie Aug 21, 2023
Maintainer

My-Responsitories
Aug 26, 2023

luked99
Jun 9, 2024