Streaming doesn't seem to be working

[sharat87]

Checklist

• I've searched for similar issues.
• I'm using the the latest version of HTTPie.

Version 2.4.0

---

What are the steps to reproduce the problem?

Hit any streaming API endpoint, even the one in the documentation. Examples:

1. http -S pie.dev/stream/3
2. http -S pie.dev/drip

What is the expected result?
That the response data shows in the output immediately when it was received. Data should stream.

What happens instead?
The full response data is buffered and is printed at the end of the response. Data doesn't stream.

Debug output

Please re-run the command with --debug, then copy the entire command & output and paste both below:

~> http --debug -S pie.dev/stream/3          
HTTPie 2.4.0
Requests 2.25.1
Pygments 2.7.4
Python 3.9.4 (default, Apr  5 2021, 01:49:30) 
[Clang 12.0.0 (clang-1200.0.32.29)]
/usr/local/Cellar/httpie/2.4.0/libexec/bin/python3.9
Darwin 19.6.0

<Environment {'colors': 256,
 'config': {'default_options': []},
 'config_dir': PosixPath('/Users/shri/.config/httpie'),
 'devnull': <property object at 0x10f8e4db0>,
 'is_windows': False,
 'log_error': <function Environment.log_error at 0x10f8e6ee0>,
 'program_name': 'http',
 'stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,
 'stderr_isatty': True,
 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
 'stdin_encoding': 'utf-8',
 'stdin_isatty': True,
 'stdout': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>,
 'stdout_encoding': 'utf-8',
 'stdout_isatty': True}>

>>> requests.request(**{'auth': None,
 'data': RequestJSONDataDict(),
 'headers': {'User-Agent': b'HTTPie/2.4.0'},
 'method': 'get',
 'params': <generator object MultiValueOrderedDict.items at 0x10f9a1b30>,
 'url': 'http://pie.dev/stream/3'})

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
CF-Cache-Status: DYNAMIC
CF-RAY: 64a69c9f1e561944-SIN
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Wed, 05 May 2021 02:41:47 GMT
NEL: {"max_age":604800,"report_to":"cf-nel"}
Report-To: {"max_age":604800,"group":"cf-nel","endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=BJF%2BeIdKlobpl92YeRYjpUHuxKkt4wN5vChg5EqcVZylH74PhJP8H%2BIJcCQr0i2CfaAczTaDltcM7sP1jUUQnoaKRTpM5Us7"}]}
Server: cloudflare
Set-Cookie: __cfduid=d7bbf175a872da20d622e08c840b1864f1620182507; expires=Fri, 04-Jun-21 02:41:47 GMT; path=/; domain=.pie.dev; HttpOnly; SameSite=Lax
Transfer-Encoding: chunked
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400
cf-request-id: 09dc00376d00001944e711d000000001

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.204.224.183",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "64a69c9f1e561944-FRA",
        "Cf-Request-Id": "09dc00376d00001944e711d000000001",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.4.0"
    },
    "id": 0,
    "origin": "49.204.224.183",
    "url": "http://pie.dev/stream/3"
}

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.204.224.183",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "64a69c9f1e561944-FRA",
        "Cf-Request-Id": "09dc00376d00001944e711d000000001",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.4.0"
    },
    "id": 1,
    "origin": "49.204.224.183",
    "url": "http://pie.dev/stream/3"
}

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.204.224.183",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "64a69c9f1e561944-FRA",
        "Cf-Request-Id": "09dc00376d00001944e711d000000001",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.4.0"
    },
    "id": 2,
    "origin": "49.204.224.183",
    "url": "http://pie.dev/stream/3"
}

Provide any additional information, screenshots, or code examples below:

Screen.Recording.2021-05-05.at.08.07.26.mov

Screen.Recording.2021-05-05.at.08.08.50.mov

[sharat87]

Example taken from docs here: https://httpie.io/docs/2.4.0#examples-use-cases.

Thank you for your work.

[elenalape]

I think there is an issue with httpbin.org/stream/:n (which is what pie.dev uses), not HTTPie, which is why it doesn't produce expected behaviour. Have you tested with other endpoints than the ones in the docs?

postmanlabs/httpbin#600

Actually, nevermind! I think this is expected behaviour. Streaming entails that all the response data is sent over a single HTTP connection, and that is what you see in your screenshots. See the comment below why they appear instantly.

[jkbrzt]

@sharat87 Thanks for the report! The --stream flag should and does seem to work with /stream/3.

As you can verify in the terminal output, those three JSON chunks get formatted with --stream (unlike without the flag where the whole response is considered but doesn’t constitute a valid JSON document). The response is very small so the chunks appear pretty much at the same time. It would be easier to verify the behavior with an endpoint that includes a delay after each line.

The /drip endpoint streams just characters without \n. There --stream won’t have any effect.

[sharat87]

Thank you for the replies. I believe the httpbin endpoint is indeed responding correctly. I've tested the /drip endpoint with curl and the output shows up in a streaming fashion. Please view the below video. The * characters appear one by one, with one second delay when using curl. But with httpie, they all appear at once after a 6 second delay:

Screen.Recording.2021-05-05.at.18.47.26.mov

[jkbrzt]

@sharat87 in HTTPie, --stream is for line-by-line processing. The /drip endpoint returns only one line of text. Therefore, --stream has no effect there. HTTPie currently doesn‘t have an equivalent to curl’s -N, --no-buffer.

[sharat87]

I see. Thank you very much for the response. I didn't realize --stream only works on line streaming. Thank you for the detail. Please consider adding this note to the documentation.

Closing.