Support HTTP/2 prior-knowledge, using `httpx.Client(http1=False, http2=True)`. #1624

bli74 · 2021-05-04T13:13:28Z

This pull request is related to httpcore pull request 269.
New parameter 'http1' is passed through to httpcore.

Both changes are used in robotframework-httpx that provides HTTP/2 support for Robotframework.

tomchristie · 2021-05-06T12:52:07Z

Okay, so design question here...

I guess we might want to be using http11=[True|False] (Indicating HTTP/1.1, and in line with the h11 package that we use under the hood.)

But I'm not completely sure if we want to prefer that or http1=[True|False].

tomchristie · 2021-05-06T16:00:00Z

Okay, so I'm okay with this. Couple of things...

I'd prefer a style where the http1 argument comes before the http2 argument.
We'll need to pin to httpcore>=0.13.3 in the setup.py.

bli74 · 2021-05-10T09:40:19Z

Okay, so design question here...

I guess we might want to be using http11=[True|False] (Indicating HTTP/1.1, and in line with the h11 package that we use under the hood.)

But I'm not completely sure if we want to prefer that or http1=[True|False].

The main reason for this patch was to enforce HTTP/2 usage. There was no need for different handliong of HTTP/1 and HTTP/1.1.
The patch uses the new parameter of httpcore and passes the value down to this library.

tomchristie · 2021-05-11T09:40:53Z

Great, thanks!

nathalie21005 · 2022-01-19T01:13:30Z

Hi @tomchristie

I am using httpx.Client(http2=True, http1=False) for one request is working, but I am not able to send multiple request for the same connection.
Also I tried httpx.AsyncClient(verify=False, http1=False,http2=True) with asyncio.run() but I am facing the same error.
ConnectionTerminated error_code:ErrorCodes.NO_ERROR

What I am sure about that the issue is related to the stream Id, but how can I initiate multiple request with different stream id in the same connection for http2?

Thank you

tomchristie · 2022-01-19T08:59:37Z

but I am not able to send multiple request for the same connection.

Can you show us what you're seeing?
What's the simplest possible way to replicate the error?
Are you accessing a public URL that you can share with us so that we can replicate the issue?

nathalie21005 · 2022-01-19T12:39:42Z

Hi @tomchristie

this is the code which is working for one http/2 request but not multiple:
import asyncio
import httpx
import time
import json

start_time = time.time()

async def Asyncmain(clients):
async with clients as client:
try:
resp = await client.request(
method="POST",
url="http://examplelink",
content=payload,
headers= header
)
res = resp.json()
finally:
await client.aclose()
print(res)

if name == 'main':
limit = httpx.Limits(max_keepalive_connections=1, max_connections=2)
for i in range(5):
client= httpx.AsyncClient(verify=False, http1=False,http2=True, limits=limit)
asyncio.run(Asyncmain(client))
print("--- %s seconds ---" % (time.time() - start_time))
print("Done")

The error I got:

Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/usr/lib/python3.8/site-packages/httpx/_transports/default.py", line 291, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 244, in handle_async_request
raise exc
File "/usr/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 228, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/lib/python3.8/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
return await self._connection.handle_async_request(request)
File "/usr/lib/python3.8/site-packages/httpcore/_async/http2.py", line 143, in handle_async_request
raise exc
File "/usr/lib/python3.8/site-packages/httpcore/_async/http2.py", line 111, in handle_async_request
status, headers = await self._receive_response(
File "/usr/lib/python3.8/site-packages/httpcore/_async/http2.py", line 228, in _receive_response
event = await self._receive_stream_event(request, stream_id)
File "/usr/lib/python3.8/site-packages/httpcore/_async/http2.py", line 259, in _receive_stream_event
await self._receive_events(request, stream_id)
File "/usr/lib/python3.8/site-packages/httpcore/_async/http2.py", line 286, in _receive_events
raise RemoteProtocolError(event)
httpcore.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:0, additional_data:None>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "test.py", line 40, in
asyncio.run(Asyncmain(client))
File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "test.py", line 23, in Asyncmain
resp = await client.request(
File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 1513, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 1600, in send
response = await self._send_handling_auth(
File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 1628, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 1665, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 1702, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/lib/python3.8/site-packages/httpx/_transports/default.py", line 291, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/lib/python3.8/contextlib.py", line 131, in exit
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:0, additional_data:None>

I tried a lot of way to initiate the AsyncClient and Client, inside the loop and outside the loop and I am still facing this issue.

Your help is much appreciated.

tomchristie · 2022-01-19T13:24:49Z

So, first up... simplify things way down, as simple as you can...

client = httpx.Client(http1=False, http2=True)

r = client.get(url)
print(r)
r = client.get(url)
print(r)

Does this still raise the exception, or not?

(Also is this a public URL?)

nathalie21005 · 2022-01-19T13:35:16Z

No, It's not a public URL.
Yeah I tried it, it work for the first one and I got the same error message. Also I need to post I don't have get method for any request.

client = httpx.Client( http2=True, http1=False)
try:
data= send_request(client)
print("result 1")
print(data)

        data= send_request(client)
        print("result 2")
        print(data)
    finally:
        client.close()

tomchristie · 2022-01-19T13:38:57Z

What server are you working against?
Are you able to use HTTP/1.1?

nathalie21005 · 2022-01-19T13:48:19Z

The server is Apache | Ubuntu. Yeah, for http1.1 is working but Http/2 should have different stream Id. so when I am trying to send multiple request still facing this issue.

tomchristie · 2022-01-19T14:11:19Z

Well, first pass answer is - don't use HTTP/2. (It's not gaining you anything)

(Unless you've got a good reason to do so, probably also don't use async, but that's a different story)

Why is the server disconnecting after the first request... don't know - really very hard for me to be able to dig into if it's something I'm not able to replicate. It'd be helpful for us at some point to add enough logging that you're able to fully inspect all the frames in both directions, but we don't have that at the moment.

nathalie21005 · 2022-01-19T14:15:47Z

Http1.1 does not give me the ability to send multiple request in parallel or open multiple TCP connection, that's why I was trying to send this request.
Did you get the chance to do multiple requests using http/2 and It worked?

Thank you so much for your help

tomchristie · 2022-01-19T14:25:13Z

Sure, this sort of thing works fine...

import httpx

client = httpx.Client(http2=True)

r = client.get("https://www.example.com")
print(r, r.http_version)
r = client.get("https://www.example.com")
print(r, r.http_version)

A sensible approach would be to really really narrow down your example until you've got something absolutely trivial that you can share as the simplest possible case that reproduces the error. Ideally you'd want to try the code against some other HTTP/2 server so you can try to narrow down if you're seeing a client issue or a server issue.

(But again, if you want an easy life here - just use HTTP/1.1. Really.)

Pass flag http1 to httpcore.

cdfeeb1

tomchristie changed the title ~~Pass flag http1 to httpcore.~~ Support HTTP/2 prior-knowledge, using httpx.Client(http1=False). May 6, 2021

tomchristie changed the title ~~Support HTTP/2 prior-knowledge, using httpx.Client(http1=False).~~ Support HTTP/2 prior-knowledge, using httpx.Client(http1=False, http2=True). May 6, 2021

bli74 and others added 2 commits May 7, 2021 06:16

Update httpcore version, reorder parameter list

cbb7f65

Merge branch 'master' into feature/http2_only

6b67e23

Merge branch 'master' into feature/http2_only

f57d02a

tomchristie merged commit 9b17671 into encode:master May 11, 2021

p1-ra mentioned this pull request May 21, 2021

Fix: proxy http2 prior to knowledge support #1647

Closed

1 task

tomchristie mentioned this pull request Jun 17, 2021

Version 0.18.2 #1690

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support HTTP/2 prior-knowledge, using `httpx.Client(http1=False, http2=True)`. #1624

Support HTTP/2 prior-knowledge, using `httpx.Client(http1=False, http2=True)`. #1624

bli74 commented May 4, 2021 •

edited by tomchristie

Loading

tomchristie commented May 6, 2021

tomchristie commented May 6, 2021

bli74 commented May 10, 2021

tomchristie commented May 11, 2021

nathalie21005 commented Jan 19, 2022

tomchristie commented Jan 19, 2022

nathalie21005 commented Jan 19, 2022 •

edited

Loading

tomchristie commented Jan 19, 2022

nathalie21005 commented Jan 19, 2022 •

edited

Loading

tomchristie commented Jan 19, 2022

nathalie21005 commented Jan 19, 2022

tomchristie commented Jan 19, 2022 •

edited

Loading

nathalie21005 commented Jan 19, 2022

tomchristie commented Jan 19, 2022

Support HTTP/2 prior-knowledge, using httpx.Client(http1=False, http2=True). #1624

Support HTTP/2 prior-knowledge, using httpx.Client(http1=False, http2=True). #1624

Conversation

bli74 commented May 4, 2021 • edited by tomchristie Loading

tomchristie commented May 6, 2021

tomchristie commented May 6, 2021

bli74 commented May 10, 2021

tomchristie commented May 11, 2021

nathalie21005 commented Jan 19, 2022

tomchristie commented Jan 19, 2022

nathalie21005 commented Jan 19, 2022 • edited Loading

tomchristie commented Jan 19, 2022

nathalie21005 commented Jan 19, 2022 • edited Loading

tomchristie commented Jan 19, 2022

nathalie21005 commented Jan 19, 2022

tomchristie commented Jan 19, 2022 • edited Loading

nathalie21005 commented Jan 19, 2022

tomchristie commented Jan 19, 2022

Support HTTP/2 prior-knowledge, using `httpx.Client(http1=False, http2=True)`. #1624

Support HTTP/2 prior-knowledge, using `httpx.Client(http1=False, http2=True)`. #1624

bli74 commented May 4, 2021 •

edited by tomchristie

Loading

nathalie21005 commented Jan 19, 2022 •

edited

Loading

nathalie21005 commented Jan 19, 2022 •

edited

Loading

tomchristie commented Jan 19, 2022 •

edited

Loading