Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement experimental HTTP/2 support #4769

merged 101 commits into from Mar 18, 2021

Implement experimental HTTP/2 support #4769

merged 101 commits into from Mar 18, 2021


Copy link

@Gallaecio Gallaecio commented Aug 31, 2020

Resolves #1854

This is most of the GSoC 2020 work by @adityaa30. There is additional work which will be handled in separate pull requests.

To do:

After merging:

adityaa30 and others added 30 commits May 31, 2020 18:08
- Request is sent successfully with its Response received as well.
However, the StreamEnded event is not received which do not fires the
response deferred
- Remove all wrapper funtions made such that stream can send header/data
to H2Connection as they were not necessary

Looks like, for small set of response data the StreamEnded event is
emitted and everything works well -- tested for both GET & POST request.
Maybe some issue with window size and/or flow control as when the
response data needs to be broken into separate chunks -- not all chunks
are received everytime which leads to indefinite waiting for next data
chunk and the connection is lost due to timeout. 😥

Working on setting up testing environment now. After testing is setup
I'll debug the above bug furthur.
Every data chunk received needs to be acknowledged to
- update the flow control window size
- get furthur data chunks from the server
- make separate function to parse http headers from Request instance
- Use itertools.count to generate next stream_id

When sending data/body more than the local flow control window -- no
window update occurs to send the remaining data frames. Hence, the
complete body is not send resulting in no response received.
- add Twisted[http2] in requirements
- add to test the current implementation

test_download times out because of no protocol negotiated between
Mockserver and HTTP/2 client
- Add StreamCloseReason enum
- Send response for different cases considering download_warnsize,
download_maxsize, fail_on_data_loss, connection lost, etc.
- remove as working testing environment is setup 🙂🙃
- Add typing_extensions as dependency to support TypedDict for
- Remove repeated dependency Twisted from
- Test for both GET & POST when
  - Only 1 request
  - Large number (=20) of requests
  - Small Data (10 KB) per request
  - Large Data (10 MB) per request
- Test when request is cancelled by the client'

Tests raises OpenSSL.SSL.Error when run using tox. However, all tests
passes when ran using `python -m unittest`.
- rename LOGGER -> logger
- remove self._write_to_transport from Stream class and handle all
transport related activities inside HTTP2ClientProtocol class
- add required test -- test by sending 1000 requests
- increase test timeout to 180 seconds to account for tests taking long
- while testing the job exceeded the maximum log length
and was terminated
- reduce the number of requests from 20 to 10
- refactor from str.format() to f-strings
- Initiating requests having hostname or (ip_address, port) different
from the peer to which HTTP/2 connection is made can lead to closing the
whole connection and close out all the pending streams.
- This change aims to fix that problem
- Add required tests
- Save hostname & port in H2ConnectionMetadataDict
- Change log level for hpack to ERROR
- H2ClientProtocol.close_stream
- Fix and add missing type hints
- More adjustments
- Rename stream id generator
- Simplify decrement
Copy link

@Gallaecio Regarding the memory issues, I'd like to point out a similarity with the DirtyReactorAggregateError exception I was getting in #4897 (solved by calling loseConnection on the associated producer): It's a long shot but it might be related.

Copy link
Member Author

Gallaecio commented Feb 23, 2021

@elacuesta I ended up just re-running the tests. I know it’s not a fix, but since the issue is not specific to any change here (it’s most likely reproducible in master, given it affects a documentation pull request like #4974), this is probably not the right pull request to fix it.

Copy link

elacuesta commented Feb 23, 2021

Oh I didn't realize other PRs were affected as well, much less one about docs. In that case, it's no reason to hold this one from merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

Successfully merging this pull request may close these issues.

HTTP 2 support
6 participants