Missing response content if header "Upgrade: h2,h2c" and "Connection: Upgrade" are available

[segal90]

Long story short

Content of a POST response is None or b'' (based on how do you want access the response content) if "Upgrade: h2,h2c" and "Connection: Upgrade" headers are in the HTTP response.
After disable Apache's mod_http2 everything works like a charm.

Expected behaviour

Ignore these headers or handle correctly the HTTP2 upgrade request but not ignore the response body.

Actual behaviour

Response body is ignored

Steps to reproduce

1. configure an Apache with mod_http2
2. run this code snippet:

auth = aiohttp.BasicAuth('user1', 'VerySecretPassword')
data = {'just': 'some', 'dummy': 'data'}
async with aiohttp.ClientSession(auth=auth, conn_timeout=10) as session:
    async with session.post(<SERVER_ADDRESS>, data=data) as response:
        print(await response.json()) 

Your environment

OS: Linux (Fedora)
aiohttp version: 2.2.5
uvloop is set as asyncio event loop

[asvetlov]

aiohttp doesn't support HTTP/2 yet.

Please describe what response do you request for the case.

[segal90]

Lack of HTTP/2 is not the problem. Problem is that the HTTP client is not ignore the request for HTTP2 from the webserver.
I just want access the HTTP response body which is now missing.

[Findus23]

Hi,

I am having the same error with https://www.soeren-hentzschel.at/feed/.

Testcase:
https://gist.github.com/Findus23/3aaedbac0dd2520a9b65d0c9a6431b22

[asvetlov]

Would you provide a Pull Request with fix?

[Findus23]

Unfortunately I am really a beginner in asynchronous python and just by change came across this error.

[asvetlov]

Cannot reproduce on aiohttp master, both Cython and Pure Python parsers.

I've tried a code from https://gist.github.com/Findus23/3aaedbac0dd2520a9b65d0c9a6431b22

[Findus23]

That's really odd.
I tried it again with a clean venv and am I using Python 3.6.4 on Debian Testing. The code is the same as before.
With the master I get aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

Exact steps to reproduce

➜  ~ mktmpenv --python=python3 
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/python3
Also creating executable in /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/preactivate
virtualenvwrapper.user_scripts creating /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/postactivate
virtualenvwrapper.user_scripts creating /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/get_env_details
This is a temporary environment. It will be deleted when you run 'deactivate'.
➜  ~ python --version
Python 3.6.4
➜  ~ nano test.py
➜  ~ pip install aiohttp
Collecting aiohttp
  Using cached aiohttp-2.3.9-cp36-cp36m-manylinux1_x86_64.whl
Collecting async-timeout>=1.2.0 (from aiohttp)
  Using cached async_timeout-2.0.0-py3-none-any.whl
Collecting yarl>=1.0.0 (from aiohttp)
  Using cached yarl-1.1.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting multidict>=4.0.0 (from aiohttp)
  Downloading multidict-4.1.0-cp36-cp36m-manylinux1_x86_64.whl (482kB)
    100% |████████████████████████████████| 491kB 1.1MB/s 
Collecting chardet (from aiohttp)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting idna>=2.0 (from yarl>=1.0.0->aiohttp)
  Using cached idna-2.6-py2.py3-none-any.whl
Installing collected packages: async-timeout, multidict, idna, yarl, chardet, aiohttp
Successfully installed aiohttp-2.3.9 async-timeout-2.0.0 chardet-3.0.4 idna-2.6 multidict-4.1.0 yarl-1.1.0
➜  ~ pip freeze
aiohttp==2.3.9
async-timeout==2.0.0
chardet==3.0.4
idna==2.6
multidict==4.1.0
yarl==1.1.0
➜  ~ python test.py 
<ClientResponse(https://www.soeren-hentzschel.at/feed/) [200 OK]>
<CIMultiDictProxy('Date': 'Tue, 30 Jan 2018 11:50:58 GMT', 'Server': 'Apache', 'Last-Modified': 'Mon, 29 Jan 2018 22:41:58 GMT', 'ETag': '"95b04997e00647ea795eb5503aad9780"', 'X-Robots-Tag': 'noindex, follow', 'Link': '<https://www.soeren-hentzschel.at/wp-json/>; rel="https://api.w.org/"', 'Upgrade': 'h2,h2c', 'Connection': 'Upgrade', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/rss+xml; charset=UTF-8')>


True
➜  ~ pip install git+https://github.com/aio-libs/aiohttp.git
Collecting git+https://github.com/aio-libs/aiohttp.git
  Cloning https://github.com/aio-libs/aiohttp.git to /tmp/pip-nefx0_40-build
Collecting attrs>=17.4.0 (from aiohttp==3.0.0a0)
  Using cached attrs-17.4.0-py2.py3-none-any.whl
Requirement already satisfied: chardet in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already satisfied: multidict>=4.0.0 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already satisfied: async_timeout>=1.2.0 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already satisfied: yarl>=1.0.0 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already satisfied: idna>=2.0 in ./lib/python3.6/site-packages (from yarl>=1.0.0->aiohttp==3.0.0a0)
Installing collected packages: attrs, aiohttp
  Found existing installation: aiohttp 2.3.9
    Uninstalling aiohttp-2.3.9:
      Successfully uninstalled aiohttp-2.3.9
  Running setup.py install for aiohttp ... done
Successfully installed aiohttp-3.0.0a0 attrs-17.4.0
➜  ~ python test.py 
<ClientResponse(https://www.soeren-hentzschel.at/feed/) [200 OK]>
<CIMultiDictProxy('Date': 'Tue, 30 Jan 2018 11:52:45 GMT', 'Server': 'Apache', 'Last-Modified': 'Mon, 29 Jan 2018 22:41:58 GMT', 'ETag': '"95b04997e00647ea795eb5503aad9780"', 'X-Robots-Tag': 'noindex, follow', 'Link': '<https://www.soeren-hentzschel.at/wp-json/>; rel="https://api.w.org/"', 'Upgrade': 'h2,h2c', 'Connection': 'Upgrade', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/rss+xml; charset=UTF-8')>

Traceback (most recent call last):
  File "test.py", line 21, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "test.py", line 16, in main
    html = await fetch(session, 'https://www.soeren-hentzschel.at/feed/')
  File "test.py", line 11, in fetch
    return await response.text()
  File "/home/lukas/.virtualenvs/tmp-b68734831c06cd1f/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 830, in text
    await self.read()
  File "/home/lukas/.virtualenvs/tmp-b68734831c06cd1f/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 799, in read
    self._content = await self.content.read()
  File "/home/lukas/.virtualenvs/tmp-b68734831c06cd1f/lib/python3.6/site-packages/aiohttp/streams.py", line 312, in read
    block = await self.readany()
  File "/home/lukas/.virtualenvs/tmp-b68734831c06cd1f/lib/python3.6/site-packages/aiohttp/streams.py", line 328, in readany
    await self._wait('readany')
  File "/home/lukas/.virtualenvs/tmp-b68734831c06cd1f/lib/python3.6/site-packages/aiohttp/streams.py", line 250, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

[Findus23]

#2588 seems to fix the issue for me:

➜  ~ pip install git+https://github.com/lphuberdeau/aiohttp.git@limit-upgrade-values -U
Collecting git+https://github.com/lphuberdeau/aiohttp.git@limit-upgrade-values
  Cloning https://github.com/lphuberdeau/aiohttp.git (to limit-upgrade-values) to /tmp/pip-lapcl_xq-build
Requirement already up-to-date: chardet in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already up-to-date: multidict>=3.0.0 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already up-to-date: async_timeout>=1.2.0 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already up-to-date: yarl>=0.11 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already up-to-date: idna>=2.0 in ./lib/python3.6/site-packages (from yarl>=0.11->aiohttp==3.0.0a0)
Installing collected packages: aiohttp
  Found existing installation: aiohttp 3.0.0a0
    Uninstalling aiohttp-3.0.0a0:
      Successfully uninstalled aiohttp-3.0.0a0
  Running setup.py install for aiohttp ... done
Successfully installed aiohttp-3.0.0a0
➜  ~ python test.py                                                                    
<ClientResponse(https://www.soeren-hentzschel.at/feed/) [200 OK]>
<CIMultiDictProxy('Date': 'Tue, 30 Jan 2018 12:12:20 GMT', 'Server': 'Apache', 'Last-Modified': 'Mon, 29 Jan 2018 22:41:58 GMT', 'ETag': '"95b04997e00647ea795eb5503aad9780"', 'X-Robots-Tag': 'noindex, follow', 'Link': '<https://www.soeren-hentzschel.at/wp-json/>; rel="https://api.w.org/"', 'Upgrade': 'h2,h2c', 'Connection': 'Upgrade', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/rss+xml; charset=UTF-8')>

<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
        xmlns:content="http://purl.org/rss/1.0/modules/content/"
        xmlns:wfw="http://wellformedweb.org/CommentAPI/"
        xmlns:dc="http://purl.org/dc/elements/1.1/"
[AND THE REST OF THE RSS FEED]

[asvetlov]

Fixed on master.

[Findus23]

Isn't the issue that without #2588 the response isn't handled correctly?
So what should I do to get the correct response? I can't fix the Apache setup as it isn't my website.