StreamReader.readchunk returns false-negative indicator end_of_HTTP_chunk #3361
Closed
Description
Long story short
readchunk function returns False after full chunk received.
It causes unexpected delays and complicated chunk parser in chunk-dependent protocols (In our case: "Watch keys" in etcd HTTP API)
Expected behaviour
end_of_HTTP_chunk indicator equals True when full chunk received.
Actual behaviour
indicator equals False when tail of chunk ('\r\n') comes separately or with next chunk
Steps to reproduce
Simple web-server to reproduce the issue:
import socket
import time
def main():
ls = socket.socket()
ls.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
ls.bind(('0.0.0.0', 1234))
ls.listen(1)
print('server is up')
while True:
s, addr = ls.accept()
s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
# read request
request = b''
while True:
data = s.recv(65536)
if not data:
print('err')
return
request += data
if b'\r\n\r\n' in request:
break
print('read complete')
# send headers
s.send(b'\r\n'.join([
b'HTTP/1.0 200 OK',
b'transfer-encoding: chunked',
b'connection: close',
b'',
b'',
]))
# send first part of the chunk
s.send(b'3\r\n***')
time.sleep(0.1)
# end of chunk
s.send(b'\r\n')
# time.sleep(0.1) # uncomment to send end of chunk separately
# next chunk
s.send(b'5\r\n@@@@@\r\n')
s.send(b'0\r\n\r\n')
s.close()
main()Client side:
import asyncio
import aiohttp
async def main():
async with aiohttp.ClientSession() as session:
async with session.get(url='http://localhost:1234/', timeout=None) as resp:
buffer = b""
async for raw_data, end_of_http_chunk in resp.content.iter_chunks():
print(raw_data, end_of_http_chunk)
buffer += raw_data
if not end_of_http_chunk:
continue
print('full chunk received', buffer)
buffer = b""
print('stop iteration', buffer)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())Expected result:
b'***' True
full chunk received b'***'
b'@@@@@' True
full chunk received b'@@@@@'
stop iteration b''
Actual:
b'***' False
b'@@@@@' True
full chunk received b'***@@@@@'
stop iteration b''
Environment
client side issue
aiohttp: 3.2.0, 3.4.4, master
OS: linux