New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`Line too long` error when proxying a request #3054

Merged
merged 1 commit into from Jun 5, 2018

Conversation

Projects
None yet
3 participants
@arthurdarcet
Contributor

arthurdarcet commented Jun 5, 2018

When sending a request with a body set to a StreamReader (ie when your aiohttp server proxy a request to another server), the body could contain very long lines.

Such line go over the high water mark, and trigger an exception.
The payload module should not write the StreamReader by (async)-iterating over the lines, but should instead iterate over whatever is available. This patch implements that.

Here is a small script reproducing the error:

import aiohttp
import asyncio


async def handle(request):
	return aiohttp.web.Response(body=b'0' * 2 ** 18)

async def main(loop):
	app = aiohttp.web.Application()
	app.add_routes([aiohttp.web.get('/', handle)])
	handler = app.make_handler()
	await loop.create_server(handler, '0.0.0.0', 1111)

	async with aiohttp.ClientSession() as session:
		async with session.get('http://0.0.0.0:1111') as resp:
			# this is just a stream of 2**18 0-bytes
			stream = resp.content

			async with session.post('http://httpbin.org/anything', data=stream):
				pass


loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))

@arthurdarcet arthurdarcet force-pushed the arthurdarcet:payload-readline branch from 36632cc to f2f1a22 Jun 5, 2018

When writing a payload from a StreamReader, use readany instead of re…
…adline: the lines could be arbitrarily long

@arthurdarcet arthurdarcet force-pushed the arthurdarcet:payload-readline branch from f2f1a22 to 5106d36 Jun 5, 2018

@codecov-io

This comment has been minimized.

codecov-io commented Jun 5, 2018

Codecov Report

Merging #3054 into master will increase coverage by <.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3054      +/-   ##
==========================================
+ Coverage   98.11%   98.11%   +<.01%     
==========================================
  Files          42       42              
  Lines        7735     7739       +4     
  Branches     1348     1348              
==========================================
+ Hits         7589     7593       +4     
  Misses         51       51              
  Partials       95       95
Impacted Files Coverage Δ
aiohttp/payload.py 98.97% <100%> (+0.02%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 466cb4a...5106d36. Read the comment docs.

@asvetlov asvetlov merged commit 0775094 into aio-libs:master Jun 5, 2018

3 checks passed

codecov/patch 100% of diff hit (target 98.11%)
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@asvetlov

This comment has been minimized.

Member

asvetlov commented Jun 5, 2018

Thanks!

asvetlov added a commit that referenced this pull request Jun 5, 2018

When writing a payload from a StreamReader, use readany instead of re…
…adline: the lines could be arbitrarily long (#3054)
@arthurdarcet

This comment has been minimized.

Contributor

arthurdarcet commented Jun 6, 2018

Thank you for the release!

@arthurdarcet arthurdarcet deleted the arthurdarcet:payload-readline branch Jun 6, 2018

@asvetlov

This comment has been minimized.

Member

asvetlov commented Jun 6, 2018

Welcome!

shikhir-arora added a commit to GieselaDev/discord.py that referenced this pull request Aug 28, 2018

Update aiohttp to 3.4.0 (changelog below)
==================

Features
--------

- Add type hints (`3049 &lt;https://github.com/aio-libs/aiohttp/pull/3049&gt;`_)
- Add ``raise_for_status`` request parameter (`3073 &lt;https://github.com/aio-libs/aiohttp/pull/3073&gt;`_)
- Add type hints to HTTP client (`3092 &lt;https://github.com/aio-libs/aiohttp/pull/3092&gt;`_)
- Minor server optimizations (`3095 &lt;https://github.com/aio-libs/aiohttp/pull/3095&gt;`_)
- Preserve the cause when `HTTPException` is raised from another exception. (`3096 &lt;https://github.com/aio-libs/aiohttp/pull/3096&gt;`_)
- Add `close_boundary` option in `MultipartWriter.write` method. Support streaming (`3104 &lt;https://github.com/aio-libs/aiohttp/pull/3104&gt;`_)
- Added a ``remove_slash`` option to the ``normalize_path_middleware`` factory. (`3173 &lt;https://github.com/aio-libs/aiohttp/pull/3173&gt;`_)
- The class `AbstractRouteDef` is importable from `aiohttp.web`. (`3183 &lt;https://github.com/aio-libs/aiohttp/pull/3183&gt;`_)


Bugfixes
--------

- Prevent double closing when client connection is released before the
last ``data_received()`` callback. (`3031 &lt;https://github.com/aio-libs/aiohttp/pull/3031&gt;`_)
- Make redirect with `normalize_path_middleware` work when using url encoded paths. (`3051 &lt;https://github.com/aio-libs/aiohttp/pull/3051&gt;`_)
- Postpone web task creation to connection establishment. (`3052 &lt;https://github.com/aio-libs/aiohttp/pull/3052&gt;`_)
- Fix ``sock_read`` timeout. (`3053 &lt;https://github.com/aio-libs/aiohttp/pull/3053&gt;`_)
- When using a server-request body as the `data=` argument of a client request, iterate over the content with `readany` instead of `readline` to avoid `Line too long` errors. (`3054 &lt;https://github.com/aio-libs/aiohttp/pull/3054&gt;`_)
- fix `UrlDispatcher` has no attribute `add_options`, add `web.options` (`3062 &lt;https://github.com/aio-libs/aiohttp/pull/3062&gt;`_)
- correct filename in content-disposition with multipart body (`3064 &lt;https://github.com/aio-libs/aiohttp/pull/3064&gt;`_)
- Many HTTP proxies has buggy keepalive support.
Let&#39;s not reuse connection but close it after processing every response. (`3070 &lt;https://github.com/aio-libs/aiohttp/pull/3070&gt;`_)
- raise 413 &quot;Payload Too Large&quot; rather than raising ValueError in request.post()
Add helpful debug message to 413 responses (`3087 &lt;https://github.com/aio-libs/aiohttp/pull/3087&gt;`_)
- Fix `StreamResponse` equality, now that they are `MutableMapping` objects. (`3100 &lt;https://github.com/aio-libs/aiohttp/pull/3100&gt;`_)
- Fix server request objects comparison (`3116 &lt;https://github.com/aio-libs/aiohttp/pull/3116&gt;`_)
- Do not hang on `206 Partial Content` response with `Content-Encoding: gzip` (`3123 &lt;https://github.com/aio-libs/aiohttp/pull/3123&gt;`_)
- Fix timeout precondition checkers (`3145 &lt;https://github.com/aio-libs/aiohttp/pull/3145&gt;`_)


Improved Documentation
----------------------

- Add a new FAQ entry that clarifies that you should not reuse response
objects in middleware functions. (`3020 &lt;https://github.com/aio-libs/aiohttp/pull/3020&gt;`_)
- Add FAQ section &quot;Why is creating a ClientSession outside of an event loop dangerous?&quot; (`3072 &lt;https://github.com/aio-libs/aiohttp/pull/3072&gt;`_)
- Fix link to Rambler (`3115 &lt;https://github.com/aio-libs/aiohttp/pull/3115&gt;`_)
- Fix TCPSite documentation on the Server Reference page. (`3146 &lt;https://github.com/aio-libs/aiohttp/pull/3146&gt;`_)
- Fix documentation build configuration file for Windows. (`3147 &lt;https://github.com/aio-libs/aiohttp/pull/3147&gt;`_)
- Remove no longer existing lingering_timeout parameter of Application.make_handler from documentation. (`3151 &lt;https://github.com/aio-libs/aiohttp/pull/3151&gt;`_)
- Mention that ``app.make_handler`` is deprecated, recommend to use runners
API instead. (`3157 &lt;https://github.com/aio-libs/aiohttp/pull/3157&gt;`_)


Deprecations and Removals
-------------------------

- Drop ``loop.current_task()`` from ``helpers.current_task()`` (`2826 &lt;https://github.com/aio-libs/aiohttp/pull/2826&gt;`_)
- Drop ``reader`` parameter from ``request.multipart()``. (`3090 &lt;https://github.com/aio-libs/aiohttp/pull/3090&gt;`_)

kornicameister added a commit to kornicameister/korni-stats-collector that referenced this pull request Sep 17, 2018

Update aiohttp to 3.4.4 (#94)
This PR updates [aiohttp](https://pypi.org/project/aiohttp) from **3.3.2** to **3.4.4**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.4.4
   ```
   ==================

- Fix installation from sources when compiling toolkit is not available (`3241 &lt;https://github.com/aio-libs/aiohttp/pull/3241&gt;`_)
   ```
   
  
  
   ### 3.4.3
   ```
   ==================

- Add ``app.pre_frozen`` state to properly handle startup signals in sub-applications. (`3237 &lt;https://github.com/aio-libs/aiohttp/pull/3237&gt;`_)
   ```
   
  
  
   ### 3.4.2
   ```
   ==================

- Fix ``iter_chunks`` type annotation (`3230 &lt;https://github.com/aio-libs/aiohttp/pull/3230&gt;`_)
   ```
   
  
  
   ### 3.4.1
   ```
   ==================

- Fix empty header parsing regression. (`3218 &lt;https://github.com/aio-libs/aiohttp/pull/3218&gt;`_)
- Fix BaseRequest.raw_headers doc. (`3215 &lt;https://github.com/aio-libs/aiohttp/pull/3215&gt;`_)
- Fix documentation building on ReadTheDocs (`3221 &lt;https://github.com/aio-libs/aiohttp/pull/3221&gt;`_)
   ```
   
  
  
   ### 3.4.0
   ```
   ==================

Features
--------

- Add type hints (`3049 &lt;https://github.com/aio-libs/aiohttp/pull/3049&gt;`_)
- Add ``raise_for_status`` request parameter (`3073 &lt;https://github.com/aio-libs/aiohttp/pull/3073&gt;`_)
- Add type hints to HTTP client (`3092 &lt;https://github.com/aio-libs/aiohttp/pull/3092&gt;`_)
- Minor server optimizations (`3095 &lt;https://github.com/aio-libs/aiohttp/pull/3095&gt;`_)
- Preserve the cause when `HTTPException` is raised from another exception. (`3096 &lt;https://github.com/aio-libs/aiohttp/pull/3096&gt;`_)
- Add `close_boundary` option in `MultipartWriter.write` method. Support streaming (`3104 &lt;https://github.com/aio-libs/aiohttp/pull/3104&gt;`_)
- Added a ``remove_slash`` option to the ``normalize_path_middleware`` factory. (`3173 &lt;https://github.com/aio-libs/aiohttp/pull/3173&gt;`_)
- The class `AbstractRouteDef` is importable from `aiohttp.web`. (`3183 &lt;https://github.com/aio-libs/aiohttp/pull/3183&gt;`_)


Bugfixes
--------

- Prevent double closing when client connection is released before the
  last ``data_received()`` callback. (`3031 &lt;https://github.com/aio-libs/aiohttp/pull/3031&gt;`_)
- Make redirect with `normalize_path_middleware` work when using url encoded paths. (`3051 &lt;https://github.com/aio-libs/aiohttp/pull/3051&gt;`_)
- Postpone web task creation to connection establishment. (`3052 &lt;https://github.com/aio-libs/aiohttp/pull/3052&gt;`_)
- Fix ``sock_read`` timeout. (`3053 &lt;https://github.com/aio-libs/aiohttp/pull/3053&gt;`_)
- When using a server-request body as the `data=` argument of a client request, iterate over the content with `readany` instead of `readline` to avoid `Line too long` errors. (`3054 &lt;https://github.com/aio-libs/aiohttp/pull/3054&gt;`_)
- fix `UrlDispatcher` has no attribute `add_options`, add `web.options` (`3062 &lt;https://github.com/aio-libs/aiohttp/pull/3062&gt;`_)
- correct filename in content-disposition with multipart body (`3064 &lt;https://github.com/aio-libs/aiohttp/pull/3064&gt;`_)
- Many HTTP proxies has buggy keepalive support.
  Let&#39;s not reuse connection but close it after processing every response. (`3070 &lt;https://github.com/aio-libs/aiohttp/pull/3070&gt;`_)
- raise 413 &quot;Payload Too Large&quot; rather than raising ValueError in request.post()
  Add helpful debug message to 413 responses (`3087 &lt;https://github.com/aio-libs/aiohttp/pull/3087&gt;`_)
- Fix `StreamResponse` equality, now that they are `MutableMapping` objects. (`3100 &lt;https://github.com/aio-libs/aiohttp/pull/3100&gt;`_)
- Fix server request objects comparison (`3116 &lt;https://github.com/aio-libs/aiohttp/pull/3116&gt;`_)
- Do not hang on `206 Partial Content` response with `Content-Encoding: gzip` (`3123 &lt;https://github.com/aio-libs/aiohttp/pull/3123&gt;`_)
- Fix timeout precondition checkers (`3145 &lt;https://github.com/aio-libs/aiohttp/pull/3145&gt;`_)


Improved Documentation
----------------------

- Add a new FAQ entry that clarifies that you should not reuse response
  objects in middleware functions. (`3020 &lt;https://github.com/aio-libs/aiohttp/pull/3020&gt;`_)
- Add FAQ section &quot;Why is creating a ClientSession outside of an event loop dangerous?&quot; (`3072 &lt;https://github.com/aio-libs/aiohttp/pull/3072&gt;`_)
- Fix link to Rambler (`3115 &lt;https://github.com/aio-libs/aiohttp/pull/3115&gt;`_)
- Fix TCPSite documentation on the Server Reference page. (`3146 &lt;https://github.com/aio-libs/aiohttp/pull/3146&gt;`_)
- Fix documentation build configuration file for Windows. (`3147 &lt;https://github.com/aio-libs/aiohttp/pull/3147&gt;`_)
- Remove no longer existing lingering_timeout parameter of Application.make_handler from documentation. (`3151 &lt;https://github.com/aio-libs/aiohttp/pull/3151&gt;`_)
- Mention that ``app.make_handler`` is deprecated, recommend to use runners
  API instead. (`3157 &lt;https://github.com/aio-libs/aiohttp/pull/3157&gt;`_)


Deprecations and Removals
-------------------------

- Drop ``loop.current_task()`` from ``helpers.current_task()`` (`2826 &lt;https://github.com/aio-libs/aiohttp/pull/2826&gt;`_)
- Drop ``reader`` parameter from ``request.multipart()``. (`3090 &lt;https://github.com/aio-libs/aiohttp/pull/3090&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/aiohttp
  - Changelog: https://pyup.io/changelogs/aiohttp/
  - Repo: https://github.com/aio-libs/aiohttp
</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment