New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3.3 socket read timeouts not working correctly #3053

Closed
thehesiod opened this Issue Jun 4, 2018 · 2 comments

Comments

Projects
None yet
3 participants
@thehesiod
Contributor

thehesiod commented Jun 4, 2018

@asvetlov looks like the new sock_read timeout implementation is not working, see testcase:

import multiprocessing
import asyncio
import aiohttp
import socket

host = "localhost"


def get_free_tcp_port():
    sckt = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sckt.bind((host, 0))
    addr, port = sckt.getsockname()
    sckt.close()
    return port


class AIOServer(multiprocessing.Process):
    """
    This is a mock AWS service which will 5 seconds before returning
    a response to test socket timeouts.
    """
    def __init__(self):
        super().__init__(target=self._run)
        self._loop = None
        self._port = get_free_tcp_port()
        self.endpoint_url = 'http://{}:{}'.format(host, self._port)
        self.daemon = True  # die when parent dies

    def _run(self):
        asyncio.set_event_loop(asyncio.new_event_loop())
        app = aiohttp.web.Application()
        app.router.add_route('*', '/ok', self.ok)
        app.router.add_route('*', '/{anything:.*}', self.stream_handler)

        try:
            aiohttp.web.run_app(app, host=host, port=self._port,
                                handle_signals=False)
        except BaseException:
            pytest.fail('unable to start and connect to aiohttp server')
            raise

    async def __aenter__(self):
        self.start()
        await self._wait_until_up()
        return self

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        try:
            self.terminate()
        except:
            pytest.fail("Unable to shut down server")
            raise

    async def ok(self, request):
        return aiohttp.web.Response()

    async def stream_handler(self, request):
        # Without the Content-Type, most (all?) browsers will not render
        # partially downloaded content. Note, the response type is
        # StreamResponse not Response.
        resp = aiohttp.web.StreamResponse(status=200, reason='OK',
                              headers={'Content-Type': 'text/html'})

        await resp.prepare(request)
        await asyncio.sleep(5, loop=self._loop)
        await resp.drain()
        return resp

    async def _wait_until_up(self):
        async with aiohttp.ClientSession() as session:
            for i in range(0, 30):
                if self.exitcode is not None:
                    pytest.fail('unable to start and connect to aiohttp server')
                    return

                try:
                    # we need to bypass the proxies due to monkey patches
                    await session.get(self.endpoint_url + '/ok', timeout=0.5)
                    return
                except (aiohttp.ClientConnectionError, asyncio.TimeoutError):
                    await asyncio.sleep(0.5)
                except BaseException:
                    pytest.fail('unable to start and connect to aiohttp server')
                    raise

        pytest.fail('unable to start and connect to aiohttp server')


async def main():
    async with AIOServer() as server, \
            aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(sock_connect=1, sock_read=1)) as session:

        try:
            response = await session.get(server.endpoint_url)
            await response.read()
        except asyncio.TimeoutError:
            print("Success")

        print("failure")


if __name__ == '__main__':
    asyncio.get_event_loop().run_until_complete(main())

shikhir-arora added a commit to GieselaDev/discord.py that referenced this issue Aug 28, 2018

Update aiohttp to 3.4.0 (changelog below)
==================

Features
--------

- Add type hints (`3049 <https://github.com/aio-libs/aiohttp/pull/3049>`_)
- Add ``raise_for_status`` request parameter (`3073 <https://github.com/aio-libs/aiohttp/pull/3073>`_)
- Add type hints to HTTP client (`3092 <https://github.com/aio-libs/aiohttp/pull/3092>`_)
- Minor server optimizations (`3095 <https://github.com/aio-libs/aiohttp/pull/3095>`_)
- Preserve the cause when `HTTPException` is raised from another exception. (`3096 <https://github.com/aio-libs/aiohttp/pull/3096>`_)
- Add `close_boundary` option in `MultipartWriter.write` method. Support streaming (`3104 <https://github.com/aio-libs/aiohttp/pull/3104>`_)
- Added a ``remove_slash`` option to the ``normalize_path_middleware`` factory. (`3173 <https://github.com/aio-libs/aiohttp/pull/3173>`_)
- The class `AbstractRouteDef` is importable from `aiohttp.web`. (`3183 <https://github.com/aio-libs/aiohttp/pull/3183>`_)


Bugfixes
--------

- Prevent double closing when client connection is released before the
last ``data_received()`` callback. (`3031 <https://github.com/aio-libs/aiohttp/pull/3031>`_)
- Make redirect with `normalize_path_middleware` work when using url encoded paths. (`3051 <https://github.com/aio-libs/aiohttp/pull/3051>`_)
- Postpone web task creation to connection establishment. (`3052 <https://github.com/aio-libs/aiohttp/pull/3052>`_)
- Fix ``sock_read`` timeout. (`3053 <https://github.com/aio-libs/aiohttp/pull/3053>`_)
- When using a server-request body as the `data=` argument of a client request, iterate over the content with `readany` instead of `readline` to avoid `Line too long` errors. (`3054 <https://github.com/aio-libs/aiohttp/pull/3054>`_)
- fix `UrlDispatcher` has no attribute `add_options`, add `web.options` (`3062 <https://github.com/aio-libs/aiohttp/pull/3062>`_)
- correct filename in content-disposition with multipart body (`3064 <https://github.com/aio-libs/aiohttp/pull/3064>`_)
- Many HTTP proxies has buggy keepalive support.
Let's not reuse connection but close it after processing every response. (`3070 <https://github.com/aio-libs/aiohttp/pull/3070>`_)
- raise 413 "Payload Too Large" rather than raising ValueError in request.post()
Add helpful debug message to 413 responses (`3087 <https://github.com/aio-libs/aiohttp/pull/3087>`_)
- Fix `StreamResponse` equality, now that they are `MutableMapping` objects. (`3100 <https://github.com/aio-libs/aiohttp/pull/3100>`_)
- Fix server request objects comparison (`3116 <https://github.com/aio-libs/aiohttp/pull/3116>`_)
- Do not hang on `206 Partial Content` response with `Content-Encoding: gzip` (`3123 <https://github.com/aio-libs/aiohttp/pull/3123>`_)
- Fix timeout precondition checkers (`3145 <https://github.com/aio-libs/aiohttp/pull/3145>`_)


Improved Documentation
----------------------

- Add a new FAQ entry that clarifies that you should not reuse response
objects in middleware functions. (`3020 <https://github.com/aio-libs/aiohttp/pull/3020>`_)
- Add FAQ section "Why is creating a ClientSession outside of an event loop dangerous?" (`3072 <https://github.com/aio-libs/aiohttp/pull/3072>`_)
- Fix link to Rambler (`3115 <https://github.com/aio-libs/aiohttp/pull/3115>`_)
- Fix TCPSite documentation on the Server Reference page. (`3146 <https://github.com/aio-libs/aiohttp/pull/3146>`_)
- Fix documentation build configuration file for Windows. (`3147 <https://github.com/aio-libs/aiohttp/pull/3147>`_)
- Remove no longer existing lingering_timeout parameter of Application.make_handler from documentation. (`3151 <https://github.com/aio-libs/aiohttp/pull/3151>`_)
- Mention that ``app.make_handler`` is deprecated, recommend to use runners
API instead. (`3157 <https://github.com/aio-libs/aiohttp/pull/3157>`_)


Deprecations and Removals
-------------------------

- Drop ``loop.current_task()`` from ``helpers.current_task()`` (`2826 <https://github.com/aio-libs/aiohttp/pull/2826>`_)
- Drop ``reader`` parameter from ``request.multipart()``. (`3090 <https://github.com/aio-libs/aiohttp/pull/3090>`_)
@mcarans

This comment has been minimized.

mcarans commented Aug 29, 2018

@asvetlov Which aiohttp version is this fixed in 3.3.1 or 3.4.0 (it's mentioned in both changelogs)?

@asvetlov

This comment has been minimized.

Member

asvetlov commented Aug 29, 2018

Both versions

kornicameister added a commit to kornicameister/korni-stats-collector that referenced this issue Sep 17, 2018

Update aiohttp to 3.4.4 (#94)
This PR updates [aiohttp](https://pypi.org/project/aiohttp) from **3.3.2** to **3.4.4**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.4.4
   ```
   ==================

- Fix installation from sources when compiling toolkit is not available (`3241 &lt;https://github.com/aio-libs/aiohttp/pull/3241&gt;`_)
   ```
   
  
  
   ### 3.4.3
   ```
   ==================

- Add ``app.pre_frozen`` state to properly handle startup signals in sub-applications. (`3237 &lt;https://github.com/aio-libs/aiohttp/pull/3237&gt;`_)
   ```
   
  
  
   ### 3.4.2
   ```
   ==================

- Fix ``iter_chunks`` type annotation (`3230 &lt;https://github.com/aio-libs/aiohttp/pull/3230&gt;`_)
   ```
   
  
  
   ### 3.4.1
   ```
   ==================

- Fix empty header parsing regression. (`3218 &lt;https://github.com/aio-libs/aiohttp/pull/3218&gt;`_)
- Fix BaseRequest.raw_headers doc. (`3215 &lt;https://github.com/aio-libs/aiohttp/pull/3215&gt;`_)
- Fix documentation building on ReadTheDocs (`3221 &lt;https://github.com/aio-libs/aiohttp/pull/3221&gt;`_)
   ```
   
  
  
   ### 3.4.0
   ```
   ==================

Features
--------

- Add type hints (`3049 &lt;https://github.com/aio-libs/aiohttp/pull/3049&gt;`_)
- Add ``raise_for_status`` request parameter (`3073 &lt;https://github.com/aio-libs/aiohttp/pull/3073&gt;`_)
- Add type hints to HTTP client (`3092 &lt;https://github.com/aio-libs/aiohttp/pull/3092&gt;`_)
- Minor server optimizations (`3095 &lt;https://github.com/aio-libs/aiohttp/pull/3095&gt;`_)
- Preserve the cause when `HTTPException` is raised from another exception. (`3096 &lt;https://github.com/aio-libs/aiohttp/pull/3096&gt;`_)
- Add `close_boundary` option in `MultipartWriter.write` method. Support streaming (`3104 &lt;https://github.com/aio-libs/aiohttp/pull/3104&gt;`_)
- Added a ``remove_slash`` option to the ``normalize_path_middleware`` factory. (`3173 &lt;https://github.com/aio-libs/aiohttp/pull/3173&gt;`_)
- The class `AbstractRouteDef` is importable from `aiohttp.web`. (`3183 &lt;https://github.com/aio-libs/aiohttp/pull/3183&gt;`_)


Bugfixes
--------

- Prevent double closing when client connection is released before the
  last ``data_received()`` callback. (`3031 &lt;https://github.com/aio-libs/aiohttp/pull/3031&gt;`_)
- Make redirect with `normalize_path_middleware` work when using url encoded paths. (`3051 &lt;https://github.com/aio-libs/aiohttp/pull/3051&gt;`_)
- Postpone web task creation to connection establishment. (`3052 &lt;https://github.com/aio-libs/aiohttp/pull/3052&gt;`_)
- Fix ``sock_read`` timeout. (`3053 &lt;https://github.com/aio-libs/aiohttp/pull/3053&gt;`_)
- When using a server-request body as the `data=` argument of a client request, iterate over the content with `readany` instead of `readline` to avoid `Line too long` errors. (`3054 &lt;https://github.com/aio-libs/aiohttp/pull/3054&gt;`_)
- fix `UrlDispatcher` has no attribute `add_options`, add `web.options` (`3062 &lt;https://github.com/aio-libs/aiohttp/pull/3062&gt;`_)
- correct filename in content-disposition with multipart body (`3064 &lt;https://github.com/aio-libs/aiohttp/pull/3064&gt;`_)
- Many HTTP proxies has buggy keepalive support.
  Let&#39;s not reuse connection but close it after processing every response. (`3070 &lt;https://github.com/aio-libs/aiohttp/pull/3070&gt;`_)
- raise 413 &quot;Payload Too Large&quot; rather than raising ValueError in request.post()
  Add helpful debug message to 413 responses (`3087 &lt;https://github.com/aio-libs/aiohttp/pull/3087&gt;`_)
- Fix `StreamResponse` equality, now that they are `MutableMapping` objects. (`3100 &lt;https://github.com/aio-libs/aiohttp/pull/3100&gt;`_)
- Fix server request objects comparison (`3116 &lt;https://github.com/aio-libs/aiohttp/pull/3116&gt;`_)
- Do not hang on `206 Partial Content` response with `Content-Encoding: gzip` (`3123 &lt;https://github.com/aio-libs/aiohttp/pull/3123&gt;`_)
- Fix timeout precondition checkers (`3145 &lt;https://github.com/aio-libs/aiohttp/pull/3145&gt;`_)


Improved Documentation
----------------------

- Add a new FAQ entry that clarifies that you should not reuse response
  objects in middleware functions. (`3020 &lt;https://github.com/aio-libs/aiohttp/pull/3020&gt;`_)
- Add FAQ section &quot;Why is creating a ClientSession outside of an event loop dangerous?&quot; (`3072 &lt;https://github.com/aio-libs/aiohttp/pull/3072&gt;`_)
- Fix link to Rambler (`3115 &lt;https://github.com/aio-libs/aiohttp/pull/3115&gt;`_)
- Fix TCPSite documentation on the Server Reference page. (`3146 &lt;https://github.com/aio-libs/aiohttp/pull/3146&gt;`_)
- Fix documentation build configuration file for Windows. (`3147 &lt;https://github.com/aio-libs/aiohttp/pull/3147&gt;`_)
- Remove no longer existing lingering_timeout parameter of Application.make_handler from documentation. (`3151 &lt;https://github.com/aio-libs/aiohttp/pull/3151&gt;`_)
- Mention that ``app.make_handler`` is deprecated, recommend to use runners
  API instead. (`3157 &lt;https://github.com/aio-libs/aiohttp/pull/3157&gt;`_)


Deprecations and Removals
-------------------------

- Drop ``loop.current_task()`` from ``helpers.current_task()`` (`2826 &lt;https://github.com/aio-libs/aiohttp/pull/2826&gt;`_)
- Drop ``reader`` parameter from ``request.multipart()``. (`3090 &lt;https://github.com/aio-libs/aiohttp/pull/3090&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/aiohttp
  - Changelog: https://pyup.io/changelogs/aiohttp/
  - Repo: https://github.com/aio-libs/aiohttp
</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment