Redesign shutdown process #7718
Conversation
for more information, see https://pre-commit.ci
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
psf-chronographer
bot
added
the
bot:chronographer:provided
Does this pose a deadlock threat? |
I think there is the possibility that a task could stop the program exiting, but at that point the rest of shutdown has already happened, so it'll be as safe as it can be for someone to kill the program. This step is unchanged from the 3.8 release. |
In fact, I just remembered that this code is copied from asyncio.run(), so it'll be consistent with asyncio generally. |
Co-authored-by: Sviatoslav Sydorenko <sviat@redhat.com>
Codecov Report
@@ Coverage Diff @@
## master #7718 +/- ##
==========================================
+ Coverage 97.36% 97.38% +0.02%
==========================================
Files 106 106
Lines 31718 31798 +80
Branches 3633 3645 +12
==========================================
+ Hits 30881 30967 +86
+ Misses 633 627 -6
Partials 204 204
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 5 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
I'm merging this, if there's any feedback, I can update it before the final release (maybe in a week or so). |
Dreamsorcerer
added
the
backport-3.9
Backport to 3.9: 💔 cherry-picking failed — conflicts found❌ Failed to cleanly apply 107dd24 on top of patchback/backports/3.9/107dd2445720cf354c84c3a271691f14e2e2875e/pr-7718 Backporting merged PR #7718 into master
🤖 @patchback |
This contains a few changes to rework the shutdown process to be a bit clearer and better match user expectations. To summarise, I'm proposing the shutdown process should look like this now: 1. Stop each site listening for new connections. 2. Close idle keep-alive connections (and set active ones to close upon completion). 3. Call on_shutdown signal (should be used to close websocket connections). 4. Wait for running tasks to complete (with timeout). 5. Close any remaining connections (with timeout again). 6. Call on_cleanup signal. 7. Cancel any remaining tasks and wait on them. Docs demo preview: https://aiohttp--7718.org.readthedocs.build/en/7718/web_advanced.html#graceful-shutdown --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <sviat@redhat.com> (cherry picked from commit 107dd24)
Sorry for the late feedback, I wasn't shut if this was ready to test or not. I tested this with Home Assistant, and shutdown is delayed if we don't patch out Home Assistant has long running tasks that are created after Patching out |
| @@ -0,0 +1,3 @@ | |||
| Fixed keep-alive connections stopping a graceful shutdown. | |||
There was a problem hiding this comment.
@Dreamsorcerer “fixed” implies a “bugfix” changelog note type, but the filename says “feature”. If it's both, it might be worth splitting into two fragments.
Maybe, you could shield them from being cancelled, doing an unshielded cancellation later, manually? |
What if there was an interface to augment the new |
|
@Dreamsorcerer I'm not very familiar with the shutdown internals, so I can't contribute an involved feedback. Just want to say that I don't see anything obviously wrong, and it seems like a good idea. |
@bdraco could you explain why? |
Home Assistant has its own tasks that are created with our helper that we can track, but there are also ~1600 dependencies that can start their own tasks so we don't have a direct way to manage them. While most of them will listen for a stop event and cancel the tasks, there are many custom integrations (and libraries) that don't and we can't wait for them to shutdown because we would end up waiting forever in many cases. To give some better context, there are likely quite a few integrations that have a task running a retry loop when a device is offline that keep trying to reconnect forever until Home Assistant shuts down the event loop from under them. |
|
@bdraco have you tried exploring having a separate event loop for those? |
|
@bdraco We already discussed this in the testing issue. My understanding is that HA is using the runners directly, so in this PR I've moved the wait logic into run_app(). This means it should have no impact on HA anymore, and there isn't anything there to patch out. For users using run_app(), this is the entry point of the event loop and we expect to control the lifecycle of the event loop. |
|
(This is also now highlighted in the docs section linked at the top of the PR) |
I agree that I shouldn't be seeing a problem anymore, but I can reproduce still. I will double check that I haven't screwed something up with my testing. |
|
TLDR: This PR solves the issue, there is an unrelated regression on my Home Assistant The problem goes away when I use Home Assistant stable + this PR. I must have a regression my Home Assistant I didn't notice since I was running 3.9 prior to this PR which was always shutting down slowly (I didn't want to forget to make sure this was sorted) since it wasn't patched and I assumed the slow shutdown was still from that. Thanks for getting this sorted. Sorry about the bad test. |
|
I found the bad commit, its definitely unrelated. Sorry again for the side trip. The good news is that 3.9 is working well in production 👍 |
[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [aiohttp](https://togithub.com/aio-libs/aiohttp) | `==3.8.6` -> `==3.9.0` | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>aio-libs/aiohttp (aiohttp)</summary> ### [`v3.9.0`](https://togithub.com/aio-libs/aiohttp/blob/HEAD/CHANGES.rst#390-2023-11-18) [Compare Source](https://togithub.com/aio-libs/aiohttp/compare/v3.8.6...v3.9.0) \================== ## Features - Introduced `AppKey` for static typing support of `Application` storage. See https://docs.aiohttp.org/en/stable/web_advanced.html#application-s-config `#​5864 <https://github.com/aio-libs/aiohttp/issues/5864>`\_ - Added a graceful shutdown period which allows pending tasks to complete before the application's cleanup is called. The period can be adjusted with the `shutdown_timeout` parameter. -- by :user:`Dreamsorcerer`. See https://docs.aiohttp.org/en/latest/web_advanced.html#graceful-shutdown `#​7188 <https://github.com/aio-libs/aiohttp/issues/7188>`\_ - Added `handler_cancellation <https://docs.aiohttp.org/en/stable/web_advanced.html#web-handler-cancellation>`\_ parameter to cancel web handler on client disconnection. -- by :user:`mosquito` This (optionally) reintroduces a feature removed in a previous release. Recommended for those looking for an extra level of protection against denial-of-service attacks. `#​7056 <https://github.com/aio-libs/aiohttp/issues/7056>`\_ - Added support for setting response header parameters `max_line_size` and `max_field_size`. `#​2304 <https://github.com/aio-libs/aiohttp/issues/2304>`\_ - Added `auto_decompress` parameter to `ClientSession.request` to override `ClientSession._auto_decompress`. -- by :user:`Daste745` `#​3751 <https://github.com/aio-libs/aiohttp/issues/3751>`\_ - Changed `raise_for_status` to allow a coroutine. `#​3892 <https://github.com/aio-libs/aiohttp/issues/3892>`\_ - Added client brotli compression support (optional with runtime check). `#​5219 <https://github.com/aio-libs/aiohttp/issues/5219>`\_ - Added `client_max_size` to `BaseRequest.clone()` to allow overriding the request body size. -- :user:`anesabml`. `#​5704 <https://github.com/aio-libs/aiohttp/issues/5704>`\_ - Added a middleware type alias `aiohttp.typedefs.Middleware`. `#​5898 <https://github.com/aio-libs/aiohttp/issues/5898>`\_ - Exported `HTTPMove` which can be used to catch any redirection request that has a location -- :user:`dreamsorcerer`. `#​6594 <https://github.com/aio-libs/aiohttp/issues/6594>`\_ - Changed the `path` parameter in `web.run_app()` to accept a `pathlib.Path` object. `#​6839 <https://github.com/aio-libs/aiohttp/issues/6839>`\_ - Performance: Skipped filtering `CookieJar` when the jar is empty or all cookies have expired. `#​7819 <https://github.com/aio-libs/aiohttp/issues/7819>`\_ - Performance: Only check origin if insecure scheme and there are origins to treat as secure, in `CookieJar.filter_cookies()`. `#​7821 <https://github.com/aio-libs/aiohttp/issues/7821>`\_ - Performance: Used timestamp instead of `datetime` to achieve faster cookie expiration in `CookieJar`. `#​7824 <https://github.com/aio-libs/aiohttp/issues/7824>`\_ - Added support for passing a custom server name parameter to HTTPS connection. `#​7114 <https://github.com/aio-libs/aiohttp/issues/7114>`\_ - Added support for using Basic Auth credentials from :file:`.netrc` file when making HTTP requests with the :py:class:`~aiohttp.ClientSession` `trust_env` argument is set to `True`. -- by :user:`yuvipanda`. `#​7131 <https://github.com/aio-libs/aiohttp/issues/7131>`\_ - Turned access log into no-op when the logger is disabled. `#​7240 <https://github.com/aio-libs/aiohttp/issues/7240>`\_ - Added typing information to `RawResponseMessage`. -- by :user:`Gobot1234` `#​7365 <https://github.com/aio-libs/aiohttp/issues/7365>`\_ - Removed `async-timeout` for Python 3.11+ (replaced with `asyncio.timeout()` on newer releases). `#​7502 <https://github.com/aio-libs/aiohttp/issues/7502>`\_ - Added support for `brotlicffi` as an alternative to `brotli` (fixing Brotli support on PyPy). `#​7611 <https://github.com/aio-libs/aiohttp/issues/7611>`\_ - Added `WebSocketResponse.get_extra_info()` to access a protocol transport's extra info. `#​7078 <https://github.com/aio-libs/aiohttp/issues/7078>`\_ - Allow `link` argument to be set to None/empty in HTTP 451 exception. `#​7689 <https://github.com/aio-libs/aiohttp/issues/7689>`\_ ## Bugfixes - Implemented stripping the trailing dots from fully-qualified domain names in `Host` headers and TLS context when acting as an HTTP client. This allows the client to connect to URLs with FQDN host name like `https://example.com./`. \-- by :user:`martin-sucha`. `#​3636 <https://github.com/aio-libs/aiohttp/issues/3636>`\_ - Fixed client timeout not working when incoming data is always available without waiting. -- by :user:`Dreamsorcerer`. `#​5854 <https://github.com/aio-libs/aiohttp/issues/5854>`\_ - Fixed `readuntil` to work with a delimiter of more than one character. `#​6701 <https://github.com/aio-libs/aiohttp/issues/6701>`\_ - Added `__repr__` to `EmptyStreamReader` to avoid `AttributeError`. `#​6916 <https://github.com/aio-libs/aiohttp/issues/6916>`\_ - Fixed bug when using `TCPConnector` with `ttl_dns_cache=0`. `#​7014 <https://github.com/aio-libs/aiohttp/issues/7014>`\_ - Fixed response returned from expect handler being thrown away. -- by :user:`Dreamsorcerer` `#​7025 <https://github.com/aio-libs/aiohttp/issues/7025>`\_ - Avoided raising `UnicodeDecodeError` in multipart and in HTTP headers parsing. `#​7044 <https://github.com/aio-libs/aiohttp/issues/7044>`\_ - Changed `sock_read` timeout to start after writing has finished, avoiding read timeouts caused by an unfinished write. -- by :user:`dtrifiro` `#​7149 <https://github.com/aio-libs/aiohttp/issues/7149>`\_ - Fixed missing query in tracing method URLs when using `yarl` 1.9+. `#​7259 <https://github.com/aio-libs/aiohttp/issues/7259>`\_ - Changed max 32-bit timestamp to an aware datetime object, for consistency with the non-32-bit one, and to avoid a `DeprecationWarning` on Python 3.12. `#​7302 <https://github.com/aio-libs/aiohttp/issues/7302>`\_ - Fixed `EmptyStreamReader.iter_chunks()` never ending. -- by :user:`mind1m` `#​7616 <https://github.com/aio-libs/aiohttp/issues/7616>`\_ - Fixed a rare `RuntimeError: await wasn't used with future` exception. -- by :user:`stalkerg` `#​7785 <https://github.com/aio-libs/aiohttp/issues/7785>`\_ - Fixed issue with insufficient HTTP method and version validation. `#​7700 <https://github.com/aio-libs/aiohttp/issues/7700>`\_ - Added check to validate that absolute URIs have schemes. `#​7712 <https://github.com/aio-libs/aiohttp/issues/7712>`\_ - Fixed unhandled exception when Python HTTP parser encounters unpaired Unicode surrogates. `#​7715 <https://github.com/aio-libs/aiohttp/issues/7715>`\_ - Updated parser to disallow invalid characters in header field names and stop accepting LF as a request line separator. `#​7719 <https://github.com/aio-libs/aiohttp/issues/7719>`\_ - Fixed Python HTTP parser not treating 204/304/1xx as an empty body. `#​7755 <https://github.com/aio-libs/aiohttp/issues/7755>`\_ - Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3. `#​7756 <https://github.com/aio-libs/aiohttp/issues/7756>`\_ - Fixed an issue when a client request is closed before completing a chunked payload. -- by :user:`Dreamsorcerer` `#​7764 <https://github.com/aio-libs/aiohttp/issues/7764>`\_ - Edge Case Handling for ResponseParser for missing reason value. `#​7776 <https://github.com/aio-libs/aiohttp/issues/7776>`\_ - Fixed `ClientWebSocketResponse.close_code` being erroneously set to `None` when there are concurrent async tasks receiving data and closing the connection. `#​7306 <https://github.com/aio-libs/aiohttp/issues/7306>`\_ - Added HTTP method validation. `#​6533 <https://github.com/aio-libs/aiohttp/issues/6533>`\_ - Fixed arbitrary sequence types being allowed to inject values via version parameter. -- by :user:`Dreamsorcerer` `#​7835 <https://github.com/aio-libs/aiohttp/issues/7835>`\_ - Performance: Fixed increase in latency with small messages from websocket compression changes. `#​7797 <https://github.com/aio-libs/aiohttp/issues/7797>`\_ ## Improved Documentation - Fixed the `ClientResponse.release`'s type in the doc. Changed from `comethod` to `method`. `#​5836 <https://github.com/aio-libs/aiohttp/issues/5836>`\_ - Added information on behavior of base_url parameter in `ClientSession`. `#​6647 <https://github.com/aio-libs/aiohttp/issues/6647>`\_ - Fixed `ClientResponseError` docs. `#​6700 <https://github.com/aio-libs/aiohttp/issues/6700>`\_ - Updated Redis code examples to follow the latest API. `#​6907 <https://github.com/aio-libs/aiohttp/issues/6907>`\_ - Added a note about possibly needing to update headers when using `on_response_prepare`. -- by :user:`Dreamsorcerer` `#​7283 <https://github.com/aio-libs/aiohttp/issues/7283>`\_ - Completed `trust_env` parameter description to honor `wss_proxy`, `ws_proxy` or `no_proxy` env. `#​7325 <https://github.com/aio-libs/aiohttp/issues/7325>`\_ - Expanded SSL documentation with more examples (e.g. how to use certifi). -- by :user:`Dreamsorcerer` `#​7334 <https://github.com/aio-libs/aiohttp/issues/7334>`\_ - Fix, update, and improve client exceptions documentation. `#​7733 <https://github.com/aio-libs/aiohttp/issues/7733>`\_ ## Deprecations and Removals - Added `shutdown_timeout` parameter to `BaseRunner`, while deprecating `shutdown_timeout` parameter from `BaseSite`. -- by :user:`Dreamsorcerer` `#​7718 <https://github.com/aio-libs/aiohttp/issues/7718>`\_ - Dropped Python 3.6 support. `#​6378 <https://github.com/aio-libs/aiohttp/issues/6378>`\_ - Dropped Python 3.7 support. -- by :user:`Dreamsorcerer` `#​7336 <https://github.com/aio-libs/aiohttp/issues/7336>`\_ - Removed support for abandoned `tokio` event loop. -- by :user:`Dreamsorcerer` `#​7281 <https://github.com/aio-libs/aiohttp/issues/7281>`\_ ## Misc - Made `print` argument in `run_app()` optional. `#​3690 <https://github.com/aio-libs/aiohttp/issues/3690>`\_ - Improved performance of `ceil_timeout` in some cases. `#​6316 <https://github.com/aio-libs/aiohttp/issues/6316>`\_ - Changed importing Gunicorn to happen on-demand, decreasing import time by ~53%. -- :user:`Dreamsorcerer` `#​6591 <https://github.com/aio-libs/aiohttp/issues/6591>`\_ - Improved import time by replacing `http.server` with `http.HTTPStatus`. `#​6903 <https://github.com/aio-libs/aiohttp/issues/6903>`\_ - Fixed annotation of `ssl` parameter to disallow `True`. -- by :user:`Dreamsorcerer`. `#​7335 <https://github.com/aio-libs/aiohttp/issues/7335>`\_ *** </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/allenporter/pyrainbird). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy41OS44IiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
](https://renovatebot.com){kind=link}
This contains a few changes to rework the shutdown process to be a bit clearer and better match user expectations.
To summarise, I'm proposing the shutdown process should look like this now:
Docs demo preview: https://aiohttp--7718.org.readthedocs.build/en/7718/web_advanced.html#graceful-shutdown