Remove usage of "run_until_complete" #16617

smurfix · 2018-09-14T13:04:29Z

Description:

This patch changes Home Assistant's startup routine so that it will run under asyncio.run. It also changes the SIGINT handler to start the shutdown sequence.

Advantages:

This allows people to embed Home Assistant in larger async programs.
SIGINT aka ^C no longer disrupts a random part of Home Assistant's code.

Disadvantages:

If Home Assistant's main thread is wedged (which basically should not happen), you may have to press ^C twice to get out.

Checklist:

The code change is tested and works locally.
Local tests pass with tox. Your PR cannot be merged unless tests pass

houndci-bot · 2018-09-14T13:04:42Z

homeassistant/helpers/signal.py

@@ -17,7 +17,9 @@ def async_register_signal_handling(hass: HomeAssistant) -> None:
    if sys.platform != 'win32':
        @callback
        def async_signal_handle(exit_code):
-            """Wrap signal handling."""
+            """Wrap signal handling: queue shutdown code, re-instate default handler."""


line too long (88 > 79 characters)

houndci-bot · 2018-09-14T13:04:42Z

homeassistant/core.py

+
+        await self.async_start()
+        if attach_signals:
+            from homeassistant.helpers.signal import async_register_signal_handling


line too long (83 > 79 characters)

* Use asyncio.run (or our own implementation on Python <3.7) * hass.start is only used by tests * setup_and_run_hass() is now async * Add "main" async hass.run method * move SIGINT handling to helpers/signal.py * add flag to .run to disable hass's signal handlers * Teach async_start and async_stop to not step on each other (more than necessary)

balloob · 2018-09-14T14:48:19Z

homeassistant/__main__.py

@@ -21,8 +21,6 @@

 def attempt_use_uvloop() -> None:
    """Attempt to use uvloop."""
-    import asyncio


Now the import is missing?

Ugh. Me being blind. Will fix.

balloob · 2018-09-14T14:50:20Z

homeassistant/core.py

+        if self.state == CoreState.not_running:  # just ignore
+            return
+        if self.state == CoreState.stopping:
+            _LOGGER.info("async_stop called twice: ignored")


Should this be a warning?

Might be a better idea. Will change.

balloob · 2018-09-14T14:51:23Z

homeassistant/util/__init__.py

@@ -30,6 +30,15 @@
 }


+try:


Can we move this to util.async_

balloob · 2018-09-14T14:52:06Z

homeassistant/helpers/signal.py

+            * queue shutdown code
+            * re-instate default handler
+            """
+            hass.loop.remove_signal_handler(signal.SIGTERM)


Why would we do this ?

Because the signal handler has initiated an orderly shutdown. Its job is now done; if we get the signal again, the user is impatient / the main loop is stuck, and standard Python signal handling should take over.

houndci-bot · 2018-09-15T09:07:47Z

homeassistant/util/async_.py

+    asyncio_run = asyncio.run
+except AttributeError:
+    def asyncio_run(main, *, debug=False):
+        loop = asyncio.new_event_loop()


undefined name 'asyncio'

houndci-bot · 2018-09-15T09:07:47Z

homeassistant/util/async_.py

@@ -12,6 +12,15 @@
 _LOGGER = logging.getLogger(__name__)


+try:
+    asyncio_run = asyncio.run


undefined name 'asyncio'

houndci-bot · 2018-09-15T13:14:44Z

homeassistant/core.py

        fire_coroutine_threadsafe(self.async_stop(), self.loop)

-    async def async_stop(self, exit_code: int = 0) -> None:
+    async def async_stop(self, exit_code: int = 0, *, force: bool = False) -> None:


line too long (83 > 79 characters)

only useful for testing

Required for keeping requirements_test_all.txt in sync, where it is in turn required to prevent auto-downgrading "attrs" during "pip install"

houndci-bot · 2018-09-15T14:06:30Z

homeassistant/util/async_.py

+    asyncio_run = asyncio.run  # type: ignore
+except AttributeError:
+    _T = TypeVar('_T')
+    def asyncio_run(main: Awaitable[_T], *, debug: bool = False) -> _T:


expected 1 blank line, found 0

smurfix · 2018-09-15T14:09:08Z

Current tests break with undefined "mock_signal" fixture.
I have no idea where that is supposed to come from …

smurfix · 2018-09-15T17:30:35Z

Meh. Finally. (Had a broken local installation.)
Ready to merge IMHO.

balloob · 2018-09-17T08:07:28Z

homeassistant/core.py

        finally:
            self.loop.close()
        return self.exit_code

+    async def run(self, *, attach_signals: bool = True) -> int:


If this is an async method, we should call it async_run to be in line with the rest of our functions.

My rationale for not naming it async_run is that it's never called from within Home Assistant, but I'll change it if you want me to, no problem.

Yeah let's change it.

balloob · 2018-09-17T08:08:06Z

requirements_test.txt

@@ -2,6 +2,7 @@
 # make new things fail. Manually update these pins when pulling in a
 # new version
 asynctest==0.12.2
+attrs==18.2.0


This shouldn't be necessary, it's already part of the core dependencies.

The problem: when I do a pip install -r requirements_test_all.txt without this entry, attrs is downgraded to some lower version, probably because of some sub-module's requirements.

Use the constraints file like Tox does: pip install -r requirements_test_all.txt -c homeassistant/package_constraints.txt

solved by using "pip -c"

balloob

ok to merge after run renamed to async_run

smurfix · 2018-09-17T10:01:12Z

All done, please merge.

awarecan · 2018-09-20T04:33:36Z

Did it tested on Windows?

I run on Windows Python 3.6.3 got some errors

Traceback (most recent call last):
  File "c:\awarecan\home-assistant\homeassistant\__main__.py", line 376, in <module>
    sys.exit(main())
  File "c:\awarecan\home-assistant\homeassistant\__main__.py", line 368, in main
    exit_code = asyncio_run(setup_and_run_hass(config_dir, args))
  File "c:\awarecan\home-assistant\homeassistant\util\async_.py", line 29, in asyncio_run
    return loop.run_until_complete(main)
  File "C:\Python\Python36\lib\asyncio\base_events.py", line 467, in run_until_complete
    return future.result()
  File "c:\awarecan\home-assistant\homeassistant\__main__.py", line 275, in setup_and_run_hass
    log_no_color=args.log_no_color)
  File "c:\awarecan\home-assistant\homeassistant\bootstrap.py", line 215, in async_from_config_file
    conf_util.load_yaml_config_file, config_path)
RuntimeError: Task <Task pending coro=<setup_and_run_hass() running at c:\awarecan\home-assistant\homeassistant\__main__.py:275> cb=[_run_until_complete_cb() at C:\Python\Python36\lib\asyncio\base_events.py:176]> got Future <Future pending cb=[_chain_future.<locals>._call_check_cancel() at C:\Python\Python36\lib\asyncio\futures.py:408]> attached to a different loop

balloob · 2018-09-20T07:09:34Z

This logic should be moved to main to fix it:

https://github.com/home-assistant/home-assistant/blob/fcb84d951e8022ace87bb19be259f401c9920ddc/homeassistant/core.py#L132-L135

balloob · 2018-09-20T07:14:35Z

This should do it: #16737

smurfix requested a review from a team as a code owner September 14, 2018 13:04

homeassistant added core cla-signed labels Sep 14, 2018

ghost added the in progress label Sep 14, 2018

houndci-bot reviewed Sep 14, 2018

View reviewed changes

smurfix force-pushed the aio_run branch from 3db7588 to 5023ed1 Compare September 14, 2018 13:05

homeassistant added the cla-signed label Sep 14, 2018

shorten over-long lines

79161c6

balloob reviewed Sep 14, 2018

View reviewed changes

homeassistant/util/__init__.py Outdated

@@ -30,6 +30,15 @@

}

try:

Copy link

Member

balloob Sep 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move this to util.async_

balloob reviewed Sep 14, 2018

View reviewed changes

houndci-bot reviewed Sep 15, 2018

View reviewed changes

smurfix added 2 commits September 15, 2018 11:13

restore missing "import asyncio"

0f0d5c2

move run_asyncio to homeassistant.util.async_

84ab2d9

smurfix force-pushed the aio_run branch from 8a8488d to 84ab2d9 Compare September 15, 2018 09:13

homeassistant added the cla-signed label Sep 15, 2018

LOGGER: warn => warning

1377fbd

houndci-bot reviewed Sep 15, 2018

View reviewed changes

Add "force" flag to async_stop

6b56341

only useful for testing

smurfix force-pushed the aio_run branch from 75be435 to 6b56341 Compare September 15, 2018 13:18

homeassistant added the cla-signed label Sep 15, 2018

Add 'attrs==18.2.0' to requirements_all.txt

4caa189

Required for keeping requirements_test_all.txt in sync, where it is in turn required to prevent auto-downgrading "attrs" during "pip install"

houndci-bot reviewed Sep 15, 2018

View reviewed changes

Fixes for mypy

80007b0

smurfix force-pushed the aio_run branch from 2031786 to 80007b0 Compare September 15, 2018 14:23

homeassistant added the cla-signed label Sep 15, 2018

Fix "mock_signal" fixture

edce113

smurfix added 3 commits September 15, 2018 17:40

Revert mistaken edit

c33c2f7

Flake8 fixes

efb8679

mypy fixes

6eb1cf9

smurfix force-pushed the aio_run branch from 049599a to 666e614 Compare September 15, 2018 15:40

homeassistant added the cla-signed label Sep 15, 2018

smurfix force-pushed the aio_run branch from 666e614 to ddb1021 Compare September 15, 2018 15:57

homeassistant added the cla-signed label Sep 15, 2018

smurfix force-pushed the aio_run branch from ddb1021 to 53f9256 Compare September 15, 2018 16:27

homeassistant added the cla-signed label Sep 15, 2018

pylint fix

595c4c7

smurfix force-pushed the aio_run branch from 53f9256 to 595c4c7 Compare September 15, 2018 17:08

homeassistant added the cla-signed label Sep 15, 2018

balloob reviewed Sep 17, 2018

View reviewed changes

Revert adding attrs== to requirements_test*.txt

f0d8a58

solved by using "pip -c"

balloob approved these changes Sep 17, 2018

View reviewed changes

Rename "run" to "async_run", as per calling conventions

019a0ee

smurfix force-pushed the aio_run branch from 3f572ce to 019a0ee Compare September 17, 2018 09:30

homeassistant added the cla-signed label Sep 17, 2018

balloob merged commit 0121e3c into home-assistant:dev Sep 19, 2018

ghost removed the in progress label Sep 19, 2018

balloob mentioned this pull request Sep 20, 2018

Fix Windows loop #16737

Merged

balloob mentioned this pull request Sep 28, 2018

0.79.0 #16940

Merged

fabaff mentioned this pull request Dec 10, 2018

demo-mode not working #19110

Closed

home-assistant locked and limited conversation to collaborators Feb 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove usage of "run_until_complete" #16617

Remove usage of "run_until_complete" #16617

smurfix commented Sep 14, 2018 •

edited

houndci-bot Sep 14, 2018

houndci-bot Sep 14, 2018

balloob Sep 14, 2018

smurfix Sep 15, 2018

balloob Sep 14, 2018

smurfix Sep 15, 2018

balloob Sep 14, 2018

balloob Sep 14, 2018

smurfix Sep 15, 2018

balloob Sep 15, 2018

houndci-bot Sep 15, 2018

houndci-bot Sep 15, 2018

houndci-bot Sep 15, 2018

houndci-bot Sep 15, 2018

smurfix commented Sep 15, 2018

smurfix commented Sep 15, 2018

balloob Sep 17, 2018

smurfix Sep 17, 2018

balloob Sep 17, 2018

balloob Sep 17, 2018

smurfix Sep 17, 2018

balloob Sep 17, 2018

balloob left a comment

smurfix commented Sep 17, 2018

awarecan commented Sep 20, 2018

balloob commented Sep 20, 2018

balloob commented Sep 20, 2018

Remove usage of "run_until_complete" #16617

Remove usage of "run_until_complete" #16617

Conversation

smurfix commented Sep 14, 2018 • edited

Description:

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smurfix commented Sep 15, 2018

smurfix commented Sep 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balloob left a comment

Choose a reason for hiding this comment

smurfix commented Sep 17, 2018

awarecan commented Sep 20, 2018

balloob commented Sep 20, 2018

balloob commented Sep 20, 2018

smurfix commented Sep 14, 2018 •

edited