New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use built in queue log handlers to avoid formatting logs in the event loop #35633
Use built in queue log handlers to avoid formatting logs in the event loop #35633
Conversation
need to experiment with where the filtering should go. Not sure if it should stay on incoming side of the queue or the processing side. The filtering is a bit more expensive than I expected, esp. with many logger config lines as it has to iterate all of them and do a string match |
I'm thinking the log filtering should come off the individual handlers and go on to the queue handler only so only the messages we care about go into the queue. Should be a bit more efficient as we only have to run the filter once and the other handlers on the other side of the queue don't have to worry about filtering |
It appears I'm solving the wrong problem here since this shouldn't be an issue since logging is already wrapped in It may be holding it wrong but it looks like none of the if So what I think its really happening is we are serializing the all the websocket data in the More investigation needed for sure |
df3a75b
to
6094a07
Compare
45e85a7
to
fdcfbbf
Compare
… loop Logging is now formatted and written in another thread to ensure there is minimal impact on the event loop when a log message is processed. This change replaces the existing AsyncHandler log handler as python 3.7+ now offers an off the shelf solution
fdcfbbf
to
20663a7
Compare
All the new logging code in |
eac6aed
to
57be92d
Compare
Managed to get it to |
Retested to make sure the name change didn't cause issues (done programmatically) but better to be safe. All good |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
Proposed change
I noticed that cameras were taking a long time to load images in the UI and realized I had
pyhap
in debug mode. After investigation it turned out that the event loop was being tied up by logging (mostly formatting).Proposed solution is to use a logging QueueHandler and QueueListener to avoid formatting log messages the event loop
This also resolves a bug where the first few messages (~39 for me) includes
2020-05-15 22:06:44 INFO (MainThread) [homeassistant.bootstrap] Config directory: /config
would be missing fromhome-assistant.log
Before (websocket_api debug on, full debug overwhelmed things):
After (full debug on):
Logging has negligible impact on the event loop. I can even turn on full debug mode with significantly less impact on the UI performance as all of the work is no longer happening in the event loop as its moved to another thread.
Type of change
Example entry for
configuration.yaml
:# Example configuration.yaml
Additional information
Checklist
black --fast homeassistant tests
)If user exposed functionality or configuration variables are added/changed:
If the code communicates with devices, web services, or third-party tools:
Updated and included derived files by running:
python3 -m script.hassfest
.requirements_all.txt
.Updated by running
python3 -m script.gen_requirements_all
..coveragerc
.The integration reached or maintains the following Integration Quality Scale: