Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix recorder crash for long state string - enforce at core level #9696

Merged
merged 8 commits into from
Oct 25, 2017

Conversation

milanvo
Copy link
Contributor

@milanvo milanvo commented Oct 4, 2017

Description:

Updated:
Recorder thread throws exception and stop recording for external DB engines (PostgreSQL, etc.), when state string is longer than 255 characters.

Fix long state of persistent notification: #9967

Breaking change: State value of entities is limited to 255 characters.

Related issue (if applicable): fixes #9366

Pull request in home-assistant.github.io with documentation (if applicable): home-assistant/home-assistant.github.io#<home-assistant.github.io PR number goes here>

Checklist:

If user exposed functionality or configuration variables are added/changed:

If the code does not interact with devices:

  • Local tests with tox run successfully. Your PR cannot be merged unless tests pass
  • Tests have been added to verify that the new code works.

@pvizeli
Copy link
Member

pvizeli commented Oct 5, 2017

A state should be not bigger as 255 characters. I don't know what should be the best solution. Maybe we trim it before we write it into state table... @balloob ?

@milanvo
Copy link
Contributor Author

milanvo commented Oct 5, 2017

Yes, 255 characters is very long for normal sensors etc. There are components, which allow long state text properly - for example Persistent notification, which I am using to simulate this problem. And probably others, like weather alerts, forecasts, etc.

@balloob
Copy link
Member

balloob commented Oct 5, 2017

I don't want to increase the column length over 255 characters. Instead we should update persistent notification to store the message in the attributes.

@balloob
Copy link
Member

balloob commented Oct 5, 2017

I don't like this fix either. Silently failing is bad, we should probably enforce this at the core level.

@milanvo
Copy link
Contributor Author

milanvo commented Oct 5, 2017

Yes, this first commit is not intended as permanent and systematic fix and is not a good way to solve it. Just try to avoid first recorder thread stop by workaround and open a discussion. Exception is not completely silenced - errors are still logged.

It is question, how many components are cause of long states - some are mentioned in linked issues

@rbflurry
Copy link
Contributor

rbflurry commented Oct 6, 2017

I had also reported this. #9092

@milanvo milanvo changed the title Fix recorder crash for long state string - possible solutions ? RFC: Fix recorder crash for long state string Oct 8, 2017
@milanvo
Copy link
Contributor Author

milanvo commented Oct 11, 2017

Hi @pvizeli

discovering core module, please confirm this is right place (State object) to check and enforce state length
https://github.com/home-assistant/home-assistant/blob/0de2266a722712c407991e421e205952e4536008/homeassistant/core.py#L527

If longer than 255, we should truncate it to 255 and log warning (or error ?) or throw exception ?

Thanks

@balloob
Copy link
Member

balloob commented Oct 16, 2017

We should throw an exception in the State constructor if a state is too long. However, we should first update the persistent_notification component to no longer set long states.

@milanvo milanvo requested a review from a team as a code owner October 17, 2017 15:22
@milanvo
Copy link
Contributor Author

milanvo commented Oct 17, 2017

Sorry for mess in commits - I was confused by merges and reverts 😉

@balloob please advise regarding persistent_notification - should it be redesigned to store message in attributes (will require GUI changes also ?) ? What should be stored in persistent_notification state then ?

Thank you

@milanvo milanvo changed the title RFC: Fix recorder crash for long state string Fix recorder crash for long state string - enforce at core level Oct 19, 2017
@milanvo
Copy link
Contributor Author

milanvo commented Oct 19, 2017

Hi,

make some progress regarding fixes:

  • implemented core changes
  • created PR for Persistent notification (+ GUI change)

@balloob
Copy link
Member

balloob commented Oct 20, 2017

Sorry for the delay in getting back to you. Other PR is great approach. Will close this PR as the other one is the proper fix.

@balloob balloob closed this Oct 20, 2017
@balloob balloob reopened this Oct 20, 2017
@balloob
Copy link
Member

balloob commented Oct 20, 2017

This fix is ok

@balloob
Copy link
Member

balloob commented Oct 20, 2017

Can be merged after #9967 has been merged.

@milanvo
Copy link
Contributor Author

milanvo commented Oct 21, 2017

Thanks for review.

Because this can break components which set "invalid" state, it could be helpful to warn users / developers in release notes to report / fix these components.

@balloob
Copy link
Member

balloob commented Oct 25, 2017

That's what the breaking change tag is for 👍

Thanks for the PR! 🐬

@balloob balloob merged commit 7987065 into home-assistant:dev Oct 25, 2017
@milanvo milanvo deleted the fix-recorder-longstate branch October 25, 2017 16:54
@ahknight
Copy link

This change will break getting weather alerts from WUnderground and a list of similar things documented in #9366. For the record, I think this is the wrong direction and the column should be made larger instead.

@arsaboo
Copy link
Contributor

arsaboo commented Oct 25, 2017

@ahknight It will truncate values larger than 255 chars, so it should not break anything. Let us know if you have tried out this fix. 255 chars is already long.

@milanvo
Copy link
Contributor Author

milanvo commented Oct 25, 2017

@arsaboo @ahknight
Conclusion of this discussion with owner is to throw an exception for long (invalid) state. Long text should be stored in entity attributes - not state. Persistent notification component is already changed and PR #9967 merged accorditg to this.

@balloob
Copy link
Member

balloob commented Oct 25, 2017

The weather component does not write alerts as state. If this is about a weather related platform for a different component, they should all be moved to weather component or move value to attribute

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

events.event_data and states.state columns are too small on MySQL/Postgresql
9 participants