Recorder breaks on: UnicodeEncodeError: 'charmap' codec can't encode character '\u0151' #76436

Mariusthvdb · 2022-08-08T08:21:46Z

The problem

system halted/restarted upon below error

This is an error on recorder choking on certain Unicode characters, which are created by various integrations so it seems.
very serious, as it completely breaks the system, which gets in an endless loop of overheating/memory and processor overload and consequent restarts.

What version of Home Assistant Core has the issue?

2022.8.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Recorder

Link to integration documentation on our website

https://www.home-assistant.io/integrations/recorder/

Diagnostics information

No response

Example YAML snippet

endless repeated logging of:

, count=1, first_occurred=1659944277.1001759>): 'charmap' codec can't encode character '\u0151' in position 133173812: character maps to <undefined>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 632, in _run_event_loop
    self._process_one_task_or_recover(task)
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 645, in _process_one_task_or_recover
    return task.run(self)
  File "/usr/src/homeassistant/homeassistant/components/recorder/tasks.py", line 211, in run
    instance._process_one_event(self.event)
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 743, in _process_one_event
    self._process_non_state_changed_event_into_session(event)
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 808, in _process_non_state_changed_event_into_session
    if data_id := self._find_shared_data_in_db(data_hash, shared_data):
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 778, in _find_shared_data_in_db
    if data_id := self.event_session.execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1712, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/lambdas.py", line 516, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2047, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 199, in execute
    args = tuple(map(db.literal, args))
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/connections.py", line 275, in literal
    s = self.string_literal(o.encode(self.encoding))
  File "/usr/local/lib/python3.10/encodings/cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character '\u0151' in position 133173812: character maps to <undefined>

home-assistant.log.1 shows never seen lines like:

"/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 778, in _find_shared_data_in_db\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n    if data_id := self.event_session.execute(\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1712, in execute\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n    result = conn._execute_20(statement, params or {}, execution_options)\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n    return meth(self, args_10style, kwargs_10style, execution_options)\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/lambdas.py", line 516, in _execute_on_connection\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n    return connection._execute_clauseelement(\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n    ret = self._execute_context(\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n

Additional information

pardon the edit, but Ive taken out the irrelevent first impressions, as they turned out to be coincidental.

The text was updated successfully, but these errors were encountered:

probot-home-assistant · 2022-08-08T08:21:51Z

recorder documentation
recorder source
_{^{(message by IssueLinks)}}

probot-home-assistant · 2022-08-08T08:21:54Z

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

Mariusthvdb · 2022-08-08T08:30:27Z

the display of the text file above that broke the system might have been a coincidence.
there's a clue this is related to Musicassistant:

2022-08-08 09:36:08.120 ERROR (Recorder) [homeassistant.components.recorder.core] Error while processing event EventTask(event=<Event mass_event[L]: type=media_item_updated, object_id=database://album/17, data=item_id=17, provider=database, name=Vivaldi: Argippo, provider_ids=[{'item_id': '13WU1MoeIkk5FTKXhpNcH5', 'prov_type': 'spotify', 'prov_id': 'spotify_ainhdb', 'available': True, 'quality': 2, 'details': None, 'url': 'https://open.spotify.com/album/13WU1MoeIkk5FTKXhpNcH5'}], metadata=description=None, review=None, explicit=None, images=[{'type': 'thumb', 'url': 'https://i.scdn.co/image/ab67616d0000b273edb86a009276b6d94ab570b1', 'is_file': False}], genres=None, mood=None, style=None, copyright=2019 Naïve, a Label of Believe Group, lyrics=None, ean=None, label=Naïve, a Label of Believe Group, links=None, performers=None, preview=None, replaygain=None, popularity=None, last_refresh=None, checksum=None, in_library=True, media_type=album, sort_name=vivaldi: argippo, uri=database://album/17, timestamp=1659944168, version=, year=2020, artists=[{'item_id': '52', 'provider': 'database', 'name': 'Europa Galante', 'provider_ids': [], 'metadata': {'description': None, 'review': None, 'explicit': None, 'images': None, 'genres': None, 'mood': None, 'style': None, 'copyright': None, 'lyrics': None, 'ean': None, 'label': None, 'links': None, 'performers': None, 'preview': None, 'replaygain': None, 'popularity': None, 'last_refresh': None, 'checksum': None}, 'in_library': False, 'media_type': 'artist', 'sort_name': 'europa galante', 'uri': 'database://artist/52', 'timestamp': 0, 'musicbrainz_id': None}, {'item_id': '51', 'provider': 'database', 'name': 'Antonio Vivaldi', 'provider_ids': [], 'metadata': {'description': None, 'review': None, 'explicit': None, 'images': None, 'genres': None, 'mood': None, 'style': None, 'copyright': None, 'lyrics': None, 'ean': None, 'label': None, 'links': None, 'performers': None, 'preview': None, 'replaygain': None, 'popularity': None, 'last_refresh': None, 'checksum': None}, 'in_library': False, 'media_type': 'artist', 'sort_name': 'antonio vivaldi', 'uri': 'database://artist/51', 'timestamp': 0, 'musicbrainz_id': None}, {'item_id': '53', 'provider': 'database', 'name': 'Fabio Biondi', 'provider_ids': [], 'metadata': {'description': None, 'review': None, 'explicit': None, 'images': None, 'genres': None, 'mood': None, 'style': None, 'copyright': None, 'lyrics': None, 'ean': None, 'label': None, 'links': None, 'performers': None, 'preview': None, 'replaygain': None, 'popularity': None, 'last_refresh': None, 'checksum': None}, 'in_library': False, 'media_type': 'artist', 'sort_name': 'fabio biondi', 'uri': 'database://artist/53', 'timestamp': 0, 'musicbrainz_id': None}, {'item_id': '54', 'provider': 'database', 'name': 'Emőke Baráth', 'provider_ids': [], 'metadata': {'description': None, 'review': None, 'explicit': None, 'images': None, 'genres': None, 'mood': None, 'style': None, 'copyright': None, 'lyrics': None, 'ean': None, 'label': None, 'links': None, 'performers': None, 'preview': None, 'replaygain': None, 'popularity': None, 'last_refresh': None, 'checksum': None}, 'in_library': False, 'media_type': 'artist', 'sort_name': 'emőke baráth', 'uri': 'database://artist/54', 'timestamp': 0, 'musicbrainz_id': None}], album_type=album, upc=3616408945965, musicbrainz_id=None>): 'charmap' codec can't encode character '\u0151' in position 2512: character maps to <undefined>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 632, in _run_event_loop
    self._process_one_task_or_recover(task)
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 645, in _process_one_task_or_recover
    return task.run(self)
  File "/usr/src/homeassistant/homeassistant/components/recorder/tasks.py", line 211, in run
    instance._process_one_event(self.event)
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 743, in _process_one_event
    self._process_non_state_changed_event_into_session(event)
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 808, in _process_non_state_changed_event_into_session
    if data_id := self._find_shared_data_in_db(data_hash, shared_data):
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 778, in _find_shared_data_in_db
    if data_id := self.event_session.execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1712, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/lambdas.py", line 516, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2047, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 199, in execute
    args = tuple(map(db.literal, args))
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/connections.py", line 275, in literal
    s = self.string_literal(o.encode(self.encoding))
  File "/usr/local/lib/python3.10/encodings/cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character '\u0151' in position 2512: character maps to <undefined>

and I will post there too, but even so, Home Assistant shouldn't break/run into trouble because of a single character.

Mariusthvdb · 2022-08-14T18:43:07Z

currently 2022.8.4

new experience: total breakdown, processor up to tops, temp mini pc nearing 80C..

2022-08-14 17:56:46.050 ERROR (Recorder) [homeassistant.components.recorder.core] Error while processing event EventTask(event=<Event state_changed[L]: entity_id=sensor.co2_living, old_state=<state sensor.co2_living=801.0; state_class=measurement, unit_of_measurement=ppm, device_class=carbon_dioxide, friendly_name=Mco Multilevel: Carbon dioxide (CO₂) level Living

device in DB https://devices.zwave-js.io/?jumpTo=0x015f:0x0901:0x3102:0.0

all because of a ZwaveJS device inclusion of an MCO Multilevel sensor for humid/temp/Co2

2022-08-14 18:04:13.215 ERROR (Recorder) [homeassistant.components.recorder.core] Error while processing event EventTask(event=<Event state_changed[L]: entity_id=sensor.co2_living, old_state=None, new_state=<state sensor.co2_living=801.0; state_class=measurement, unit_of_measurement=ppm, device_class=carbon_dioxide, friendly_name=Mco Multilevel: Carbon dioxide (CO₂) level Living, hide_attributes=['templates'], templates=icon_color=if (state > 2100) return 'purple'; if (state > 1600) return 'red'; if (state > 1100) return 'orange'; if (state > 900) return 'brown'; return 'green';
 @ 2022-08-14T18:04:13.107912+02:00>>): 'charmap' codec can't encode character '\u2082' in position 140: character maps to <undefined>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 632, in _run_event_loop
    self._process_one_task_or_recover(task)
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 645, in _process_one_task_or_recover
    return task.run(self)
  File "/usr/src/homeassistant/homeassistant/components/recorder/tasks.py", line 211, in run
    instance._process_one_event(self.event)
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 741, in _process_one_event
    self._process_state_changed_event_into_session(event)
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 847, in _process_state_changed_event_into_session
    if attributes_id := self._find_shared_attr_in_db(attr_hash, shared_attrs):
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 760, in _find_shared_attr_in_db
    if attributes_id := self.event_session.execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1712, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/lambdas.py", line 516, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2047, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 199, in execute
    args = tuple(map(db.literal, args))
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/connections.py", line 275, in literal
    s = self.string_literal(o.encode(self.encoding))
  File "/usr/local/lib/python3.10/encodings/cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character '\u2082' in position 140: character maps to <undefined>

log running into 740Mb...

continuous restarts. total catastrophe.

please have a look? this can not be true Recorder choking on core inclusion of devices

bschatzow · 2022-08-14T21:13:04Z

I don't know if it is a coincidence but I had lots of errors with Mariadb from the August beta and beyond. I switched to the built in sql for now and no issues since. If you are using Mariadb, maybe try temporarily using the built in one and see if they stop.

Mariusthvdb · 2022-08-14T21:27:41Z

Yes, I noticed that.
and, tbh, I am not going to use those devices again, until after recognition and a fix by the dev team. This is such a risky business, I wont submit my HA instance to that anymore....

bdraco · 2022-08-15T08:14:22Z

Did you save the giant log? It would be helpful to get at least a few megs of it to see why its stuck

Mariusthvdb · 2022-08-15T09:33:22Z

heck, no.... sorry.. it was mostly filled with:

///////////////////////////////////////////////////////////////////

in endless lines. Doesnt the above logging give a clue? Seems quite explicit about the unexcepted char mapping ?

bdraco · 2022-08-15T15:43:49Z

I am interested in why it didn't move on and kept repeating.

Mariusthvdb · 2022-08-15T15:55:14Z

Sure. I understand.

Yet I don't dare to repeat the inclusion... it was a horrifying experience I could get out of by mere chance...
You're saying it should not choke on these charmaps?

bdraco · 2022-08-15T15:59:54Z

You're saying it should not choke on these charmaps?

It should have logged once, and moved on.

Where can I buy one of the problem devices?

Mariusthvdb · 2022-08-15T21:58:47Z

they are quite expensive devices..... mine is an older version (been measuring nicely for over 5 years now). It's back on the former Hub for now, although Id love to integrate it into HA zwave again.

https://www.robbshop.nl/merken/mco-home

bdraco · 2022-08-15T22:10:44Z

I'm afraid I won't be able to help with this one since the device uses EU zwave frequencies so I wouldn't be able to get a good test setup

LooSik · 2022-08-20T17:27:55Z

@Mariusthvdb

I had simillar issues when recently updating, my MySQL recorder configuration was missing ?charset=utf8mb4 ( not sure when this was added as neccessity ) but adding that solved the issue 'charmap' codec can't encode character '\uXXXX' in position errors happening in the log.

So I would certainly check whatever your MariaDB configuration has the extra charset param in there. Recorder docs do have it for mariadb so might be it?

Mariusthvdb · 2022-08-21T08:59:51Z

O that might well be yes. Great suggestion. Thx.

I don't use any of that suffixes in the auth string.

Nick, would you also believe that to be required? Must confess I haven't touched that in years, so if a change was introduced, I certainly missed that.

It would also be good to have some explanation as to the why in that case, because for me that auth string is just a copy and paste. And not anything I truly understand ....

using:

db_url: mysql://<uid>:<pw>@core-mariadb/homeassistant #?charset=utf8

but no idea why this is omitting pymysql and if we should do that

mmichon · 2022-08-22T17:22:08Z

I had simillar issues when recently updating, my MySQL recorder configuration was missing ?charset=utf8mb4 ( not sure when this was added as neccessity ) but adding that solved the issue 'charmap' codec can't encode character '\uXXXX' in position errors happening in the log.

This solved it for me: ?charset=utf8mb4 was missing in my recorder->db_url in configuration.yaml. Change that, restart Hass, errors disappear.

bdraco · 2022-08-22T18:00:33Z

You definitely need a four byte character set in order to store emoji characters

Mariusthvdb · 2022-08-28T19:05:48Z

So I added the correct string, and don't see the specific error, but see a daily spike

After which a restart is automatically started.

This seems to coincide

What could be causing this...

Earlier occurrences:

bdraco · 2022-08-28T19:32:41Z

If you had the wrong String when your tables were created they are in the wrong charset

Mariusthvdb · 2022-08-28T20:31:53Z

Really sorry but I don't have a
clue what is wrong now.

The adapted db url string, the charmap code, or the recorded db. Or even
Something else. Would be nice if some diagnostic logging was done here.

Is there a way to be sure and start afresh?
Or at least diagnose the cause of this issue.

Feels a bit like guessing right now.

bdraco · 2022-08-28T20:33:42Z

https://mathiasbynens.be/notes/mysql-utf8mb4

bdraco · 2022-08-28T20:34:07Z

Or if you want the short version

https://dba.stackexchange.com/questions/8239/how-to-easily-convert-utf8-tables-to-utf8mb4-in-mysql-5-5

Mariusthvdb · 2022-08-29T04:42:08Z

So because I used the incorrect charmap before, and now added the correct, I need to convert the existing db to use that too?

And because I have not done that yet , there is no documentation whatsoever in HA that instructs us, recorder breaks daily now, and HA with that??

Seems a horrible hazard for a core addon or recorder itself ofc to do so, without further logging or guard..

This happens when not restarting manually now (can't do so obviously):

And that can hardly be called Streamlining...

The system does not auto restart now so it seems, so we're locked out!

Tbh, I am away from home now, and I have no idea how to proceed, and get control over my home back...

Mariusthvdb · 2022-08-30T14:50:47Z

Btw is this related?

#77127

Mariusthvdb · 2022-09-04T11:53:50Z

Closing this as the recorder no longer throws an issue on the charmap.

will open a new issue though on memory usage. The spikes didn’t go away when away from home, and, while back at home, are replaced by continuous increase of memory.
Must be another issue.

probot-home-assistant bot added the integration: recorder label Aug 8, 2022

Mariusthvdb mentioned this issue Aug 8, 2022

MA breaks recorder : 'charmap' codec can't encode character '\u0151' music-assistant/hass-music-assistant#825

Closed

Mariusthvdb closed this as completed Sep 4, 2022

github-actions bot locked and limited conversation to collaborators Oct 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recorder breaks on: UnicodeEncodeError: 'charmap' codec can't encode character '\u0151' #76436

Recorder breaks on: UnicodeEncodeError: 'charmap' codec can't encode character '\u0151' #76436

Mariusthvdb commented Aug 8, 2022 •

edited

probot-home-assistant bot commented Aug 8, 2022

probot-home-assistant bot commented Aug 8, 2022

Mariusthvdb commented Aug 8, 2022 •

edited

Mariusthvdb commented Aug 14, 2022 •

edited

bschatzow commented Aug 14, 2022

Mariusthvdb commented Aug 14, 2022

bdraco commented Aug 15, 2022

Mariusthvdb commented Aug 15, 2022 •

edited

bdraco commented Aug 15, 2022

Mariusthvdb commented Aug 15, 2022

bdraco commented Aug 15, 2022

Mariusthvdb commented Aug 15, 2022

bdraco commented Aug 15, 2022

LooSik commented Aug 20, 2022

Mariusthvdb commented Aug 21, 2022 •

edited

mmichon commented Aug 22, 2022

bdraco commented Aug 22, 2022

Mariusthvdb commented Aug 28, 2022 •

edited

bdraco commented Aug 28, 2022

Mariusthvdb commented Aug 28, 2022

bdraco commented Aug 28, 2022

bdraco commented Aug 28, 2022

Mariusthvdb commented Aug 29, 2022 •

edited

Mariusthvdb commented Aug 30, 2022

Mariusthvdb commented Sep 4, 2022 •

edited

Recorder breaks on: UnicodeEncodeError: 'charmap' codec can't encode character '\u0151' #76436

Recorder breaks on: UnicodeEncodeError: 'charmap' codec can't encode character '\u0151' #76436

Comments

Mariusthvdb commented Aug 8, 2022 • edited

The problem

What version of Home Assistant Core has the issue?

What was the last working version of Home Assistant Core?

What type of installation are you running?

Integration causing the issue

Link to integration documentation on our website

Diagnostics information

Example YAML snippet

Additional information

probot-home-assistant bot commented Aug 8, 2022

probot-home-assistant bot commented Aug 8, 2022

Mariusthvdb commented Aug 8, 2022 • edited

Mariusthvdb commented Aug 14, 2022 • edited

bschatzow commented Aug 14, 2022

Mariusthvdb commented Aug 14, 2022

bdraco commented Aug 15, 2022

Mariusthvdb commented Aug 15, 2022 • edited

bdraco commented Aug 15, 2022

Mariusthvdb commented Aug 15, 2022

bdraco commented Aug 15, 2022

Mariusthvdb commented Aug 15, 2022

bdraco commented Aug 15, 2022

LooSik commented Aug 20, 2022

Mariusthvdb commented Aug 21, 2022 • edited

mmichon commented Aug 22, 2022

bdraco commented Aug 22, 2022

Mariusthvdb commented Aug 28, 2022 • edited

bdraco commented Aug 28, 2022

Mariusthvdb commented Aug 28, 2022

bdraco commented Aug 28, 2022

bdraco commented Aug 28, 2022

Mariusthvdb commented Aug 29, 2022 • edited

Mariusthvdb commented Aug 30, 2022

Mariusthvdb commented Sep 4, 2022 • edited

Mariusthvdb commented Aug 8, 2022 •

edited

Mariusthvdb commented Aug 8, 2022 •

edited

Mariusthvdb commented Aug 14, 2022 •

edited

Mariusthvdb commented Aug 15, 2022 •

edited

Mariusthvdb commented Aug 21, 2022 •

edited

Mariusthvdb commented Aug 28, 2022 •

edited

Mariusthvdb commented Aug 29, 2022 •

edited

Mariusthvdb commented Sep 4, 2022 •

edited