Bug - struct.error: unpack requires a buffer of 4 bytes

[perseus177]

Environment

Passive BLE monitor integration (Xiaomi Mijia BLE MiBeacon monitor)
Installed version: 0.9.4

Version	2021.1.0
Installation Type	Home Assistant Supervised
Development	false
Supervisor	true
Docker	true
Virtual Environment	false
Python Version	3.8.7
Operating System Family	Linux
Operating System Version	5.4.72-v7+
CPU Architecture	armv7l

---

Logger: homeassistant
Source: custom_components/ble_monitor/init.py:366
First occurred: 10. januára 2021, 16:57:47 (1 occurrences)
Last logged: 10. januára 2021, 16:57:47
Error doing job: Fatal error: protocol.data_received() call failed.

Traceback (most recent call last):
File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 862, in _read_ready__data_received self._protocol.data_received(data) File "/config/custom_components/ble_monitor/aioblescan_ext.py", line 1497, in data_received self.process(packet) File "/config/custom_components/ble_monitor/__init__.py", line 483, in process_hci_events msg, binary, measuring = self.parse_raw_message(data) File "/config/custom_components/ble_monitor/__init__.py", line 736, in parse_raw_message result.update(resfunc(xvalue)) File "/config/custom_components/ble_monitor/__init__.py", line 366, in obj0d10 (temp, humi) = TH_STRUCT.unpack(xobj) struct.error: unpack requires a buffer of 4 bytes

[Ernst79]

Hi, thanks for reporting.

This message indicates that the BLE message is recognized as Xiaomi Mi_Beacon message, 0d10 means that the message should contain temperature and humidity data in a 4 bytes long data message (2 for temperature, 2 for humidity), but apparently, the BLE data message is not 4 bytes long.

So, to figure out what is wrong, first a couple of questions, what sensor do you have?
And two, is this error an incident (happens once) or does it occur more often?

[perseus177]

Hello

I have

• 3 sensors LYWSDCGQ (Temperature and Humidity)
• 1 Kettle YM-K1501

Error occurred twice from yesterday.

[Ernst79]

And did you change something (update of ble_monitor?). Did you had it with earlier installations/versions, or is it a new installation?

[perseus177]

Changes:

• No change in configuration
• HA was updated to 2021.1.0 from 118 version - 7.01.2021
• BLE was not updated

[Ernst79]

Ok, than we need to figure out what is wrong with the data. Unfortunately, we will need to get some more debug information.

Could you add the 4 lines with _LOGGER.info to __init__.py. I've added the two lines before and one after, to show where to add it. It's close to the end of the file.

            xnext_point = xdata_point + 3 + xvalue_length
            xvalue = data[xdata_point + 3:xnext_point]
            _LOGGER.info("xvalue: %s", xvalue)
            _LOGGER.info("xvalue_length: %s", xvalue_length)
            _LOGGER.info("xvalue_typecode: %s", xvalue_typecode)
            _LOGGER.info("data: %s", data.hex())
            resfunc, tbinary, tmeasuring = self._dataobject_dict.get(xvalue_typecode, (None, None, None))

After that, please turn on debug logging by adding the following to configuration.yaml.

logger:
  default: warn
  logs:
    custom_components.ble_monitor: info

Now, restart Home Assistant and wait till the error occurs. Copy the info around the moment the error occurs from the log here.

[perseus177]

Hi I added Logger info but result is

Logger: homeassistant.setup
Source: loader.py:416
First occurred: 12:48:20 (1 occurrences)
Last logged: 12:48:20
Setup failed for ble_monitor: unknown error

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/setup.py", line 166, in _async_setup_component component = integration.get_component() File "/usr/src/homeassistant/homeassistant/loader.py", line 416, in get_component cache[self.domain] = importlib.import_module(self.pkg_path) File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 779, in exec_module File "<frozen importlib._bootstrap_external>", line 916, in get_code File "<frozen importlib._bootstrap_external>", line 846, in source_to_code File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/config/custom_components/ble_monitor/__init__.py", line 732 _LOGGER.info("xvalue: %s", xvalue)
^
TabError: inconsistent use of tabs and spaces in indentation

[Ernst79]

Sound like you have an error in the indentation (number of tabs/spaces). Can you post a printscreen of your code?

The lines should start at the same position.

[perseus177]

[Ernst79]

@perseus177 I thing there is something wrong with your installation. The above looks fine. I tried it myself and installed 0.9.4 and added the 4 lines. Runs fine.

And the results in the log:

2021-01-14 14:06:57 INFO (Thread-6) [custom_components.ble_monitor] xvalue: b'\xbc\x00'
2021-01-14 14:06:57 INFO (Thread-6) [custom_components.ble_monitor] xvalue_length: 2
2021-01-14 14:06:57 INFO (Thread-6) [custom_components.ble_monitor] xvalue_typecode: b'\x04\x10'
2021-01-14 14:06:57 INFO (Thread-6) [custom_components.ble_monitor] data: 043e2802010000f34f6b8d7cc41c020106030295fe141695fe712098002bf34f6b8d7cc40d041002bc00d9

I suggest you reinstall ble_monitor via HACS. Perhaps that also solves your issue.

[perseus177]

@Ernst79 Ernst79

• I updated via HACS to 0.9.6 version
• restarted HA
• Added _LOGGER.info to init.py
• Added logger to configuration.yaml
• restart HA

Same problem

Logger: homeassistant.setup
Source: loader.py:416
First occurred: 16:09:42 (1 occurrences)
Last logged: 16:09:42
Setup failed for ble_monitor: unknown error

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/setup.py", line 166, in _async_setup_component
component = integration.get_component()
File "/usr/src/homeassistant/homeassistant/loader.py", line 416, in get_component
cache[self.domain] = importlib.import_module(self.pkg_path)
File "/usr/local/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 779, in exec_module
File "", line 916, in get_code
File "", line 846, in source_to_code
File "", line 219, in _call_with_frames_removed
File "/config/custom_components/ble_monitor/init.py", line 736
_LOGGER.info("xvalue: %s", xvalue)
^
TabError: inconsistent use of tabs and spaces in indentation

[Ernst79]

I have really no idea why this doesn't work in your case.

Perhaps try to remove the empty space before the _LOGGER and type spaces yourself. Perhaps it has tabs in stead of spaces.

or try to remove the first 3 _LOGGER lines, only leave the one with _LOGGER.info("data: %s", data.hex()). if that doesn't work, move this line up to just above line 714 (while True:) in your screenshot (make sure the underscore starts at the same position as while).

[perseus177]

I will hope that error will not display again. You can close this ticket.

[Ernst79]

ok, let us know it it does, than we try further

[perseus177]

@Ernst79 Still same problem on version Home Assistant 2021.1.5 and BLE Monitor 1.0.5

Logger: homeassistant
Source: custom_components/ble_monitor/init.py:444
First occurred: 14:55:03 (1 occurrences)
Last logged: 14:55:03

Error doing job: Fatal error: protocol.data_received() call failed.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 862, in _read_ready__data_received
self._protocol.data_received(data)
File "/config/custom_components/ble_monitor/aioblescan_ext.py", line 1497, in data_received
self.process(packet)
File "/config/custom_components/ble_monitor/init.py", line 563, in process_hci_events
msg, binary, measuring = self.parse_raw_message(data)
File "/config/custom_components/ble_monitor/init.py", line 846, in parse_raw_message
result.update(resfunc(xvalue))
File "/config/custom_components/ble_monitor/init.py", line 444, in obj0d10
(temp, humi) = TH_STRUCT.unpack(xobj)
struct.error: unpack requires a buffer of 4 bytes

[Ernst79]

I’ll prepare a version for you that catches the actual data in case an error occurs. Please give me a few days, I’m on holiday now

[Ernst79]

Could you try the following: First, update to 1.1.0. Next, in the folder config/custom_components/ble_monitor, there is a file __init__.py, replace the following code around line 444.

        def obj0d10(xobj):
            (temp, humi) = TH_STRUCT.unpack(xobj)
            return {"temperature": temp / 10, "humidity": humi / 10}

Change this to

        def obj0d10(xobj):
            try:
                (temp, humi) = TH_STRUCT.unpack(xobj)
                return {"temperature": temp / 10, "humidity": humi / 10}
            except:
                _LOGGER.error("Error in unpacking temperature and humidity: Data is: %s (in hex: %s)", xobj, xobj.hex())
                return None

If the error occurs, it will display an error in the log. I think, ble_monitor will keep on working now, but it will display an error if it occurs. For a final fix, can you send me the error from the log. It will make me understand what data it is trying to unpack and why this doesn't work.

[perseus177]

Hi @Ernst79

• BLE updated to 1.1.1
• init.py - changed
• HA restarted

I am waiting for error, when error will display I will add it here

[perseus177]

@Ernst79

Logger: custom_components.ble_monitor
Source: custom_components/ble_monitor/init.py:450
Integration: Passive BLE monitor (documentation)
First occurred: 12:22:19 (2 occurrences)
Last logged: 12:58:26

Error in unpacking temperature and humidity: Data is: b'\x00\x00\xce\x01\xb0' (in hex: 0000ce01b0)
Error in unpacking temperature and humidity: Data is: b'\x00\x01\xb6\x04>' (in hex: 0001b6043e)

Logger: homeassistant
Source: custom_components/ble_monitor/init.py:881
First occurred: 12:22:19 (2 occurrences)
Last logged: 12:58:26
Error doing job: Fatal error: protocol.data_received() call failed.

Traceback (most recent call last):
File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 862, in _read_ready__data_received
self._protocol.data_received(data)
File "/config/custom_components/ble_monitor/aioblescan_ext.py", line 1853, in data_received
self.process(packet)
File "/config/custom_components/ble_monitor/init.py", line 570, in process_hci_events
msg, binary, measuring = self.parse_raw_message(data)
File "/config/custom_components/ble_monitor/init.py", line 666, in parse_raw_message
return self.parse_xiaomi(data, xiaomi_index, is_ext_packet)
File "/config/custom_components/ble_monitor/init.py", line 881, in parse_xiaomi
result.update(resfunc(xvalue))
TypeError: 'NoneType' object is not iterable

[Ernst79]

Thanks, I see these messages have 5 bytes in stead of 4. I'll will make a fix for this.

[Ernst79]

Fixed in 1.1.3.

[perseus177]

@Ernst79
After update to 1.1.3

Logger: custom_components.ble_monitor
Source: custom_components/ble_monitor/init.py:676
Integration: Passive BLE monitor (documentation)
First occurred: 12:28:26 (129 occurrences)
Last logged: 18:03:41

Invalid data: Invalid index
Invalid data: Invalid MAC address

[Ernst79]

Please update to 1.1.4. This is solved in 1.1.4.

[perseus177]

@Ernst79 updated to 1.1.4

I dont know if this is new bug or still the same

Logger: homeassistant
Source: custom_components/ble_monitor/init.py:703
First occurred: 12:36:55 (1 occurrences)
Last logged: 12:36:55
Error doing job: Fatal error: protocol.data_received() call failed.

Traceback (most recent call last):
File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 862, in _read_ready__data_received
self._protocol.data_received(data)
File "/config/custom_components/ble_monitor/aioblescan_ext.py", line 1853, in data_received
self.process(packet)
File "/config/custom_components/ble_monitor/init.py", line 572, in process_hci_events
msg, binary, measuring = self.parse_raw_message(data)
File "/config/custom_components/ble_monitor/init.py", line 669, in parse_raw_message
return self.parse_xiaomi(data, xiaomi_index, is_ext_packet)
File "/config/custom_components/ble_monitor/init.py", line 703, in parse_xiaomi
framectrl, = struct.unpack('>H', framectrl_data)
struct.error: unpack requires a buffer of 2 bytes

[Ernst79]

This is a new one, different BLE message (this one is Humidity only, the previous was temperature and humidity in one message. Strange thing is that you seem to be the only one that has this issue. Can't really explain it.

Anyway, I will create a similar fix.

[Ernst79]

I added the same fix (skip if the message doesn't have the correct length) for most messages. Is added in 1.2.0-beta.