'Encoded value not valid' error on data format 4 tags

[holitics]

Environment (please complete the following information):

• OS: Raspbian Stretch
• ruuvitag_sensor package version: 0.13
• RuuviTag firmware version: Not sure ?
• Data format: 4

I get a stderr sometimes on data format 4 tags, but I seem to still get the data. Not sure what 'c' is - maybe that is the problem.

E2:A6:C1:89:4E:03
{'data_format': 4, 'temperature': 23.0, 'humidity': 42.0, 'pressure': 1003.0, 'identifier': 'c'}

From stdout / stderr:

Encoded value: BFQX>+ not valid
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.7/site-packages/ruuvitag_sensor/decoder.py", line 89, in decode_data
decoded = bytearray(base64.b64decode(encoded, '-_'))
File "/usr/lib/python3.7/base64.py", line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Invalid base64-encoded string: number of data characters (5) cannot be 1 more than a multiple of 4
Decoded data is null. MAC: E2:A6:C1:89:4E:03 - Raw: 1F0201060303AAFE1716AAFE10F9037275752E76692F2342465158CE00043E2B02
Can't init device hci0: Connection timed out (110)

[holitics]

I also get weird stuff in the log, when connecting to this RuuviTag, I get strange things in the Identifier:

{'data_format': 4, 'temperature': 23.0, 'humidity': 46.0, 'pressure': 1003.0, 'identifier': 'c\x04>+\x02\x01\x03\x01v$\x1a;\x1f\x1e\x06\x00\x01\t \x02\AlawDjM\x04>\x02\x01\x03\x01VsJ\x1e\x02\x01\x06\x03\x03\x16\x16\x10\x03ruu.vi/#AlwXAMR8\x04>+\x02\x01\x03\x01v$\x1a;\x1f\x1e\x06\x00\x01\t \x02\AlawDjM\x04>+\x02\x01\x03\x01v$\x1a;\x1f\x1e\x06\x00\x01\t \x02\AlawDjM\x04>\x1b\x02\x01\x00\x00\x16P\x0f\x02\x01\x1a\x0bL\x00\t\x06\x03\n\x00\x00\x04>+\x02\x01\x03\x01\x03N\x1f\x02\x01\x06\x03\x03\x17\x16\x10\x03ruu.vi/#BFwXAMR8c\x04>+\x02\x01\x03\x01v$\x1a;\x1f\x1e\x06\x00\x01\t \x02\AlawDjM\x04>\x1b\x02\x01\x00\x00\x16P\x0f\x02\x01\x1a\x0bL\x00\t\x06\x03\n\x00\x00\x04>+\x02\x01\x03\x01v$\x1a;\x1f\x1e\x06\x00\x01\t \x02\AlawDjM\x04>\x02\x01\x03\x01VsJ\x1e\x02\x01\x06\x03\x03\x16\x16\x10\x03ruu.vi/#AlwXAMR8\x04>+\x02\x01\x03\x01v$\x1a;\x1f\x1e\x06\x00\x01\t \x02\AlawDjM\x04>+\x02\x01\x03\x01\x03N\x1f\x02\x01\x06\x03\x03\x17\x16\x10\x03ruu.vi/#BFwXAMR8c\x04>+\x02\x01\x03\x01v$\x1a;\x1f\x1e\x06\x00\x01\t \x02\AlawDjM\x04>+\x02\x01\x03\x01v$\x1a;\x1f\x1e\x06\x00\x01\t \x02\AlawDjM\x04>\x02\x01\x03\x01VsJ\x1e\x02\x01\x06\x03\x03\x16\x16\x10\x03ruu.vi/#AlwXAMR8\x04>\x02\x01\x03\x01\x12Y\x1e\x02\x01\x06\x03\x03\x16\x16\x10\x03ruu.vi/#AlQYAMR8\x04>+\x02\x01\x03\x01v$\x1a;\x1f\x1e\x06\x00\x01\t \x02\Ala'}

[holitics]

I also just got the incorrect padding error (I know this has been previously reported). I am only getting these problems with the data_format 4 tags. My data_format 2 tags are working great.

=====

Encoded value: BFwXMTg not valid
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.7/site-packages/ruuvitag_sensor/decoder.py", line 89, in decode_data
decoded = bytearray(base64.b64decode(encoded, '-_'))
File "/usr/lib/python3.7/base64.py", line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Incorrect padding
Decoded data is null. MAC: E2:A6:C1:89:4E:03 - Raw: 1F0201060303AAFE1716AAFE10F9037275752E76692F23424677584D00546763D3

[ojousima]

If there is no special reason to use data format 4, I'd update to https://lab.ruuvi.com/dfu 2.4.2 and use data format 5 / RAWv2.

Original reasoning to use URL-encoded data was to use Google's Nearby notifications to let users view the tags without needing to install any app, but since the Nearby has been discontinued there isn't a any special benefit in using Eddystone format anymore.

[holitics]

Thanks for the information. ATM no reason to use Eddystone, but there must be a large number of tags in the wild that are using this format? I received a number of these last year and just got around to start playing with them.

I'll update to latest firmware and give it a shot.

[ojousima]

I think the last tags which send in format by default 4 shipped out in summer 2018, but there certainly are a lot of tags using that format in the wild, even the current tags can be configured to send in format 4 for backwards compatibility

[ttu]

Hi,

I have been suspecting that Data Format 4 data conversion returns wrong results sometimes. I just don't have any idea how to try to fix it.

ruuvitag-sensor/ruuvitag_sensor/data_formats.py

Line 43 in 212a19b

base16_split = [raw[i:i + 2] for i in range(0, len(raw), 2)]

Tried to change the hex string conversion, but is seems to be giving same kind of results, so I'm not anymore sure if conversion is broken or is there something from with the data coming from the Bluez.

RAW: 1F0201060303AAFE1716AAFE10F9037275752E76692F2342465158CE00043E2B02

by = bytes.fromhex(raw)

b'\x1f\x02\x01\x06\x03\x03\xaa\xfe\x17\x16\xaa\xfe\x10\xf9\x03ruu.vi/#BFQX\xce\x00\x04>+\x02'

Working case e.g.

RAW: 1F0201060303AAFE1716AAFE10F6037275752E76692F234248415A414D576F77C9

b'\x1f\x02\x01\x06\x03\x03\xaa\xfe\x17\x16\xaa\xfe\x10\xf6\x03ruu.vi/#BHAZAMWow\xc9'

data: BHAZAMWow

Also Data Format 2 uses same conversion as DF4, so I'm not sure what can be wrong.

If you want you can try to update FW to 1.2.1 and try again with DF4 and tell me if you still get the errors.

In any case I will add a disclaimer to the README, that there is no actual need to use Eddystone format anymore as @ojousima said.

[holitics]

Hi there, I've updated the FW and now the tag works really nicely with DF3:

Found TAG E2:A6:C1:89:4E:03, DATA {'data_format': 3, 'humidity': 41.5, 'temperature': 24.84, 'pressure': 1013.37, 'acceleration': 1008.3932764551735, 'acceleration_x': -28, 'acceleration_y': -3, 'acceleration_z': 1008, 'battery': 3133}

I'll try to test the latest with DF4 but it will have to wait until later next week.

[ttu]

Added note that DF 4 should not be used (ccbc4e7). Will close this for now as it is unlikely that will find time to fix this.