Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Работа термометров в Home Assistant c Bluetooth LE Long Range. #297

Open
pvvx opened this issue Feb 17, 2023 · 89 comments
Labels
documentation Improvements or additions to documentation

Comments

@pvvx
Copy link
Owner

pvvx commented Feb 17, 2023

Предварительное описание решения.

Для примера можно взять:
Low-cost Realtek RTL8761 BT adapter

image

USB ID 0bda:8771 Realtek Semiconductor Corp. Bluetooth Radio

Сurrent version of Linux fw on RTL8761 supports BT5.1:

# hciconfig hci1 version
hci1:   Type: Primary  Bus: USB
        BD Address: 8C:88:2B:00:E1:EA  ACL MTU: 1021:6  SCO MTU: 255:12
        HCI Version: 5.1 (0xa)  Revision: 0x999
        LMP Version: 5.1 (0xa)  Subversion: 0x646b
        Manufacturer: Realtek Semiconductor Corporation (93)

Для работы HA c LE Long Range в интеграции “Bluetooth” желательно установить последнюю версию bluez и добавить в файле /lib/systemd/system/bluetooth.service
ExecStart=/usr/local/libexec/bluetooth/bluetoothd --experimental

Из-за неверной инициализации адаптеров BT5.+ в Bluez, перед запуском Home Assistant необходимо переключить BT адаптер на работу в режим Bluetoch 5.0:
hcitool -i hci1 cmd 08 31 03 05 05
или во всех режимах PHY 1M/2M/Coded:
hcitool -i hci1 cmd 08 31 03 07 07
Номер hciN - свой.
Проверить переключение возможно в btmon (log btmon):

@ RAW Open: hcitool (privileged) version 2.22
< HCI Command: LE Set Default PHY (0x08|0x0031) plen 3
        All PHYs preference: 0x03
          No TX PHY preference
          No RX PHY preference
        TX PHYs preference: 0x07
          LE 1M
          LE 2M
          LE Coded
        RX PHYs preference: 0x07
          LE 1M
          LE 2M
          LE Coded
> HCI Event: Command Complete (0x0e) plen 4
      LE Set Default PHY (0x08|0x0031) ncmd 2
        Status: Success (0x00)

В итоге, в интеграции "BTHome" получаем успешный прием термометров со стандартной рекламой и работающих в LE Long Range:

image

Если адаптер будет работать неустойчиво, тогда HA может его переинициализировать и настройка на работу в Coded PHY (Long Range) собьется. Ожидать официальной поддержки в интеграции “Bluetooth” нет никакой возможности. Выбранный автором API “Bleak” не позволяет управлять PHY адаптера и не предвидится никакой поддержки BLUETOOTH SPECIFICATION Version 5.0 от 2016 года.

BLUETOOTH SPECIFICATION Version 5.0 | Vol 1, Part C page 291
Core Specification Change History
9 CHANGES FROM v4.2 TO 5.0
9.1 NEW FEATURES
Several new features are introduced in the Bluetooth Core Specification 5.0
Release. The major areas of improvement are:
• Slot Availability Mask (SAM)
• 2 Msym/s PHY for LE
• LE Long Range
• High Duty Cycle Non-Connectable Advertising
• LE Advertising Extensions
• LE Channel Selection Algorithm #2 

Проверить адаптер на совместимость возможно с помощью команд:

# btmgmt phy
Supported phys: BR1M1SLOT BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE1MTX LE1MRX LE2MTX LE2MRX LECODEDTX LECODEDRX
Configurable phys: BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE2MTX LE2MRX LECODEDTX LECODEDRX
Selected phys: BR1M1SLOT BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LECODEDTX LECODEDRX

# hciconfig -a
hci0:   Type: Primary  Bus: USB
        BD Address: 8C:88:2B:20:8B:42  ACL MTU: 1021:6  SCO MTU: 255:12
        UP RUNNING
        RX bytes:2020 acl:0 sco:0 events:178 errors:0
        TX bytes:23965 acl:0 sco:0 commands:178 errors:0
        Features: 0xff 0xff 0xff 0xfe 0xdb 0xfd 0x7b 0x87
        Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3
        Link policy: RSWITCH HOLD SNIFF PARK
        Link mode: SLAVE ACCEPT
        Name: 'nanopineoplus2'
        Class: 0x000000
        Service Classes: Unspecified
        Device Class: Miscellaneous,
        HCI Version: 5.1 (0xa)  Revision: 0x999
        LMP Version: 5.1 (0xa)  Subversion: 0x646b
        Manufacturer: Realtek Semiconductor Corporation (93)

Установка вариантов PHY в btmgmt работает не совсем корректно (в некоторых вариантах имеются ошибки). Но и с помощью этой программы возможно установить необходимые PHY.

Для этого необходимо:

  1. Вывести текущие установки PHY:
# btmgmt -i 1 phy
Supported phys: BR1M1SLOT BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE1MTX LE1MRX LE2MTX LE2MRX LECODEDTX LECODEDRX
Configurable phys: BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE2MTX LE2MRX LECODEDTX LECODEDRX
Selected phys: BR1M1SLOT BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE1MTX LE1MRX
  1. К вариантам 'Selected phys' добавить опции ‘LECODEDTX’ и ‘LECODEDRX’:
# btmgmt -i 1 phy BR1M1SLOT BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE1MTX LE1MRX LECODEDTX LECODEDRX
PHY Configuration successfully set

Для установки PHY через API в bluez смотреть /doc/mgmt-api.txt.

Set PHY Configuration Command
=============================

	Command Code:		0x0045
	Controller Index:	<controller id>
	Command Parameters: 	Selected_PHYs (4 Octet)
	Return Parameters:

	This command is used to set the default PHY to the controller.

	This will be stored and used for all the subsequent scanning
	and connection initiation.

	The list of supported PHYs can be retrieved via the
	Get PHY Configuration command. Selecting unsupported or
	deselecting default PHYs will result in an Invalid Parameter
	error.

	This can be called at any point to change the Selected PHYs.

	Refer Get PHY Configuration command for PHYs parameter.

	This command generates a Command Complete event on success
	or a Command Status event on failure.

	Possible errors:	Invalid Parameters
				Invalid Index

Или через hci_send_cmd(). Пример для Python (неполный и не совсем корректный):

import os
import sys
import struct
import bluetooth._bluetooth as _bt

OGF_LE_CTL = 0x08
OCF_SET_DEFAULT_PHY = 0x0031

def set_hci_phy(hci_sock, phy):
    # save current filter
    old_filter = hci_sock.getsockopt(_bt.SOL_HCI, _bt.HCI_FILTER, 14)
    # Setup socket filter to receive only events related to the
    # write_inquiry_mode command
    flt = _bt.hci_filter_new()
    _bt.hci_filter_set_ptype(flt, _bt.HCI_EVENT_PKT)
    _bt.hci_filter_set_event(flt, _bt.EVT_CMD_COMPLETE);
    _bt.hci_filter_set_opcode(flt, _bt.cmd_opcode_pack(OGF_LE_CTL, OCF_SET_DEFAULT_PHY))
    hci_sock.setsockopt( _bt.SOL_HCI, _bt.HCI_FILTER, flt )
    #< HCI Command: ogf 0x08, ocf 0x0031, plen 3 [ 03 05 05 ]
    _bt.hci_send_cmd(hci_sock, OGF_LE_CTL, OCF_SET_DEFAULT_PHY, phy)
    #> HCI Event: 0x0e plen 4 02 31 20 00
    pkt = hci_sock.recv(255)
    status = struct.unpack("xxxxxxB", pkt)[0]
    print(status)
    # restore old filter
    hci_sock.setsockopt( _bt.SOL_HCI, _bt.HCI_FILTER, old_filter )
    return status

if __name__ == "__main__":
    dev_id = 1
    hci_sock = _bt.hci_open_dev(dev_id)
    status = set_hci_phy(hci_sock, struct.pack("<BBB", 3, 5, 5))
    _bt.hci_close_dev(dev_id)
@vartom
Copy link

vartom commented Feb 24, 2023

НА в докере, не получилось заставить работать с LR по предложенной инструкции. Может быть нужно ещё что то предпринять?

@pvvx
Copy link
Owner Author

pvvx commented Feb 24, 2023

На ходу, когда уже загружен HA, переключение адаптера не работает.
Требуется остановить HA.
Или выполнить:

hcitool -i hci1 cmd 08 31 03 04 04
hciconfig -a hci1 down
hciconfig -a hci1 up

В Linux, в Bluez, адаптер BT5.0 по старту системы инициализируется неправильно. От этого все проблемы в Linux.
Это аналогично тому, что адаптер с USB3.0 принудительно переключается в USB1.1, а API для переключения в USB3.0 нет. В итоге неясно, как Bluez получил какие либо сертификации. Остается одна надежда на ChatGPT и массовые увольнения “программистов”.

Интеграция "bluetooth" работает через прокладку "Bleak". "Bleak" в свою очередь работает через DBUS, Далее следующая "прокладка" - "Bluez", далее kernel. Это упрощенно, т.к. там десятки так называемых API прокладок.
Писатели "Bleak" вообще не в курсе что в 2016 году вышел стандарт BT5.0 и есть какие-то переключения PHY у адаптеров.
Просто недавно в bluez починили ошибки и в запрос сканирования теперь вставляется значение PHY по умолчанию, которые заданы на низком уровне адаптеру.
Но утилиты bluez ещё не умеют создавать connect с разными вариантами PHY описанными в BT5.0. Там ещё не исправлены ошибки.

WEB Bluetooth API нормально не работает в Linux. В Android оно может соединяться с любыми PHY, но меню сканирования у Chrome ещё не умеет работать с BT5.0. В Windows ситуация с WEB Bluetooth API почти аналогична с Linux, но там другие проблемы. Linux тормозят Bluez и Товальдс, подписывая кривые решения.

API Android работает без проблем с Coded PHY и всеми остальными форматами PHY. Примером является nRFConnect и многие другие приложения на Android.

Apple вообще никогда не придерживалось стандартов Bluetooth SIG. И там свои предрассудки.

По идее, в докере, всё должно быть аналогично, если произвести переключение в той части, которая управляет адаптером. Адаптеры BT5.0 умеют сканировать во всех вариантах PHY с 2016 года.
Бесшовно принимают рекламные пакеты от Bluetooth и всех вариантов BLE BT5.0, включая не стандартные на 2M PHY, если включить эти опции. Но работать с LE Long Range обычным пользователям не дают писатели софта и корпорации по неизвестным причинам. Возможно это подрывает коммерцию ZigBee...

@pvvx
Copy link
Owner Author

pvvx commented Feb 24, 2023

Интеграция "ble_monitor" в HA включает сканирование только на 1M PHY. Автор ещё не умеет переключать адаптер на работу в BT5.0.

@vartom
Copy link

vartom commented Feb 27, 2023

Спасибо за описание. Удалось таки перевести датчики lywsd03mmc на использование BLE LR в HA в докере. Пришлось перейти с ble_monitor на xiaomi_ble и заново их определить.
Через btmon проверил подключение действительно в LE Coded.
Посмотрю как это будет работать.

@pvvx
Copy link
Owner Author

pvvx commented Feb 27, 2023

Посмотрю как это будет работать.

Потребление от батарейки сильно возрастает в режиме LE Long Range. Более чем в 3 раза из-за увеличения времени передачи в 8 раз на пониженной модуляции.
Желательно изменить период трансляции рекламных сообщений на более чем 5 секунд.
Включение LE Long Range актуально если термометр расположен на удалении и адаптер его плохо видит.
Тестовые проверки при включении LE Long Range в Xiaomi LYWSD03MMC показывают, что связь с ним сохраняется на расстояниях более 500 метров и до 1 км по прямой видимости. В основном это требуется на отдельных частных участках с большими размерами. В городских "человейниках" эта опция бесполезна, т.к. 2.4ГГц плохо распространяется через препятствия в виде железо-бетонных и каменных стен.

Для достижения той же дальности в режиме "Legacy", по сравнению с режимом "LE long Range", требуется увеличение отдачи RF TX с +7 дБм (без учета коррекции ошибок в режиме "LE long Range"). Это недоступно на CR2032 из-за увеличения тока во столько же раз, во сколько требуется увеличение мощности передатчика.
По сравнению с Bluetooth 4.0 требуется ещё увеличить мощность передатчика в несколько раз...
При переключении на "LE Long Range" время передачи возрастает в 8 раз, но ток от батареи остается прежним (6..7мА при +0 дБм).

Стабильность работы проверена на приеме на дешевый адаптер без доп. антенн (указанный в заголовке темы) от термометра в соседнем доме-бане, находящимся примерно в 100 метрах. При этом уровень приема (RSSI) колеблется на уровне -102..-108 дБм по показаниям данного адаптера. Смартфон при этом спокойно соединяется с термометром в nRFConnect...

image
Это температура в предбаннике... При протопке печка дает повышение, без печки температура удерживается электронагревателем (мелкая волнистость линии - авто включение/выключение обогревателя)...

@pvvx
Copy link
Owner Author

pvvx commented May 13, 2023

image

aliexpress.com

# lsusb
Bus 007 Device 002: ID 0a12:0001  USB2.0-BT
# hciconfig -a
hci0:   Type: Primary  Bus: USB
        BD Address: 04:7F:0E:4A:0C:C9  ACL MTU: 1021:9  SCO MTU: 255:4
        UP RUNNING INQUIRY
        RX bytes:6445 acl:0 sco:0 events:463 errors:0
        TX bytes:3579 acl:0 sco:0 commands:347 errors:0
        Features: 0xbf 0xee 0xcd 0xfe 0xdb 0xff 0x7b 0x87
        Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV3
        Link policy: RSWITCH SNIFF
        Link mode: PERIPHERAL ACCEPT
        Name: 'BlueZ 5.64'
        Class: 0x000000
        Service Classes: Unspecified
        Device Class: Miscellaneous,
        HCI Version: 5.1 (0xa)  Revision: 0x89
        LMP Version: 5.1 (0xa)  Subversion: 0x89
        Manufacturer: Barrot Technology Limited (2279)

hcitool -i hci0 cmd 08 31 03 05 05

Long Range - ok.

@almirus
Copy link

almirus commented May 22, 2023

@pvvx
Добрый день! Xiaomi LYWSD03MMC прошит LYWSD03MMC Custom Firmware Version 4.2 на компе веб скрипты (сканер) видит устройство, дает прошить, иногда не может получить температуру влажность, иногда получает :)
19:48:54: Watching advertisements from "ATC_0B27B9"... 19:48:54: Argh! TypeError: device.watchAdvertisements is not a function
НО в HA со свистком Bluetooth 5.0 Realtek один раз добавился через Xiaomi BLE. Потом я недолго поигрался с параметрами через веб конфигуратор, и потом, чтобы я не делал, устройство более не добавляется (не видится)в HA (при этом CGPR1 продолжает нормально работать). Я и прошивал и на оригинальную и на модифицированную и батарею вынимал, и команду DD слал и 56, ничего не помогает.
Термометр в 3 метрах от Bluetooth свистка

@pvvx
Copy link
Owner Author

pvvx commented May 22, 2023

на компе веб скрипты (сканер) видит устройство, дает прошить, иногда не может получить температуру влажность, иногда получает :)
19:48:54: Watching advertisements from "ATC_0B27B9"... 19:48:54: Argh! TypeError: device.watchAdvertisements is not a function

На Windows "Advertising.html" практически не работает. Это проблема в Windows c Web bluetooth API.
Работает только на Android.

НО в HA со свистком Bluetooth 5.0 Realtek один раз добавился через Xiaomi BLE.

В HA две интеграции для работы с термометрами - "Bluetooth" и "Bluetooth Low Energy Monitor - Passive BLE monitor".
"Bluetooth Low Energy Monitor" работает со всеми выбранными в TelinkMiFlasher.html форматами.
"Bluetooth" имеет расширения "BTHome" и "Xiaomi BLE". Для них необходимо выбрать в TelinkMiFlasher.html формат BTHome или Mijia. А так-же поставить галочку "AdFlag":
image

со свистком Bluetooth 5.0 Realtek

Рисунки и названия адаптеров от продавцов ни о чем не говорят.
В Windows надо смотреть свойства адаптера в "диспетчере устройств".

По USB VID/PID можно узнать какой чип.

image

По номерам HCI/LMP - какой версии bluetooth:

image

@almirus
Copy link

almirus commented May 22, 2023

@pvvx
Может стоит в шапке веб конфигуратора указывать что нужно с мобильного заходить или я не нашел. с телефона конфигуратор заработал. Спасибо
Попробовал два формата BTHome, Mijia - не находит. Настройки же сразу применяются, батарею вытаскивать не нужно?

@pvvx
Copy link
Owner Author

pvvx commented May 22, 2023

Настройки применяются сразу, но BLE реклама не работает во время соединения.
Подключаться для настроек в TelinkMiFlasher.html можно в любой ОС.
А вот прием BLE рекламы в Windows зависит многих факторов. И пока в Chrome и других это опция - экспериментальная и предупреждение описано.
Если не что-то там не работает - обращайтесь примерно туда:
https://github.com/WebBluetoothCG/web-bluetooth
https://github.com/WebBluetoothCG/web-bluetooth/blob/main/implementation-status.md

@almirus
Copy link

almirus commented May 22, 2023

По USB VID/PID можно узнать какой чип.

image

@almirus
Copy link

almirus commented May 22, 2023

цель зависти все это не на виндовс, а на RPI 4 c Home Assistant и с этим свистком

@pvvx
Copy link
Owner Author

pvvx commented May 22, 2023

Google: USB\VID_0BDA&PID_8771 а это и есть 8761B

цель зависти все это не на виндовс, а на RPI 4 c Home Assistant и с этим свистком

А это уже ваши проблемы с версиями ОС на RPI 4 и Home Assistant.
Данный адаптер = тормоз. Ловит в 5..10 раз меньше реклам в единицу времени чем другие...

@almirus
Copy link

almirus commented May 22, 2023

image
то есть просто ждать и выгадать верное время для соединения ?

@pvvx
Copy link
Owner Author

pvvx commented May 22, 2023

Может отсоединяется по причине посаженной батарейки. Для соединения желательно чтобы было более 40% (+20С).
Далее будет работать c HA и до 0%.

@almirus
Copy link

almirus commented May 22, 2023

батарея 99%. а подключение к chrome на телефоне может мешать подключению к RPI4?

@pvvx
Copy link
Owner Author

pvvx commented May 22, 2023

Во время подключения BLE реклама, ту которую принимает HA, не передается.
При подключении потребление больше...

@almirus
Copy link

almirus commented May 22, 2023

спасибо за уделенное время. пока безуспешно. на тлф в браузере все ок
на HA - тишина, хотя с таким же конфигом неделю назад термометр соединился, а теперь ни в какую

@pvvx
Copy link
Owner Author

pvvx commented May 22, 2023

Отключите соединение с термометром в Android и в других. Закройте эксплорер.
В HA термометр появляется с задержкой... Лучше перезапустить всю систему с HA и подождать пару минут.

@almirus
Copy link

almirus commented May 22, 2023

@pvvx еще маленький вопрос, значок Bluetooth на самом устройстве что обозначает и должен ли он отображаться в момент рекламы (или соединения)?

@pvvx
Copy link
Owner Author

pvvx commented May 22, 2023

Значок Bluetooth активен только во время соединения и мигает если нажата впаянная кнопка или геркон с выбором в меню функции геркона/кнопки "соединение" (обычно для Long Range).

@almirus
Copy link

almirus commented May 22, 2023

@pvvx вместо Xiaomi BLE, установил аддон Passive BLE Monitor
image
И он нашел оба BLE устройства ¯_(ツ)_/¯ (на том же железе)

@Haldyz
Copy link

Haldyz commented Jan 3, 2024

Hello,
I have bought BT dongle from Aliexpress.
I want to use it in BLE mode on my Raspberry Pi 4, where is running my HA in Supervisor mode.
Once i plugged in the dongle to USB 2.0 port on my RPi, I can see this:

#lsusb
Bus 001 Device 003: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
# hciconfig -a
hci0:	Type: Primary  Bus: USB
	BD Address: 04:7F:0E:5A:C6:9E  ACL MTU: 1021:9  SCO MTU: 255:4
	UP RUNNING 
	RX bytes:12589 acl:0 sco:0 events:656 errors:0
	TX bytes:7093 acl:0 sco:0 commands:462 errors:0
	Features: 0xbf 0xee 0xcd 0xfe 0xdb 0xff 0x7b 0x87
	Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV3 
	Link policy: RSWITCH SNIFF 
	Link mode: SLAVE ACCEPT 
	Name: 'raspberrypi #1'
	Class: 0x4c0000
	Service Classes: Rendering, Capturing, Telephony
	Device Class: Miscellaneous, 
	HCI Version:  (0xa)  Revision: 0x89
	LMP Version:  (0xa)  Subversion: 0x89
	Manufacturer: not assigned (2279)

I have executed also following command:
hcitool -i hci0 cmd 08 31 03 05 05

I can see the Adapter in HA:
image

I have already:

  • flashed my LYWSD03MMC with latest ATC firmware
  • allowed BT5+ PHY
  • allowed LE Long Range
  • setup advertising type to BTHome
  • allowed AdFlags

Are these settings correct?

I'm asking because my thermometer is discovered in HA (via BTHome) only when its in the same room like my RPi with BT dongle.
When its placed in next room, behind 2 doors on the same floor of my house, there is no signal.
How can determine that its working in BLE mode? My understanding was that with BLE dongle I can cover all Xiaomi thermometers in my house. Is it correct?

My intention is to get rid off esp32_ble_trackers which I have in each floor of my house and extend thermometers battery life.

Thank you

@pvvx
Copy link
Owner Author

pvvx commented Jan 3, 2024

This adapter (ID 0a12:0001) works very poorly.
image
After purchasing it, it should be thrown in the trash.

@Haldyz
Copy link

Haldyz commented Jan 4, 2024

I would buy a new one, but i would like to have some recommendation. This one i have bough this one from the same Ali link you have shared previously. It seems to be the same like on your picture marked as V5.1 (Barrot BR865101).
image

I have also one more dongle, which is marked as V5.0, where is Barrot 8041A02 chip:
image

This one is recognized on my RPi like bellow, but its not connecting to Mi Thermometers at all:

#lsusb
Bus 001 Device 003: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
#hciconfig -a
hci2:	Type: Primary  Bus: USB
	BD Address: 00:1A:7D:DA:71:11  ACL MTU: 679:9  SCO MTU: 48:16
	UP RUNNING 
	RX bytes:684 acl:0 sco:0 events:48 errors:0
	TX bytes:3362 acl:0 sco:0 commands:48 errors:0
	Features: 0xff 0xff 0x87 0xfa 0xdb 0xbf 0x7b 0x83
	Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3 
	Link policy: RSWITCH HOLD SNIFF PARK 
	Link mode: SLAVE ACCEPT 
	Name: 'raspberrypi #3'
	Class: 0x4c0000
	Service Classes: Rendering, Capturing, Telephony
	Device Class: Miscellaneous, 
	HCI Version: 5.0 (0x9)  Revision: 0x810
	LMP Version: 5.0 (0x9)  Subversion: 0x2512
	Manufacturer: Cambridge Silicon Radio (10)

You have previously recommended to buy a Realtek RTL8761 BT adapter, but its not easy to find it on Ali according to the chipset name. Can you recommend some dongle which is tested and working well please?

@lbarjak
Copy link

lbarjak commented Jan 4, 2024 via email

@pvvx
Copy link
Owner Author

pvvx commented Jan 4, 2024

Can you recommend some dongle which is tested and working well please?

For "LE Long Range" only RTL8761. Otherwise, there are either a lot of errors in the firmware, or very poor sensitivity of the receiver.
Many WiFi-BT PCIe work well...

LE Long Range -> RTL8761BU (BT5.3) The best reception with the built-in antenna: sensitivity is better than -108..110 dBm. Realtek updates firmware. The firmware is available for download. Loaded into the adapter's RAM at startup. It is possible to create an alternative firmware if you obtain information from Realtek.

image

USB\VID_0BDA&PID_8771&REV_0200 - https://aliexpress.com/item/4001097686145.html RTL8761B

image

https://www.home-assistant.io/integrations/bluetooth/#realtek-rtl8761bu-adapters

@Haldyz
Copy link

Haldyz commented Jan 5, 2024

Thank you, I have ordered ASUS USB-BT500 from my local store, so I hope it will work well.
PS: your link to Ali is not valid, but its possible to search according to image (dongle with "easy idea" mark).

@Haldyz
Copy link

Haldyz commented Jan 9, 2024

Hello. I have finally received my new dongle - ASUS USB-BT500, which should be based on RTL8761BU.
But it seems it doesn't work on my RPi. I can see it in hciconfig, but the HCI even LMP Version is (0xa), which I don't think its right.

# hciconfig:
hci0:	Type: Primary  Bus: USB
	BD Address: C8:7F:54:98:B9:2B  ACL MTU: 1021:6  SCO MTU: 255:12
	UP RUNNING 
	RX bytes:16805 acl:0 sco:0 events:1389 errors:0
	TX bytes:19200 acl:0 sco:0 commands:1353 errors:0
	Features: 0xff 0xff 0xff 0xfe 0xdb 0xfd 0x7b 0x87
	Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3 
	Link policy: RSWITCH HOLD SNIFF PARK 
	Link mode: SLAVE ACCEPT 
	Name: 'raspberrypi #1'
	Class: 0x4c0000
	Service Classes: Rendering, Capturing, Telephony
	Device Class: Miscellaneous, 
	HCI Version:  (0xa)  Revision: 0xb
	LMP Version:  (0xa)  Subversion: 0x8761
	Manufacturer: Realtek Semiconductor Corporation (93)

# lsusb:
Bus 001 Device 003: ID 0b05:190e ASUSTek Computer, Inc. 

I have tested for now with one thermometer (LYWSD03MMC) and your latest firmware (v4.6) with following settings, but it's not connected via BTHome addon in HA.
image
image
image

Any idea what can be wrong please? Maybe a driver issue?
The only thing I have done was following settings followed by reboot of my RPi:
hcitool -i hci0 cmd 08 31 03 05 05

@lbarjak
Copy link

lbarjak commented Jan 9, 2024 via email

@Haldyz
Copy link

Haldyz commented Jan 9, 2024

My RPi 4B is running on 32bit OS with Linux Kernel 5.10.103-v7l+ armv7l.
On Asus pages is a 32bit driver which support kernel 2.6.32 - 4.15 - so my Kernel seems to be not supported.
When I checked /lib/firmware/ there are already rtl8761bu_config & rtl8761bu_fw files dated from 4.4.2023

sudo apt install build-essential
says build-essential is already the newest version (12.6).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

sudo make install INTERFACE=usb
make: *** No rule to make target 'install'. Stop.

I can try to copy and overwrite the files rtl8761bu_config & rtl8761bu_fw from dwonloaded driver to /lib/firmware/, but I'm not sure, if its a good idea (because of Kernel version).

I'm sorry, I'm not a Linux guru.

@lbarjak
Copy link

lbarjak commented Jan 9, 2024 via email

@Haldyz
Copy link

Haldyz commented Jan 9, 2024

Thank you @lbarjak . I have downloaded fw and config files from the link above and I have copied them to my RPi:

sudo cp -iv rtl8761bu_fw.bin /lib/firmware/rtl_bt/rtl8761bu_fw.bin
sudo cp -iv rtl8761bu_config.bin /lib/firmware/rtl_bt/rtl8761bu_config.bin

Then it was needed to switch the adapter to work with CodedPHY:
sudo hcitool -i hci0 cmd 08 31 03 04 04
and now it's finally working, even if the HCI & LMP Version is still reported as (0xa).

The main issue was probably the setting for CodedPHY. not the driver.
I have tried 3 options: for CodedPHY:

08 31 03 04 04
08 31 03 05 05
08 31 03 07 07

and seems that only 04 is working fine for me.
I do not understand, how could I determine, which settings is the best for my adapter, but I'm happy it's finally working.

One more thing I want to ensure - is the following settings the best option for BLE?
295167173-37c37e93-98ed-41de-8fbd-99fb3718d684

And the last question is about 'BTHome integration' vs 'Passive BLE Monitor' in HA.
I'm using BTHome, because 'Passive BLE Monitor' is not available for me - most probably because it's already somehow integrated directly in Home Assistant, but i do not understand, what it means? Where I can find it now?

PS: "Packet Id" & "Signal Strength" is not available in BTHome.

Update:
it's working only with the thermometers on the same floor of my house.
For thermometers which are downstairs, there is no signal, so I will need to keep them under ESPHome.

Thank you

@pvvx
Copy link
Owner Author

pvvx commented Jan 9, 2024

Realtek has long released a firmware (rtl8761bu_fw.bin, rtl8761bu_config.bin) version with BT5.3 for RTL8761BU.
I didn't update because... It also works with the old version.
Windows already has BT5.3 firmware installed.

08 31 03 04 04 
08 31 03 05 05
08 31 03 07 07

Bit0 - PHY 1Mbit (BLE Legacy)
Bit1 - PHY 2Mbit (Non-standard for BLE advertising)
Bit2 - PHY 125kbit (Coded PHY (S=8), LE Long Range)
Bit3 - PHY 500kbit (Coded PHY (S=2), Non-standard for BLE advertising)

@Haldyz
Copy link

Haldyz commented Jan 11, 2024

I'm sorry Victor for a dumb question, but I really do not understand the relation between i.e. 04 04 and the Bit0 - Bit3 you have described above. Please can you share more details? I'm interesting in which case to use 04/05/07 or maybe some other? My intention is to have as long range as possible.

And if you stated that Realtek has long released a firmware with BT5.3 for RTL8761BU, do you mean that adapter based on RTL8761BU which was sold as BT5.0 will support BT5.3 with latest firmware?

Thank you

@fanoush
Copy link

fanoush commented Jan 11, 2024

I'm sorry Victor for a dumb question, but I really do not understand the relation between i.e. 04 04 and the Bit0 - Bit3 you have described above

https://en.m.wikipedia.org/wiki/Binary_number#Binary_counting
https://en.m.wikipedia.org/wiki/Binary_number#/media/File%3ABinary_counter.gif

@Haldyz
Copy link

Haldyz commented Jan 12, 2024

Thank you @fanoush

So my understanding is following:

Bit0 - PHY 1Mbit (BLE Legacy)
Bit1 - PHY 2Mbit (Non-standard for BLE advertising)
Bit2 - PHY 125kbit (Coded PHY (S=8), LE Long Range)
Bit3 - PHY 500kbit (Coded PHY (S=2), Non-standard for BLE advertising)

04 dec is 0100 bin ->  PHY 2Mbit (Non-standard for BLE advertising)
05 dec is 0101 bin ->  PHY 2Mbit (Non-standard for BLE advertising) & PHY 500kbit (Coded PHY (S=2), Non-standard for BLE advertising)
06 dec is 0110 bin ->  PHY 2Mbit (Non-standard for BLE advertising) & PHY 125kbit (Coded PHY (S=8), LE Long Range)
07 dec is 0111 bin ->  PHY 2Mbit (Non-standard for BLE advertising) & PHY 125kbit (Coded PHY (S=8), LE Long Range) & PHY 500kbit (Coded PHY (S=2), Non-standard for BLE advertising)

Then becomes a question - which PHY is supported by my adapter (RTL8761BU)?
I was trying to check it with following commands, but they are invalid.

# btmgmt phy
Invalid command in menu main: phy

# btmgmt -i 0 phy
Invalid command in menu main: phy

I can try it just by testing of each combination, so if I understand it right, the best should be 08 31 03 07 07, where all bellow options should be available:
Bit1 - PHY 2Mbit (Non-standard for BLE advertising)
Bit2 - PHY 125kbit (Coded PHY (S=8), LE Long Range)
Bit3 - PHY 500kbit (Coded PHY (S=2), Non-standard for BLE advertising)

Am I right?

PS:

# sudo apt list --installed |grep bluez
bluez-firmware/oldoldstable,now 1.2-4+rpt8 all [installed,automatic]
bluez/oldoldstable,now 5.50-1.2~deb10u4 armhf [installed,automatic]

# cat /lib/systemd/system/bluetooth.service |grep bluetoothd
ExecStart=/usr/lib/bluetooth/bluetoothd --experimental

@pvvx
Copy link
Owner Author

pvvx commented Jan 12, 2024

For BLE advertising, the standard will be 0x05 to receive the advertising header on the main channels. That is, only 1M PHY and Coded S8 PHY are allowed in the Bluetooth SIG standard.

2M PHY is used only after negotiation during connection. Or with extended advertising, after receiving a header on 1M PHY or Сoded S8 PHY, which indicates that the second part will be on 2M PHY on an additional channel.

Therefore, 2 values are used - for the main channel and for the additional one. But this is not in the Linux API, which makes it impossible to support BT5.0. - Linux is dead.

Then becomes a question - which PHY is supported by my adapter (RTL8761BU)?

This chip supports all PHYs.
You can find out what your current version of Linux and the firmware loaded into the adapter supports using the command:
btmgmt phy

But that doesn't indicate what Bluez supports. Bluez is limited to pre-2014 knowledge.

@ilgrank
Copy link

ilgrank commented Feb 15, 2024

@pvvx : but if bluez is frozen in 2014, and windows is in the same bad situation, and ESP32 simply don't support BLE LR.. What are you using on the remote side of your sensors? I get you have a realtek USB adapter, the one I bought too on aliexpress.. But what OS? What BLE to MQTT solution (if any)?
Thanks!

@fanoush
Copy link

fanoush commented Feb 15, 2024

What BLE to MQTT solution (if any)?

One option could be dedicated usb dongle based on nrf52840. I do have long range/coded phy working in Espruino which is javascript interpreter acessible over usb serial (or BLE of course). There is also
https://github.com/espruino/EspruinoHub which is BLE to MQTT solution that works with Espruino based devices.

See e.g. this https://forum.espruino.com/conversations/393987/ or https://forum.espruino.com/conversations/361380/

I don't use home assistant or MQTT but I believe it should work.

One possible hardware is E104-BT5040U which is ~US$10 thing on Aliexpress https://www.aliexpress.com/item/1005004625411410.html

It is compatible with original Nordic 52840 dongle (PCA10059) so all the tools work with it just fine, you can upload new firmware over USB with nRF Connect for Desktop which is available both for Windows and Linux.

Here is quick sample of scanning for all devices advertising over coded PHY showing one of my thermometers switched to long range mode.

NRF.findDevices(function(devices) { console.log(devices);}, {timeout : 2000, active : false, phy:"coded"});
=undefined
[
  BluetoothDevice: {
    "id": "a4:c1:38:c9:52:1e public",
    "rssi": -55,
    "data": new Uint8Array([2, 1, 6, 18, 22, 26, 24, 30, 82, 201, 56, 193, 164, 122, 9, 66, 20, 217, 10, 82, 136, 4, 11, 9, 76, 89, 87, 83, 68, 48, 51, 45, 49, 69]).buffer,
    "name": "LYWSD03-1E",
    "serviceData": {
      "181a": new Uint8Array([30, 82, 201, 56, 193, 164, 122, 9, 66, 20, 217, 10, 82, 136, 4]).buffer
     }
   }
 ]
>

EDIT: or for cheaper 52840 boards check https://github.com/joric/nrfmicro/wiki/Alternatives
there are boards like this https://www.aliexpress.com/item/1005006271779544.html
However I am not sure how good is the antenna if you really need long range. One cheap device with external antenna connector was the Particle Xenon which is now discontinued and hard to get (have Espruino working on that one too)

@pvvx
Copy link
Owner Author

pvvx commented Feb 15, 2024

@fanoush - NRF has not released USB-BT adapters or software for these options.

@pvvx
Copy link
Owner Author

pvvx commented Feb 15, 2024

What BLE to MQTT solution (if any)?

I don't use MQTT. I receive BLE in Home Assistant using two simple USB-BT adapters. One works for the near zone, the second for the far zone in the “LE Long Range” mode.
Backup BLE receiving channels are organized on various small SoCs, with my programs.

@fanoush
Copy link

fanoush commented Feb 15, 2024

NRF has not released USB-BT adapters or software for these options.

I am not talking about (normal HCI) USB-BT adapters. With espruino these dongles can run your stored javascript code, talk to anything over BLE and then write to USB serial any output you need. So the logic is in javascrit code inside the dongle, if the MQTT broker (like EspruinoHub) or the Home Assistant directly can read from serial port it should work.

@pvvx
Copy link
Owner Author

pvvx commented Feb 15, 2024

No versatility. For each task on NRF you will have to create a new program.

Today there are cheaper chips with full support for BT 5.4 from WCH. Less than 1 dollar.

The CH32V208W is a little more expensive. It has BT5.4, USB, Ethernet,... The SDK contains examples of working with BLE+MQTT+Ethernet...

@fanoush
Copy link

fanoush commented Feb 15, 2024

No versatility. For each task on NRF you will have to create a new program.

Oh! Well, I think you don't quite get it with the versatility, but never mind. if you have solution that works for you than that's great.

Here is one example of 'no versatility' https://www.espruino.com/Pixl.js+BLE+Ethernet+Bridge - it is NRF device with BLE and ethernet shield connected over SPI. In just few lines of javascript you convert any BLE advertisement seen around into MQTT message and send it to server over ethernet.

You don't need to write that stuff in C like with SDK for CH32V208W . If there would be e.g. micropython for that CH32V208W with working BLE api that can do long range then it would be similar.

@pvvx
Copy link
Owner Author

pvvx commented Feb 15, 2024

Who cares - I can do it faster in C.

But no one has been able to write a normal BLE gateway on nRF. For years already. And who in the modern world needs chips with libraries in blobs? NRF - proprietary software. - Correction of errors in nRF binary libs has to wait for years.

@lbarjak
Copy link

lbarjak commented Feb 15, 2024 via email

@fanoush
Copy link

fanoush commented Feb 16, 2024

But no one has been able to write a normal BLE gateway on nRF. For years already.

Not sure what you mean by "normal BLE gateway". If you mean HCI then we already discussed that, it is part of ZephyrOS and works. However due to poor Linux support as you said many times it is not very useful to get long range/coded phy working in Linux.

And who in the modern world needs chips with libraries in blobs? NRF - proprietary software

This was already discussed too. There are several open source production ready certified BLE stacks for nRF5x chips. Hardly any other chip has such wide open source support. There is ZephyrOS which is also current supported official SDK from Nordic (called nRF Connect SDK), there is also Apache NimBLE and there is at least one commercial one with full sources on GitHub which I think needs license for commercial use (forgot its name, maybe it was this one https://github.com/packetcraft-inc/stacks ?). SoftDevices are legacy and you are definitely not forced (or even recommended) to use them if you don't appreciate the advantages (stability, simplicity, clear separation from your code).

@pvvx
Copy link
Owner Author

pvvx commented Feb 17, 2024

This was already discussed too.

Again, one nRF advertisement, but in reality there are no devices for users or DIY on nRF.

Not sure what you mean by "normal BLE gateway".

I want to enable 'PAWR' on all thermometers, but users have no adapters capable of working with PAwR.
MQTT is just suitable for working with BLE 'PAwR'.
Create a cheap NRF project for working with BLE 'PAwR', and do not write advertising posts ...

@fanoush
Copy link

fanoush commented Feb 17, 2024

Again, one nRF advertisement, but in reality there are no devices for users or DIY on nRF.

Well I was just disproving your untruths. Would be nice if you could stop so there is no need to correct you over and over again (it is also off topic here).

Sadly you just did it again. There are of course tons of nrf5x based devices directly targeted for DYI both made by companies and also individuals and there is also a lot of hackable cheap nrf52 based stuff on aliexpress (smartwatches, fitness trackers, NFC+BLE keychains). For companies just check sparkfun or adafruit for their own nrf5x boards, adafruit even maintains Arduino and also Circuitpython for nrf52 just check list here https://github.com/adafruit/circuitpython/tree/main/ports/nrf/boards , also Microbit is NRF, the version 2 is 52833 and widely available (version 1 was bad IMO but 2 is nice). The Espruino stuff is mostly nrf52, previously I linked nrf52 boards for doing custom BLE/USB HID keyboards https://github.com/joric/nrfmicro/wiki/Alternatives For some reason they like 52840 based boards for this, not sure why, but they are probably quite popular as they got cloned and sold on aliexpress in quantities (google "ZMK firmware"). The Pinetime watch is nrf52832 - they are using Apache NimBLE there in their firmware. In fact it is one of many cheap hackable OTA updatable DaFit nrf5x based smartwatches which come in many shapes and sizes. One example is the 12 EUR "C17" nrf52840 one a.k.a. Magic 3 you can find on aliexpress now but there are many others. So you must be living in some parallel universe to say that thing I quoted. Or you just like to troll.

I want to enable 'PAWR' on all thermometers, but users have no adapters capable of working with PAwR. MQTT is just suitable for working with BLE 'PAwR'.

Oh, that is actually interesting stuff I did not know yet. Some time ago I tried to sync precise time via advertising but got bitten by that random delay and also wanted to send different scan response based on who is asking (so having bidirectional communication via advertising+scan response), it is nice someone was thinking in same direction and fixing both issues :-) GATT and connections are too heavy for a lot of stuff.

@pvvx
Copy link
Owner Author

pvvx commented Feb 17, 2024

There are of course tons of nrf5x based devices directly targeted for DYI

Where is the firmware for this thermometer to work in BTHome?

You are describing some toys that are not needed in IoT.
There are many more DIY boards available with other BLE chips. And there are many more of them than with SoС from the NRF

Low cost BLE Advertisements Repeater into Ethernet TCP/IP

@fanoush - How long will you advertise nRF or will you do something? :) :)

So you must be living in some parallel universe to say that thing I quoted.

Or do you need to block access to the repository "from this universe"? :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

10 participants