Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strange Bluetooth problem #3689

Closed
danergo opened this issue Jun 24, 2020 · 35 comments
Closed

Strange Bluetooth problem #3689

danergo opened this issue Jun 24, 2020 · 35 comments

Comments

@danergo
Copy link

danergo commented Jun 24, 2020

I'm using RPi4 for communicating with a BLE peripherial devices via gatttool:

# gatttool -b OTHERWORKINGDEVICE --characteristics
handle = 0x0002, char properties = 0x0a, char value handle = 0x0003, uuid = 00002a00-0000-1000-8000-00805f9b34fb
...

I have at least 3 peripherial devices which I can query with gatttool.

But there is one device which timeouts 9 times of 10 during connection:

# gatttool -b NOTWORKINGDEVICE --characteristics
connect error: Connection timed out (110)

Now I could consider the "notworkingdevice" faulty. But "unfortunately" it is working from other devices (phones, with nRF Connect application) without any issues.

So I have hooked up a Bluetooth Link Layer Sniffer and took some measurements during the Pi tries to connect:

BT_Sniff

Some legend:

  • Orange-filled box is the "faulty" device (BLE Peripherial)
  • Red-filled box is the Pi (BLE Central)
  • Red rectangles: marks channel change

I believe those Client Rx MTU requests are coming from the Peripherial (based on their Signal dBm).

However I tried running btmon on the Pi during the connection and it is simply not showing these MTU Requests. It seems Pi sends out a "LE Read Remote Used Features" and timeouts while waiting for the answer (I also tried increasing the timeout, but it did not help):

< HCI Command: LE Read Remote Used... (0x08|0x0016) plen 2  #11 [hci0] 7.477210
        Handle: 64
> HCI Event: Command Status (0x0f) plen 4                   #12 [hci0] 7.479342
      LE Read Remote Used Features (0x08|0x0016) ncmd 1
        Status: Success (0x00)
> HCI Event: Command Complete (0x0e) plen 14                #13 [hci0] 7.479357
      LE Read Remote Used Features (0x08|0x0016) ncmd 1
        Status: Success (0x00)
        00 00 00 00 00 00 00 00 00 00                    ..........      
> HCI Event: LE Meta Event (0x3e) plen 12                   #14 [hci0] 7.993969
      LE Read Remote Used Features (0x04)
        Status: Connection Timeout (0x08)
        Handle: 64
        Features: 0x2d 0x00 0x00 0x00 0x00 0x00 0x00 0x00
          LE Encryption
          Extended Reject Indication
          Slave-initiated Features Exchange
          LE Data Packet Length Extension

My Pi's details:
RPi4 Model B 1.1 (1GB RAM), Q2 2019 (Mfg by Sony)
PRETTY_NAME=“Raspbian GNU/Linux 10 (buster)”
NAME=“Raspbian GNU/Linux”
VERSION_ID=“10”
VERSION=“10 (buster)”

I also tried to put the peripherial very close (<1.5m) to the Pi and nothing has changed.

As last resort I tried to update the firmware but nothing has changed (I'm on 634e380a4d041492f859712bd2c81112a535b515).

Given the above facts, what are my options to further investigate what causes the problem?
Is it possible to update the Cypress module itself?

@pelwell
Copy link
Contributor

pelwell commented Jun 24, 2020

Can you capture simultaneous sniffer and btmon logs for the period covering a successful connection to one of your other devices? Two pairs of logs, one for the success case and one for the failure case, or one pair including both would be great.

@pelwell
Copy link
Contributor

pelwell commented Jun 24, 2020

P.S. I have wireshark and btmon, so the raw captures would be great - if you are unhappy publicly posting captures containing real addresses then you can email the captures or a download link to phil@raspberrypi.com.

@danergo
Copy link
Author

danergo commented Jun 24, 2020

Will do that soon, thanks.

@pelwell
Copy link
Contributor

pelwell commented Jun 24, 2020

Captures received - thanks.

@pelwell
Copy link
Contributor

pelwell commented Jun 24, 2020

I'm not used to reading BLE captures, but it looks like there are 5 connection attempts in sniff.pcap - the device in the first four don't respond to the LL_FEATURE_REQ, but the last one does - there are two LL_FEATURE_RSPs. Which of these were successful and which were failures?

@danergo
Copy link
Author

danergo commented Jun 24, 2020

I don't think that's the problem, please check this two images below:

I marked the packets with black which were sent by RPi (I believe based on RSSI), and red circles are where channel is switched.

Good packets (successful connection): (packets marked until No207)
Good packets

Bad packets: (packets marked until No82)
Bad packets

For your question: it seems LL_FEATURE_RSPs are only arriving after RPi send out this Connection parameter update request.

If my assumptions on the packets are correct, the question is, why RPi is not sending "Connection parameter update request".
Moreover it seems stuck in a state of requesting the MTU forever (until the whole connection times out).

BlueZ or Cypress chip does not handle well if there is a channel change right after the MTU request?

@danergo
Copy link
Author

danergo commented Jun 26, 2020

@pelwell: do we have some news by any chance?

Thank you

@pelwell
Copy link
Contributor

pelwell commented Jun 26, 2020

This problem has reached the limits of my BLE knowledge - I'm reasonably comfortable with HCI traffic, but the BLE on-the-wire activity is completely different - almost as if it was a completely different protocol bolted onto Bluetooth. ;-)

I prefer to have a reasonable level of understanding before before raising an issue with Cypress, but in this case I may have to make an exception.

@max17048
Copy link

As this belongs to 4B and sounds familiar to me.

What I need is bluetoothctl running on a 3B and 4B w/, same setup. Internal BT is switched off in config.txt (dtoverlay=disable-bt). Two external BT-Adapter are used.

Startet with:
sudo bluetoothctl power on
sudo bluetoothctl scan on

On 3B & Zero W, everything work fine, on 4B, BT stalls after a couple of minutes depending on environment Data (captured with btmon).

Used HW:
3B 1.2 (w/o 5GHz), Sony UK, a02082
Zero W 1.1, Sony UK, 9000c1
4B (4GB) 1.2, Sony UK, c03112
2x USB-BT-Dongle, same chip, 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)

OS: Linux 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux
Rpi-Buster-Lite, actual

Different setup, sam issue. What do you neet to check this problem as it belongs to HCI.

@danergo
Copy link
Author

danergo commented Jun 26, 2020

@max17048: I haven't planned this issue to be a general "Bluetooth problem collection". I don't know raspberrypi's rules of issues here, but almost any time is better to open a new issue with your problem. If it relates to something, it can be marked as related.

@pelwell
Copy link
Contributor

pelwell commented Jun 26, 2020

That's a completely different problem - probably USB-related - please open a new issue.

@danergo
Copy link
Author

danergo commented Jun 26, 2020

@pelwell: I've already opened an issue with Cypress: https://community.cypress.com/thread/55984, you might have better priority with them :)

@pelwell
Copy link
Contributor

pelwell commented Jun 26, 2020

I won't be opening a second issue - you already have engagement on that thread (more than I get, sometimes).

@milnepe
Copy link

milnepe commented Jun 27, 2020

I believe I have the same issue with Silicon Labs Thunderboard Sense2 & latest BG22 BLE boards. After updating raspbian they no longer connect. I can connect to both with nRF Connect on my Android phone.

Model : Raspberry Pi 3 Model B Plus Rev 1.3

pi@rpi02:~ $ uname -a
Linux rpi02 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux
pi@rpi02:~ $ cat /etc/debian_version 
10.3
pi@rpi02:~ $ gatttool -b $THUNDERBOARDSENSE2 --characteristics
handle = 0x0002, char properties = 0x0a, char value handle = 0x0003, uuid = 00002a00-0000-1000-8000-00805f9b34fb

pi@rpi02:~ $ gatttool -b $THUNDERBOARDBG22 --characteristics
handle = 0x0002, char properties = 0x20, char value handle = 0x0003, uuid = 00002a05-0000-1000-8000-00805f9b34fb

pi@rpi02:~ $ uname -a
Linux rpi02 4.19.118-v7+ #1311 SMP Mon Apr 27 14:21:24 BST 2020 armv7l GNU/Linux
pi@rpi02:~ $ cat /etc/debian_version
10.4
pi@rpi02:~ $ gatttool -b $THUNDERBOARDSENSE2 --characteristics
connect error: Connection timed out (110)

pi@rpi02:~ $ gatttool -b $THUNDERBOARDBG22 --characteristics
connect error: Connection timed out (110)

@danergo
Copy link
Author

danergo commented Jun 27, 2020

Seems the same from a distance, could you please post btmon logs during the connection trials preferrably both with 10.3 and 10.4? Then I can tell you for sure if this is the same.

Anyway I'm also on:

# cat /etc/debian_version
10.4

But I've upgraded my kernel to a super-fresh one:

# uname -a
Linux hs 5.4.47-v7l+ #1322 SMP Wed Jun 17 17:58:52 BST 2020 armv7l GNU/Linux

Note: this kernel update haven't solved anything, my issue is pending in Cypress's house.

@milnepe
Copy link

milnepe commented Jun 28, 2020

Here are btmon outputs while attempting to connect to Thunderboard Sense2 with gatttool. RPI02 (Debian 10.4) fails to connect but RPI03 (Debian 10.3) connects. Both RPi are Model : Raspberry Pi 3 Model B Plus Rev 1.3 and are using the same Thunderboard.

Also attached is output from Arduino Nano33BLESense connecting to RPI02 (Debian 10.4) successfully.

MAC addresses have been modified with sed.

thunderboardsense2-10.3.success.txt
thunderboardsense2-10.4.failure.txt
nano33blesense-10.4.success.txt

Note that the Thunderboard is correctly identified when it connects (right hand diff) but identified as an iBeacon when it does not (left hand diff)
image

@danergo
Copy link
Author

danergo commented Jun 28, 2020

Thanks @milnepe, indeed looks the same issue and as I can see ThunderBoardSense2 uses MTU negotiation, which is the case with my problematic device too. At least we have some light!

This could explain why I can connect to many BLE devices from the same RPi -> they aren't dealing with MTU stuff.

If there is a channel switch after the MTU Request (not visible in btmon!), then the process can get into a loop, which usually ends with timeout:

Bad attempt

(Note: this is the same image I've posted before).

If you check my successful attempt up there, you can see there is no channel switch after MTU request there, and in that case the connection succeeds.

@pelwell: if this happens to be a Debian 10.4 issue, where should we report this? Because it seems a Debian 10.4 issue with Raspberry Pi.

Thanks!

@danergo
Copy link
Author

danergo commented Jun 30, 2020

@milnepe: is your environment clean or fully occupied by 2.4GHz wifi signals?

I think the problem is that Cypress chip does not take into account the used Wifi channels, and usually tries initiating the connection in the already occupied channels.

And sometimes it has luck, but most of the time it has not.

That's why it works from other devices, because they are better in masking the wrong channels.

@pelwell: what do you think, can we somehow force the Cypress chip to not use at least those channels that it's already using for Wifi?

@pelwell
Copy link
Contributor

pelwell commented Jun 30, 2020

Both the 43455 and 43430 are supposed to include a link between the two sides allowing them to exchange "coexistence" information. There was a relatively recent (April 2020) patch that added a new entry to the "nvram" file (brcmgmac43455-sdio.txt) - "btc_params50=0x972c"; this is required since firmware release 177 to tune the coexistence mechanism.

@danergo
Copy link
Author

danergo commented Jun 30, 2020

That's cool, what is the best way to use this new firmware release and nvram file?

@milnepe
Copy link

milnepe commented Jun 30, 2020

All my tests were done on clean installations with no wifi (Ethernet only)

@pelwell
Copy link
Contributor

pelwell commented Jul 2, 2020

Those versions are in the current firmware-brcm80211 package.

@danergo
Copy link
Author

danergo commented Jul 2, 2020

Thanks. This issue is pending, as I'm waiting for inputs from various sources, but I won't forget to close it once I find resolution.

@benlhy
Copy link

benlhy commented Aug 16, 2020

Hello, I am also facing the same issue where I get connect error: Connection timed out (110) when trying to connect to a Thunderboard BG22 from a Rpi4 running the latest firmware-brcm80211 package. The Thunderboard works perfectly when paired to another device (nRF Connect). On the thunderboard, it detects the Rpi connecting and disconnecting.

I get successful connections for 1 in 30 attempts (on average) but I'm working in a environment with lots of 2.4Ghz Wifi signals.

@milnepe
Copy link

milnepe commented Nov 23, 2020

I don't know if this sheds any further light on the issue but I isolated the package version where the problem started - bluez-firmware_1.2-4+rpt3_all.deb

I flashed Raspbian Buster 2020-02-13 and put bluez-firmware_1.2-4+rpt2 on hold, then did an apt full-upgrade. At this stage there is no issue connecting to the Thunderboard. I then manually upgraded to bluez-firmware_1.2-4+rpt3 - this is when the Thunderboard no longer connects.

Here's my 3B+ at bluez-firmware_1.2-4+rpt2 - everything working :)

pi@rpi03:~ $ uname -a
Linux rpi03 5.4.72-v7+ #1356 SMP Thu Oct 22 13:56:54 BST 2020 armv7l GNU/Linux

pi@rpi03:~ $ apt list --installed | grep bluez
...
bluez-firmware/now 1.2-4+rpt2 all [installed,upgradable to: 1.2-4+rpt6]
bluez/testing,now 5.50-1.2~deb10u1+rpt2 armhf [installed,automatic]


Here's the same device after upgrading to bluez-firmware 1.2-4+rpt3 - connection no longer possible :(

pi@rpi03:~ $ apt list --installed | grep bluez
...
bluez-firmware/now 1.2-4+rpt3 all [installed,upgradable to: 1.2-4+rpt6]
bluez/testing,now 5.50-1.2~deb10u1+rpt2 armhf [installed,automatic]


I diff'd the package sources and the only difference is the update to BCM43430A1.hcd & BCM4345C0.hcd

pmilne@gigabyte:~/bluez$ diff -r bluez-firmware_1.2-4+rpt2.debian/ bluez-firmware_1.2-4+rpt3.debian/
Binary files bluez-firmware_1.2-4+rpt2.debian/broadcom/BCM43430A1.hcd and bluez-firmware_1.2-4+rpt3.debian/broadcom/BCM43430A1.hcd differ
Binary files bluez-firmware_1.2-4+rpt2.debian/broadcom/BCM4345C0.hcd and bluez-firmware_1.2-4+rpt3.debian/broadcom/BCM4345C0.hcd differ
diff -r bluez-firmware_1.2-4+rpt2.debian/debian/changelog bluez-firmware_1.2-4+rpt3.debian/debian/changelog
0a1,7

bluez-firmware (1.2-4+rpt3) buster; urgency=medium

  • Update BCM43430A1.hcd
  • Update BCM4345C0.hcd

-- Serge Schneider serge@raspberrypi.com Tue, 19 May 2020 08:56:02 +0100

Below are the two btmon traces - 1.2-4+rpt2 (LH) and 1.2-4+rpt3 (RH) - you can see event #14 time out with the updated firmware. It also reads one less byte during the Device Connected event which may be why it's now treating the remote as an iBeacon?

btmon-screenshot

It looks like the problem is in the Broadcom firmware binaries?

@pelwell
Copy link
Contributor

pelwell commented Nov 23, 2020

There is a new 43455 firmware available here to test: https://www.raspberrypi.org/forums/viewtopic.php?f=117&t=291609
Barring a major regression it will ship with future images.

@danergo
Copy link
Author

danergo commented Nov 23, 2020

@pelwell Thanks! Sounds promising. Will report back, but my problematic device is currently much further than BT range :)

@pelwell
Copy link
Contributor

pelwell commented Nov 23, 2020

Actually, you should grab the Bluetooth firmware from https://drive.google.com/file/d/1DVOtBjrsoR2NhwEBVn3ei0sv-xTIBCxR/view?usp=sharing - you'll need to copy it to /lib/firmware/brcm/BCM4345C0.hcd and reboot. It wasn't covered by that announcement and needs one of its own.

@pelwell
Copy link
Contributor

pelwell commented Nov 23, 2020

@milnepe
Copy link

milnepe commented Nov 23, 2020

The New Bluetooth firmware for CYW43455 (3B+, 4B, CM4) has fixed the Thunderboard connection issue by updating BCM4345C0.hcd mentioned above https://www.raspberrypi.org/forums/viewtopic.php?f=117&t=292113 Thanks!

@danergo
Copy link
Author

danergo commented Nov 23, 2020

Can't wait it to be tested! I'll do it and refer.

pelwell added a commit to RPi-Distro/bluez-firmware that referenced this issue Nov 25, 2020
This update fixes the failure of CYW43455 to accept BLE connections.

Original firmware name: BCM4345C0_003.001.025.0171.0339.hcd

See: raspberrypi/linux#3689

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
@benlhy
Copy link

benlhy commented Nov 28, 2020

Works for me!

@TwoTenPvP
Copy link

Works perfectly!

Just make sure you reboot after the upgrade.

Thanks!

@danergo
Copy link
Author

danergo commented Jan 1, 2021

Hi, @pelwell. Thanks a lot for the new firmware. Just have got a chance to test it and indeed it seems solving this issue.

I did a few tests, here is btmon's output:

< HCI Command: LE Read Remote Used.. (0x08|0x0016) plen 2  #31 [hci0] 11.423228
        Handle: 64
> HCI Event: Command Status (0x0f) plen 4                  #32 [hci0] 11.423829
      LE Read Remote Used Features (0x08|0x0016) ncmd 1
        Status: Success (0x00)
> HCI Event: Command Complete (0x0e) plen 14               #33 [hci0] 11.423841
      LE Read Remote Used Features (0x08|0x0016) ncmd 1
        Status: Success (0x00)
        00 00 00 00 00 00 00 00 00 00                    ..........
> ACL Data RX: Handle 64 flags 0x02 dlen 7                 #34 [hci0] 11.502093
      ATT: Exchange MTU Request (0x02) len 2
        Client RX MTU: 247
> HCI Event: LE Meta Event (0x3e) plen 12                  #35 [hci0] 11.502720
      LE Read Remote Used Features (0x04)
        Status: Success (0x00)
        Handle: 64
        Features: 0x2d 0x00 0x00 0x00 0x00 0x00 0x00 0x00
          LE Encryption
          Extended Reject Indication
          Slave-initiated Features Exchange
          LE Data Packet Length Extension

Compared to my original log:

  • LE Meta Event (0x3e) now succeeds (it was timed out before)
  • MTU size change request is now being sent (it was omitted before)

Thanks guys!
(Note: shall I close this issue?)
Happy New Year!

@pelwell
Copy link
Contributor

pelwell commented Jan 1, 2021

Thanks for the feedback, and for the original clear bug report.

@pelwell pelwell closed this as completed Jan 1, 2021
paul-1 added a commit to piCorePlayer/firmware-rpi-bt that referenced this issue Oct 30, 2023
This updated Bluetooth firmware concludes the Spectra fixes for
CYW43455.

Original firmware name: BCM4345C0_003.001.025.0174.0342.hcd

Signed-off-by: Phil Elwell <phil@raspberrypi.com>

Bug fix CYW43455 firmware
This update fixes the failure of CYW43455 to accept BLE connections.

Original firmware name: BCM4345C0_003.001.025.0171.0339.hcd

See: raspberrypi/linux#3689

Signed-off-by: Phil Elwell <phil@raspberrypi.com>

Update to BCM43455 firmware with Spectra fixes
This BCM43455 Bluetooth firmware update works around most of the
report "Spectra" class of vulnerabilities. CVE-2020-10370 will be
addressed in a follow-up release.

Original firmware name: BCM4345C0_003.001.025.0170.0336.hcd

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants