[TW#15023] Bluetooth Controller: LE Connection Interval only partially used

[mringwal]

I was running performance tests with BTstack's LE Streamer and LE Streamer Client. If ESP32 is LE Streamer Client (master role, receiving data) from LE Streamer running on my Mac with a CSR 8510 USB Bluetooth dongle without DLE and an ATT MTU of 197, I'm getting 30-33 kB/s which is pretty good.

In the other direction, when ESP32 is sending data in slave role, throughput is less than half.

In a test with minimum Connection Interval of 7.5 ms, it is only sending 3 packets and then waiting for the next Connection Event. Interestingly, the More Data bit is still set in the last packet and the master sends another empty packet, but nothing from ESP32.

A quick test with 100 ms Connection Interval showed about 80 ms not used.

I did check that BTstack is providing the outgoing data fast enough. BTstack sends 200 byte ACL packets which the Bluetooth Controller must split. According to the HCI Number Complete ACL Packets, there are always at least 7 packets buffered, so there are at least 56 ACL fragments that could be sent.

I've attached the two air traces for the TI Sniffer.

esp32-streamer-csr-client-low-throughput.zip

[Weijian-Espressif]

hi @mringwal , do you use the same connection parameters and MTU size when ESP32 is as master or slave? Can you give me the air trace when ESP32 is as master.

[mringwal]

hi @Weijian-Espressif. sorry for the late reply. All tests did use the same code: same connection parameters and MTU size.

I'll attach the log for the other direction: ESP32 is master and receives data, CSR is slave and sends data:
csr-streamer-esp32-client.psd.zip

Thanks for looking into this.

[Weijian-Espressif]

@mringwal esp_ble_gap_set_pkt_data_len(esp_bd_addr_t remote_device, uint16_t tx_data_length);
This api determines the fragment of acl, you can set the packet data length.

[mringwal]

@Weijian-Espressif Sorry, I don't understand your response. esp_ble_gap_set_pkt_data_len(...) is an Bluedroid API call but the issue isn't with Bluedroid (in fact, I'm not using Bluedroid at all).

The reported issue is about the Bluetooth Controller implementation in components/bt/lib.
When trying to send at full speed, while BTstack provides new ACL packets to send fast enough, the ESP32 Bluetooth Controller only uses a part of the available time in the connection interval to send data.

[Weijian-Espressif]

@mringwal files show esp32 as a slave will be lost some packet, once the ble lost the packet, esp32 will continue send data in the next connection interval. if as a master, it is good . now we are looking for the reason.

[mringwal]

the esp32 in slave role/when sending data doesn't loose any packets, it just could send them faster. I did not test master+sending or slave+receiving. When sending in slave role works well, i'll retest the other two configurations as well.

[Weijian-Espressif]

@mringwal Once a retransmission occurs, data will continue to send at the next connection interval.

[mringwal]

Hi, thanks for checking. In the image you've posted, the Slave sends packet #506 with SN=1, which is then ACKed by the Master in #507. Now, the Slave could send the next ACL packet after #507.

Packet #508 is the first packet of the next connection interval sent by the Master. I don't know right now what the "ACK Status = RETRY" means, but I'm interested why the Slave didn't send another (and many more) packets before that.

[negativekelvin]

@mringwal it looks like the slave did not receive 507 (even though the sniffer did) and master waits for next connection interval to retry?

[mringwal]

@negativekelvin yes, that's possible, although the sniffer did receive the packet.

the problem is that this happens in every connection interval and the throughput goes down because of this. also, there doesn't seem to be any packet loss when sending in the other direction. To me, this looks like either a limitation of the implementation or just a bug.

[negativekelvin]

@mringwal Yes I think this is the "packet loss" they are investigating so hopefully bugfix is coming

[Weijian-Espressif]

@mringwal we have fixed some bugs. please update to the latest IDF and try. Thanks.

[mringwal]

@Weijian-Espressif thanks for reaching out. I didn't manage to configure github to get a notification if there's a commit to the bt lib submodule.

According to the commit message, the fixes are related to this issue. I'll give it a try then! (and I need to figure out why my TI Sniffer doesn't work anymore, but if the the throughput goes up, that would be great)

[mringwal]

Hi @Weijian-Espressif. I've updated and re-run the same combinations. I get 35 kB/s between two CSR modules. I still get 30-33 kB/kB if ESP32 is LE Streamer Client. I get around 15-17 kB/s when ESP32 is LE Streamer. So, I cannot see much improvement.

Could you/did you try LE Streamer / LE Streamer Client on your side? Just disable the hci_dump_open call and run both examples on two devices.

[Yulong-espressif]

@mringwal I think the Weijian who describe to you just one of the throughput performance issues, we have find the most important issue affecting throughput. Now we have test in our side, the throughput in the BLE only mode with sending notify can up to 95kB/s. I will release the controller code as soon as possible.

[mringwal]

Hi @Yulong-espressif. 95 kB/s sounds great. That is with Data Length Extension. Did you get 90+ kB/s between two ESP32? Please let me know here (or by mail). I couldn't find a way to get notifications on updates on bt-controller submodule commits.

[mringwal]

hi! Any news? When will the fixes be merged in to the main repo?
I'd be happy to try the internal version - it's just a binary anyway.

[Yulong-espressif]

@mringwal We will release it next week, thank you.

[mringwal]

awesome!

[mringwal]

There was an update 'component/bt: Fixed the bug of ble multi connection not stable issue.' which sounds unrelated. I've updated the bt lib but don't get better better throughput.

Is this the fix to use more bandwidth and send faster?

FYI, I've also extended le_streamer_client in BTstack to select only write_without_response, notify, or both for better testing.

[Yulong-espressif]

@mringwal The throughput optimization submitted to the merge request, but still not merged IDF master, if you are in a hurry, we can send bt lib to you.

[mringwal]

Ah, ok. I'm not in a hurry, I'd just want to see that all problems get fixed - it's matthias@bluekitchen-gmbh.com

[mringwal]

@Yulong-espressif hi! has this fix made its way into the master branch yet?

[mringwal]

ping. any news about the merge of the fix?

[Yulong-espressif]

@mringwal, Sorry for the delay, will merge this week. Thanks.

[FayeY]

Has been merged into master, please have a try. Thanks.

[mringwal]

@FayeY I'm happy to confirm that the full connection interval is used in the original setup with the CSR 8510, getting the max 50 kB/s duplex throughput without data length extensions. Excellent. Thanks for fixing!