A2DP Source stream audio to two Sink device (IDFGH-253)

[rob-bits]

Hello all, @blueMoodBHD @mywang-espressif @igrr @FayeY

I want to add new functionality to the A2DP source capability of the esp-idf. Especially I want to stream audio from ESP32 to two bluetooth speaker.

I want to reach this funtionality with the design guidelines of the bluetooth.org from the study called "WHITE PAPER ON USAGE OF MULTIPLE HEADPHONES":
(https://www.bluetooth.org/DocMan/handlers/DownloadDoc.ashx?doc_id=110679)

In this study there are use cases how should a source device act if the user wants to connect two sink audio device.

One of the scenario follows the next steps:

1. AVDTP & AVCTP Connection establish with Sink device 1
2. A2DP SEP Disc/Selec with Sink device 1
3. SET_CONFIG for Sink device 1
4. OPEN connection with Sink device 1
5. START meadia Sink device 1 (ESP_A2D_MEDIA_CTRL_START)
6. STREAM Media to Sink device 1
   ---- new device connection----
7. GAP/ACL Connect to Sink device 2
8. AVDTP Establish, SEP Disc/Select with Sink device 2
9. Suspend media to Sink device 1 (ESP_A2D_MEDIA_CTRL_SUSPEND)
10. SET_CONFIG to Sink device 2
11. OPEN connection with Sink device 2
12. START Media on Sink device 2 (ESP_A2D_MEDIA_CTRL_START)
13. STREAM Media to Sink device 1
14. STREAM Media to Sink device 2
15. HAPPY USERs

Would somebody give some idea how to start for the 7-8, and 10th steps? The base application would be the A2DP_source example code.

The study states we need to have two (or more) Stream End Point, SEPs on the SRC device that support the SBC codec.

I have already figured out the SEPs are initialized with the bta_av_api_register() function in the bt/bluedroid/bta/av/bta_av_main.c file. The SEPs are handled in a tBTA_AV_SCB struct with the following code:
tBTA_AV_SEP seps[BTA_AV_MAX_SEPS];
Here it can be seen the BlueDroid stack is already prepared somehow to handle more connection.
As I go through the code in the current esp-idf there are two streams are handled. One seperated stream for the source device and one seperated stream for sink device. And that is why the BTA_AV_MAX_SEPS equals two. I guess...

In the bt/bluedroid/btc/profile/std/a2dp/btc_a2dp_source.c file, the btc_a2dp_source_send_aa_frame() function prepares the audio data/frame what we want to send to the sink device. And it would be good if the created frame would be sended to both of the connected devices in a queue. In that manner all SNK devices will be receiving identical A2DP media packets. But I have no idea where to start in this manner. Maybe the btc_a2dp_source_task_handler() function needs some rework too.

The study mentioned the following:

When the SRC connects to another SNK device the configuration of the SBC codec should be the same as the configuration of the SBC codec in the existing SNK device.

In the code hwo can I attach the same SBC configuration for both of the device?

Anyway, these are very small slices of the "game" and I know this is not a small rewrite. I just really need some help and advice which functions shall be rewrited.

Any advice are welcomed!

Rob

UPDATE 2018.06.22

I modified the btc_av.c , btc_avrc.c and the main API state handler to handle the multiple connections.

Basically I modified two main things:

In the btc_av.c the following:

typedef struct {
    int service_id[BT_NUM_OF_DEVICES];
    tBTA_AV_HNDL bta_handle[BT_NUM_OF_DEVICES];
    uint8_t peer_bda[BT_NUM_OF_DEVICES][BD_ADDR_LEN];
    btc_sm_handle_t sm_handle[BT_NUM_OF_DEVICES];
    UINT8 flags[BT_NUM_OF_DEVICES];
    tBTA_AV_EDR edr[BT_NUM_OF_DEVICES];
    UINT8   peer_sep[BT_NUM_OF_DEVICES];  /* sep type of peer device */
	UINT8	connected_dev_num; //< count how much device connected
	UINT8	active_dev_idx; //< index number of the activated device
	UINT8	conn_slot; //< how much device is already connected
	UINT8   media_ctrl_pending;  //< flag for indication of the media control is pending
} btc_av_cb_t;

And in the btc_avrc.c:

typedef struct {
    BOOLEAN                     rc_connected[BT_NUM_OF_DEVICES];
    UINT8                       rc_handle[BT_NUM_OF_DEVICES];
    tBTA_AV_FEAT                rc_features[BT_NUM_OF_DEVICES];
    uint8_t                     rc_addr[BT_NUM_OF_DEVICES][BD_ADDR_LEN];
    UINT16                      rc_pending_play[BT_NUM_OF_DEVICES];
    btc_rc_cmd_ctxt_t           rc_pdu_info[MAX_CMD_QUEUE_LEN]; //todo: this is not prepared fow two sink
    btc_rc_reg_notifications_t  rc_notif[MAX_RC_NOTIFICATIONS]; //todo: this is not prepared fow two sink
    unsigned int                rc_volume[BT_NUM_OF_DEVICES];
    uint8_t                     rc_vol_label[BT_NUM_OF_DEVICES];
	uint8_t						active_dev_idx; //< index number of the activated device
	uint8_t						conn_slot; //< index number of the activated device
} btc_rc_cb_t;

And I modified all of the functions whose are using these variables. But I run into an error when the second device tries to connec: av scb not available for avdt connection

This came from the bta_av_sig_chg() function in the bt/bluedroid/bta/av/bta_av_act.c file:

           /* check if we found something */
            if (xx == BTA_AV_NUM_LINKS) {
                /* We do not have scb for this avdt connection.     */
                /* Silently close the connection.                   */
                APPL_TRACE_ERROR("av scb not available for avdt connection");
                AVDT_DisconnectReq (p_data->str_msg.bd_addr, NULL);
                return;
}

Any advice what can be the problem?

[yoelrc88]

Hi @rob-bits, I am also interested in this topic. Thanks for the white paper reference. Will be following this conversation.
Thanks

[rob-bits]

This is my current error log. It can be seen it get stuck at the av "scb not available for avdt connection" part. But before this error message there is an another "BT_OSI: alarm_set failed to start timer, err 0x103", I do not know how much is that relating to this topic.

@yoelrc88, have you had any history activity in this topic?

debug_two_device_connection_02.pdf

[panafana]

Hi @rob-bits im also interested in working on this. Currently studying the A2DP standard if i find anything i will contact you

[rob-bits]

Hi @panafana

Thank you very much for your comment. Well I spent +20 hours to work out a proof-of-concept example with esp-idf for this application. But I gained no success. I could not connect two device in the same time and so I cannot test the multiple streaming.

The thing is the the bluedroid stack is more complicated as I can undenstand it and follow all of its layers. The current stack does not support any multi connection for the AVDTP or handling more SEPs as the application requires. So it requires too much work. And I did not get any support from the dev team so it is a dead end for me.

BUT, I found the btstack from Bluekitchen which is compatible with the ESP32. It is more transparent for me and more low level what I like. It already supports the multi connection. And I have already reach more success. I can connect two BT speaker to the ESP32 and I can stream music, seperatly for both of them. With a software switch I can immediatly change the sink device. Now I have some issues when I am trying to stream to both of the devices, but I think I am not too far from the final goal.

And the dev team of bluekitchen is very helpful.

Anyway here is my open issue at the btstack repository:
bluekitchen/btstack#143

If you made any success in this topic feel free to share it. And I accept and acknowledge any help or suggestion in this topic.

Cheers,

Rob

[panafana]

@rob-bits could you maybe share some code of your project so i can get up to speed faster?

[rob-bits]

@panafana Which one would you like to have? The ESP-IDF or the btstack related code?

[panafana]

@rob-bits the btstack but is it java for android or is it in C?

[rob-bits]

@panafana It is in c. I am on it. If I uploaded the code I will update this comment.

UPDATE:
So I uploaded my code into github. Please check the follow example project:

https://github.com/rob-bits/btstack-master/tree/test_branch/port/esp32/example/my_a2dp_source

It is really a proof-of-concept code. It is not prepared to shared with anyone so there can be some debug related informations what are useful for me but maybe it is nosense for others.. So use it as it is. I am open to any discussion about this topic. It is tested on the ESP32.

In the code you have to modify the following two lines:

in my_a2dp_source.c:
187 static const char * device_addr_string_0 = "fc:58:fa:49:20:f1";  //PPA11BT
188 static const char * device_addr_string_1 = "00:58:02:cc:02:64";  //BTS-06

In here you have to write the addresses of the BT audio speakers. Currently the automatic serach and identification by its name is not implemented. And for the proo-of-concept it is not needed. So you will need your devices bd addresses.

There are two "switches" in the application. One is the idx_handler.config the other is the idx_handler.stream. The first one is used during the initialization of the connection of a BT speaker. When its value is 0 then it handling the first device connection, when its value is 1 then it is handling the 2nd device. Currently it can be changed manually from terminal with the '0' and '1' charachters.
The second "switch" the idx_handler.stream handles which device should play the music. If both of the devices are connected then this variable value will define which speaker is playing the music. If 0 is its value then the 1st speaker is playing, if its value is 1 then the second. This can be changed manually through from terminal with '2' and '3' characters.

If you want to test the pllication please follow the following scenario:

When the esp32 powered on and it is in idle then, and terminal is opened:
press D //this disables the automatic connection from sink device
press 0 //this selects the first devices for initialization
press 2 //this selects the first device for stream
now turn on the first device
if it is powered on and in idle state then:
press b //start the pair/connection process
wait for the end of the pair/connect process. If succeed then it should play the music.
Now press T //this stops the timer which feeds the data to the idx_handler.stream device. So the audio play is suspended for the first device
now press 1 //select the second device for configuration
press 3 //select the second device for stream
turn on the second device
if the second device is powered on and in idle state then:
press b //start the pair/connect process
wait for the end of the pair/connect process. If succeed then it should play the music.
press 2 //change the destination for the stream
now the music should be stopped because the 1st device is selected for stream
press t to start the music on device 1
now if you press the 3 the second device should play the music, if you press 2 again then the first should play

Thats all. I hope you can test it as I.
The SD card playback is not working yet. Currently it playes some music from memory.

Rob

[panafana]

@rob-bits Awesome man!
My main goal is to try and port this to be used in phones somehow (dont know if there is any OS restrictions in Android that prevent it).
If i am unsuccessful i will buy the bluetooth module for my arduino and work on that.
Again many thanks!

[rob-bits]

@panafana Okay. If you reach any success keep me updated then :)

[eric-gitta-moore]

Hello, how do you implement the connection to the Bluetooth headset? I use ESP32 to connect to the Bluetooth headset and always stuck in "esp_a2d_source_connect". I just modified the official example a2dp_source. But connecting a computer can be, that is, connecting a Bluetooth headset does not work. Sometimes it suddenly crashes and then restarts and suddenly connects.

[pad52]

Any updates in this?

[jack0c]

This will not be implemented in any Espressif chips, we will develop BLE audio chips to replace this. Chips after ESP32C6 will support BLE audio(BLE5.2).

[ethernetz]

Are there any esp32 devices out right now that support BLE audio? If so, will those devices allow me to stream audio to multiple speakers?

[jack0c]

Sorry, the BLE audio chip is not ready, still in R&D stage.

[jack0c]

Next generation chips will support:

1. ESP32H4, BLE audio
2. ESP32S6, BLE audio + Classic BT TWS solution(A2DP source+2 A2DP sink)

Both of which will be taped out in 2024.