Bluetooth: Convert HCI drivers to use a proper Zephyr driver API

[jhedberg]

Current status:

• Create a base binding and API definition
• Update the host to support new HCI driver instances
• Update h4.c
• Update native controller HCI driver
• Fix samples/bluetooth/hci_uart_async/ to work with the new driver model
• Fix bsim to work both with native and UART-based controllers (right now CI is failing)
• Figure out what to do with bt_read_static_addr(). Taken care of by Bluetooth: Simplify Kconfig options and interface between the host and controller #72580
• Figure out what to do with BT_HAS_HCI_VS, BT_HCI_VS and BT_HCI_VS_EXT (perhaps DTS properties?). Some preparation in Bluetooth: Simplify Kconfig options and interface between the host and controller #72580.
• Fix controller unit tests: no devicetree available for unit_testing "board" right now: Bluetooth: controller: Add BT_CTLR_HCI Kconfig option #72871
• Update ipc.c
• Update userchan.c
• Update h5.c
• Update hci_ambiq.c (and associated DT binding)
• Remove hci_b91.c: Bluetooth: drivers: Remove unmaintained B91 HCI driver #73289
• Update hci_esp32.c
• Update hci_psoc6_bless.c (and associated DT binding)
• Update hci_spi_st.c (and associated DT binding)
• Update hci_stm32wba.c
• Update ipm_stm32wb.c (and associated DT binding)
• Update spi.c
• Update hci_da1469x.c
• Update hci_nxp.c
• Update slz_hci.c
• Update documentation, especially stuff regarding /chosen properties
• Make it possible for nrf52_bsim to use also UART for HCI
• Update various apps under samples/ and tests/ that use the legacy HCI driver API
• Get rid of the Kconfig choice section for HCI drivers
• Get rid of BT_NO_DRIVER
• Deprecate the old API
• Lots of testing
  • bsim with virtual driver (hci_driver.c)
  • ipc.c tested in CI
  • Test on hardware with virtual bus (i.e. combined build)
  • Test on hardware with IPC (i.e. split build)
  • Test on hardware with UART (i.e. split build)
  • userchan.c
  • Test with an OoT driver (e.g. NCS)

Right now the host hci_core.c uses #if DT_HAS_CHOSEN(zephyr_bt_hci) to separate the new-style HCI drivers from the legacy HCI driver support. The old API has been deprecated, but will only be removed after a deprecation period.

The following drivers depend on "west blobs", so they're not covered by CI, but were manually compile tested:

• hci_psoc6_bless.c (west blobs fetch hal_infineon)
• hci_esp32.c (west blobs fetch hal_espressif)
• hci_stm32wba.c (west blobs fetch hal_stm32 doesn't work for me, see Bluetooth: Convert HCI drivers to use a proper Zephyr driver API #72323 (comment) )
• slz_hci.c (west blobs fetch hal_silabs, same problem for me as with stm32, i.e. had to manually download)

What this PR doesn't try to tackle:

• Find a solution for and fix driver dependencies on host HCI command sending APIs, which are e.g. not available with HCI_RAW.

  zephyr/drivers/bluetooth/hci/apollox_blue.c

  Lines 289 to 333 in d3c0dfb

  	#if defined(CONFIG_BT_HCI_RAW)
  	struct bt_hci_cmd_hdr hdr;
  	hdr.opcode = sys_cpu_to_le16(HCI_VSC_UPDATE_NVDS_CFG_CMD_OPCODE);
  	hdr.param_len = HCI_VSC_UPDATE_NVDS_CFG_CMD_LENGTH;
  	buf = bt_buf_get_tx(BT_BUF_CMD, K_NO_WAIT, &hdr, sizeof(hdr));
  	if (!buf) {
  	return -ENOBUFS;
  	}
  	net_buf_add_mem(buf, &am_devices_cooper_nvds[0], HCI_VSC_UPDATE_NVDS_CFG_CMD_LENGTH);
  	ret = bt_send(buf);
  	if (!ret) {
  	/* Give some time to make NVDS take effect in BLE controller */
  	k_sleep(K_MSEC(5));
  	/* Need to send reset command to make the NVDS take effect */
  	hdr.opcode = sys_cpu_to_le16(BT_HCI_OP_RESET);
  	hdr.param_len = 0;
  	buf = bt_buf_get_tx(BT_BUF_CMD, K_NO_WAIT, &hdr, sizeof(hdr));
  	if (!buf) {
  	return -ENOBUFS;
  	}
  	ret = bt_send(buf);
  	}
  	#else
  	uint8_t *p;
  	buf = bt_hci_cmd_create(HCI_VSC_UPDATE_NVDS_CFG_CMD_OPCODE,
  	HCI_VSC_UPDATE_NVDS_CFG_CMD_LENGTH);
  	if (!buf) {
  	return -ENOBUFS;
  	}
  	p = net_buf_add(buf, HCI_VSC_UPDATE_NVDS_CFG_CMD_LENGTH);
  	memcpy(p, &am_devices_cooper_nvds[0], HCI_VSC_UPDATE_NVDS_CFG_CMD_LENGTH);
  	ret = bt_hci_cmd_send_sync(HCI_VSC_UPDATE_NVDS_CFG_CMD_OPCODE, buf, NULL);
  	if (!ret) {
  	/* Give some time to make NVDS take effect in BLE controller */
  	k_sleep(K_MSEC(5));
  	}
  	#endif /* defined(CONFIG_BT_HCI_RAW) */

• Decide what to do with BT_HCI_ACL_FLOW_CONTROL currently referencing specific HCI drivers
• net_buf allocations by drivers. Drivers should own their own memory rather than calling into the host side to do these allocations.
• Making the HCI driver API support system calls. This is related to the above first point. We also can't have the driver make direct calls using the recv callback to the host, rather there needs to be another mechanism for the host to fetch data from the driver.

Fixes #51503

[jhedberg]

@jerome-pouiller @Thalley could you take another look? Due to the size of this PR it'd be better to get it in sooner rather than later, especially since the feature freeze is coming this week. Since there doesn't seem to be any fundamental objections to the architectural/API change introduced here, any follow-up fixes & cleanups could happen after the feature freeze as well. There are a couple of other open HCI driver PRs which conflict with this one, which also target the next release, so it'd be good to start merging these sooner rather than later so we get the conflicts resolved and everything merged on time.

[Thalley]

@jerome-pouiller @Thalley could you take another look? Due to the size of this PR it'd be better to get it in sooner rather than later, especially since the feature freeze is coming this week. Since there doesn't seem to be any fundamental objections to the architectural/API change introduced here, any follow-up fixes & cleanups could happen after the feature freeze as well. There are a couple of other open HCI driver PRs which conflict with this one, which also target the next release, so it'd be good to start merging these sooner rather than later so we get the conflicts resolved and everything merged on time.

Indeed, and the number of commits easily hide comments on Github (...).

I'd be OK with merging as is, but I would have to have at least #72323 (comment) resolved ASAP so that we can do things properly (which I believe was the main point of this change in the first place ;) )

stale

[carlescufi]

@rugeGerritsen @aescolar @yeaissa @erwango @npal-cy @alwa-nordic please take a look ASAP, we would like to include this in Zephyr 3.7.0, which means merging it before this Friday

[jerome-pouiller]

Beside my comment about efr32xg13. This PR is fine for me.