Scaling of board overlay mechanism

[PerMac]

Is your enhancement proposal related to a problem? Please describe.
There are thousands of overlay files located under sample/* and tests/* ... */boards/. E.g. https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/drivers/uart/uart_async_api/boards (142 files)
https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/drivers/dma/chan_blen_transfer/boards (173 files).
Every time someone wants to add a support of such tests on another board they have to add extra board specific overlays. In many cases such overlays are yet another copy of the same file, just with a different name.
Such mechanism doesn't scale.

Additional context
We discussed the toping during testing wg meetings and we don't know how it can be changed. Maybe there won't be a single solution, but a combination of several, including rework of some tests. One approach is proposed by @hakehuang which allows for reuse of the same overlay files by multiple boards, at the cost of adding entries in test yaml responsible for mapping such boards with more general overlays https://github.com/zephyrproject-rtos/zephyr/blob/main/tests/drivers/uart/uart_async_api/testcase.yaml#L87. However, it can significantly increase complexity of test yaml resulting with tests with hundreds of namespaced configs.
Other idea is to move some implementation of overlays to a board/soc/variant/more general level, as a kind of an extension, that could be added (and reused) whenever such overlay is requested by tests?
We might also need to rethink of how overlays are applied, possibly using SoC overlays or feature-based overlays instead of board-specific ones.

As a testing working group we wanted to flag the issue of "exploding" overlays and ask other maintainers for their take on it and possible solutions.

[PerMac]

cc @nashif @hakehuang

[JarmouniA]

We might also need to rethink of how overlays are applied, possibly using SoC overlays

This is a good solution for most STM32 boards.

[hakehuang]

This is a good solution for most STM32 boards.

the overlay usually have 3 type of contents:

1. pin mux for testcases which is board specific
2. clock setting which is also board specific
3. IP settings for DUT, which is IP based.

so I do not think the soc is a good place to go, instead if you looking above, the good place will be

item 1 and 2 goes to board
item 3 goes to vender under different IP.

[nashif]

This issue is very meta and involves every aspect of zephyr I suspect:

• Build System
• DTS
• Kconfig
• Testing
• Drivers
• Boards/SoCs

Some of the tests/samples with the most overlays used to deal with different HW directly in the test code using ifdefs (we have some of those tests still), at some point we moved to overlays and removed some of the complexity from the tests, however, IMO this was just a stopgap and we move the problem somewhere else. Looking at many of those overlays you notice lots of duplicatiion and some patterns that we should resolve and fix somewhere else I think so we do not have to add an overlay for each board in each test.

Beside DTS, there is Kconfig part which is suffers against the same problem.

Soc overlays would reduce the number of files if the number of files was the problem, however we need to look at further abstraction or some common "language" when dealing with drivers and peripherals without having to provide a DTS/Kconfig overlay for every single other board.
Another option would be to allow board to define snippets that can be reused by many tests, so we do not need to provide an overlay for each tests/sample.

More thoughts/input would be appreciated, need more people looking at this.

[bjarki-andreasen]

For the comparator build_all tests, I created multiple "soc specific" overlays and included them for a range of boards which contain the socs I care about:

zephyr/tests/drivers/build_all/comparator/testcase.yaml

Lines 17 to 52 in 11c543e

	drivers.build_all.comparator.nrf_comp.diff:
	extra_args:
	- DTC_OVERLAY_FILE="nrf_comp/diff.overlay"
	platform_allow:
	- nrf52dk/nrf52810
	- nrf52dk/nrf52832
	- nrf52833dk/nrf52820
	- nrf52833dk/nrf52833
	- nrf52840dk/nrf52811
	- nrf52840dk/nrf52840
	- nrf5340dk/nrf5340/cpuapp
	- nrf54h20dk/nrf54h20/cpuapp
	- nrf54l15dk/nrf54l15/cpuapp
	- ophelia4ev/nrf54l15/cpuapp
	- nrf9280pdk/nrf9280/cpuapp
	integration_platforms:
	- nrf52dk/nrf52810
	- nrf5340dk/nrf5340/cpuapp
	drivers.build_all.comparator.nrf_comp.se_aref:
	extra_args:
	- DTC_OVERLAY_FILE="nrf_comp/se_aref.overlay"
	platform_allow:
	- nrf52dk/nrf52810
	- nrf52dk/nrf52832
	- nrf52833dk/nrf52820
	- nrf52833dk/nrf52833
	- nrf52840dk/nrf52811
	- nrf52840dk/nrf52840
	- nrf5340dk/nrf5340/cpuapp
	- nrf54h20dk/nrf54h20/cpuapp
	- nrf54l15dk/nrf54l15/cpuapp
	- ophelia4ev/nrf54l15/cpuapp
	- nrf9280pdk/nrf9280/cpuapp
	integration_platforms:
	- nrf52dk/nrf52810
	- nrf5340dk/nrf5340/cpuapp

this could be extended to point to an "soc specific" CONF file as well. Its not the pretties thing, but its functional :) (ignore the duplicate nrf54l one, that was not intended)

[henrikbrixandersen]

the overlay usually have 3 type of contents:

1. pin mux for testcases which is board specific
2. clock setting which is also board specific
3. IP settings for DUT, which is IP based.

so I do not think the soc is a good place to go

Agreed.

instead if you looking above, the good place will be

item 1 and 2 goes to board item 3 goes to vender under different IP.

@hakehuang Could you please elaborate a bit on this suggestion?

[hakehuang]

@hakehuang Could you please elaborate a bit on this suggestion?

1. the pin mux change to a give test case usually is based on board design, so such chagnes shall bind with board not soc.
2. clock settings are both board and soc related, so such changes also bind with board and soc.
3. peripheral IP settings are soc based. but its system features such as interrupt number and dma channel are usually soc bind.
   So we can't move all application/test related settings to soc.
   My suggestion would be split them to 3 parts boad,soc,ip, and create an overlay sequences board over soc over ip.

[henrikbrixandersen]

@hakehuang Could you please elaborate a bit on this suggestion?

1. the pin mux change to a give test case usually is based on board design, so such chagnes shall bind with board not soc.
2. clock settings are both board and soc related, so such changes also bind with board and soc.
3. peripheral IP settings are soc based. but its system features such as interrupt number and dma channel are usually soc bind.
   So we can't move all application/test related settings to soc.
   My suggestion would be split them to 3 parts boad,soc,ip, and create an overlay sequences board over soc over ip.

How will this reduce the number of overlay files needed?

[hakehuang]

How will this reduce the number of overlay files needed?

take this commit as example,
513e45f
this is just partial down, original .overlay is change to per-IP combiantion. and with this, we clearly see what IP and which instance we are testing.

[hakehuang]

How will this reduce the number of overlay files needed?

take this commit as example,
513e45f
this is just partial done, original .overlay is change to per-IP combiantion. and with this, we clearly see what IP and which instance we are testing.

[ajf58]

We might also need to rethink of how overlays are applied, possibly using SoC overlays

This is a good solution for most STM32 boards.

This would also work for RP2040, and RP235xx SoCs.

It's also been proposed before, e.g. use some heirachy of directories, e.g.:

soc_series/foo.overlay
soc/foo12A.overlay

At the moment I misuse(?) the pre-processor, and #include one overlay from the others.

[nordic-segl]

@nashif wrote

Another option would be to allow board to define snippets that can be reused by many tests, so we do not need to provide an overlay for each tests/sample.

I did POC here (define snippet for test purposes) but it has several major drawbacks
nrfconnect/sdk-nrf#24853