cmake: introduce snippets for common configuration

[JordanYates]

Introduces the concept of snippet's to application configuration. A snippet is a piece of functionality that is not the default behaviour of a board, but is a common configuration that users may wish to apply.

One example snippet concept is "usb" which may route the console to USB instead of a serial port. Another common snippet could be "mcuboot", which would define the flash partitions required for applications to be loaded by a bootloader.

The name "snippet" was chosen mainly to avoid collisions with existing terms. Alternatives considered include:

• overlay (collision with devicetree overlay concept)
• config (collision with Kconfig)
• fragment (collision with .conf files [Kconfig fragments])
• part (collision with --pristine build option, flash partitions)
• layout (implies memory configuration)

The Oxford dictionary definition of snippet appears to match this concept well, "a small piece or brief extract".

Snippets are a required devicetree .overlay file and an optional .conf file that are appended to the existing CMake variables DTC_OVERLAY_FILE and CONF_FILE. The initial search paths for these files are ${APPLICATION_SOURCE_DIR}/${BOARD}/snippets and ${BOARD_DIR}/snippets, with the application directory having precedence to allow for overriding board defaults.

This is a first step towards #34167, as it enables generating zephyr.dts files with only the partitions required by an application.
Related to the Zephyr developer summit talk "Common Application Configuration for boards".

Please keep bike-shedding about the name "snippet" to a minimum, I fully expect this to go to "dev-review" which would decide the name anyway 😄.

[JordanYates]

CC @carlescufi @tejlmand @mrrosen

[carlescufi]

Is this related or can address what is trying to be accomplished here by @jfischer-no?

[carlocaione]

I choose to ignore the bike-shedding suggestion just to point out that yocto calls these configuration fragments ;)

[tejlmand]

Thanks for one more idea.

Some initial comments on the approach, mostly based on the commit message description:

One example snippet concept is "usb" which may route the console to USB
instead of a serial port.

This is so far the most valid use-case we have.

Another common snippet could be "mcuboot",
which would define the flash partitions required for applications to
be loaded by a bootloader.

I believe the mcuboot use-case should be addressed with the multi image support that keeps popping up.
There is finally a RFC trying to address at a higher level than building the Zephyr sample build infrastructure, #40555.

The name "snippet" was chosen mainly to avoid collisions with existing
terms.

Like linker snippets 😆

zephyr/cmake/extensions.cmake

Lines 1178 to 1184 in 6ee8182

	set(sections_path "${snippet_base}/snippets-sections.ld")
	set(ram_sections_path "${snippet_base}/snippets-ram-sections.ld")
	set(data_sections_path "${snippet_base}/snippets-data-sections.ld")
	set(rom_start_path "${snippet_base}/snippets-rom-start.ld")
	set(noinit_path "${snippet_base}/snippets-noinit.ld")
	set(rwdata_path "${snippet_base}/snippets-rwdata.ld")
	set(rodata_path "${snippet_base}/snippets-rodata.ld")

Guess we will collide with something regardless of the term we choose.
That said, snippet is not bad.

Snippets are a required devicetree .overlay file and an optional
.conf file that are appended to the existing CMake variables
DTC_OVERLAY_FILE and CONF_FILE.

Why is the dtc overlay required, and the Kconfig optional.
I see no reasoning for this decision, so could you please elaborate on this.

The initial search paths for these
files are ${APPLICATION_SOURCE_DIR}/${BOARD}/snippets and
${BOARD_DIR}/snippets, with the application directory having
precedence to allow for overriding board defaults.

But what is the interaction with sample specific overlay ?
How is it known if a snippet is valid to apply in a given sample.

With this PR, then a board and a sample having:

boards/<arch>/my_board/snippets/awesome.overlay
boards/<arch>/my_board/snippets/awesome.conf

samples/cool_stuff/app.overlay

or
samples/cool_stuff/boards/my_board.overlay
samples/cool_stuff/boards/my_board.conf

I know that cool_stuff works with the regular board configuration, but how will a user know whether awesome snippet is working or has been tested with cool_stuff ?
And how is it intended to be ensured that awesome snippet works thoughout the build system. (Testing all samples for this board with this snippet 😮 )

Current solution, where a awesome.conf / awesome.overlay resides in a sample then I know that sample is compatible with the settings in awesome.conf / awesome.overlay.
Or if the sample has app.overlay, then I know it works with that.

I agree that the usb use-case is very generic, and should not cause trouble, but what is the threshold for when something can be made a snippet ?
And is usb really our only good use-case on this.

[tejlmand]

Two comments on this:

• Although -s has been deprecated for some time, then I think it's good practice to have a period between removing the argument before using that same argument for something new.
• Why should snippet have it's own west build argument when CONF_FILE, DTC_OVERLAY_FILE, SHIELD etc. doesn't have ?

[JordanYates]

• Although -s has been deprecated for some time, then I think it's good practice to have a period between removing the argument before using that same argument for something new.

In an ideal world yes, but the end result is going to be the same. If -s is removed, any scripts using it will crash. If -s is reused, any scripts using it will crash. Unfortunately I can't go back and remove it a year ago. Of course if the name changes from "snippet" then there is no need to use -s.

• Why should snippet have it's own west build argument when CONF_FILE, DTC_OVERLAY_FILE, SHIELD etc. doesn't have?

Because its goal is to cover common use cases, which IMO means the ease-of-use of west build -b board -s usb samples/basic/minimal over west build -b board samples/basic/minimal -- -DSNIPPETS="usb" warrants it.

[tejlmand]

I would say -DOVERLAY_CONFIG or DTC_OVERLAY_FILE are much better candidates for having extra arguments than snippets.

[mbolivar-nordic]

I disagree; I think snippets do rise to the level of deserving their own option.

I want to ask what the practical differences between snippets and shields are at this point, in terms of their behavior. Shields seem to be a special case of snippets, morally speaking.

Taking @petejohanson since he's taken an interest in adding more comprehensive shield related options to west build elsewhere.

[tejlmand]

I don't think we should do this for mcuboot.

If we should do anything related to partition layout when mcuboot is enabled, then we should do this on higher level where we build both the sample and mcuboot, aka as multi image, see this RFC: #40555

[Laczen]

@tejlmand, exactly how is RFC #40555 going to handle the problem that is addressed by the present PR. What is done here is overruling the definition that is given in the dts for the flash layout. More specific what is going to be the solution in the multi-image PR to support:
a. A non-mcuboot image,
b. A mcuboot image with only the slot0 image generated,
c. A mcuboot image with mcuboot + slot0 image generated,
(d. A mcuboot + slot1 image (encrypted) generated),

[tejlmand]

@tejlmand, exactly how is RFC #40555 going to handle the problem that is addressed by the present PR. What is done here is overruling the definition that is given in the dts for the flash layout. More specific what is going to be the solution in the multi-image PR to support:

For this PR, I only see a single mcuboot, so that must be a (being the case without this PR) and b.

Not c and d.

Regarding #40555, the topic you raise is outside the specific scope of that PR, but it opens up for a broader use-case, so let me sketch the general idea.

With #40555 it is possible to build the sample with or without mcuboot (or any other multi image, like the openamp_remote case).

In such a scenario, which would also be preparing for system devicetree, we can then apply a given set of overlay when a given bootloader is used.

If mcuboot is enabled when building, then we can ensure a devicetree setup with a flash layout suitable for mcuboot is selected and applied to all images part of the build.
This will further avoid the risk of the application being build with one snippet and mcuboot with another because you forgot to apply the same settings there.

The #40555 itself doesn't address this possibility, because it's just an RFC and I have tried to find the balance between amount of time spent to present the idea, while not yet spend too much time if the direction is rejected.

[carlescufi]

I choose to ignore the bike-shedding suggestion just to point out that yocto calls these configuration fragments ;)

That's terrible given that .conf files are called "Kconfig fragments". Very confusing.

[JordanYates]

Another common snippet could be "mcuboot",
which would define the flash partitions required for applications to
be loaded by a bootloader.

I believe the mcuboot use-case should be addressed with the multi image support that keeps popping up.
There is finally a RFC trying to address at a higher level than building the Zephyr sample build infrastructure, #40555.

This is not intending to perform a multi-image build, or even multi-image support. It is aimed at the one outcome from the "Common application configuration for boards" talk that I thought was broadly agreed on. Namely that zephyr.dts should only contain information that is valid for that particular build. There is nothing stopping #40555 from setting the "mcuboot" snippet to pull in the correct flash partitions and Kconfig settings.

The name "snippet" was chosen mainly to avoid collisions with existing
terms.

Like linker snippets 😆

Whoops, you deserve credit for the name, I forgot that is where I heard it.

Why is the dtc overlay required, and the Kconfig optional.
I see no reasoning for this decision, so could you please elaborate on this.

Because my primary use case was for applying devicetree overlays (fixing flash partitions and chosen nodes). The Kconfig file can easily be made required if that is the consensus view.

I know that cool_stuff works with the regular board configuration, but how will a user know whether awesome snippet is working or has been tested with cool_stuff ? And how is it intended to be ensured that awesome snippet works thoughout the build system. (Testing all samples for this board with this snippet 😮 )

Because at least initially I see the scope for snippets to be functionality that is not specific to any particular application. If the mcuboot snippet is tested for a board once anywhere, you can be quite confident that it will work on any sample/application. I don't consider it much different from the current setup, how do I know that samples/basic/servo_motor will work on anything other than a bbc_microbit? We can't guarantee everything. Test all board level snippets against samples/basic/minimal to at least ensure they compile, and test board specific snippets in sample folders (not that I see much use for that in-tree).

And is usb really our only good use-case on this.

No. As I've said many times in many places, there are problems that can only be solved cleanly if zephyr.dts is minimal and correct. This is a mechanism to provide that, which is why I believe mcuboot should be a snippet for every board that supports it. If that and USB are the only two in-tree uses, I still think it is worthwhile.

[tejlmand]

This is not intending to perform a multi-image build, or even multi-image support. It is aimed at the one outcome from the "Common application configuration for boards" talk that I thought was broadly agreed on. Namely that zephyr.dts should only contain information that is valid for that particular build. There is nothing stopping #40555 from setting the "mcuboot" snippet to pull in the correct flash partitions and Kconfig settings.

I know, but in #40555 I try to avoid create two different workflows for the user when using west build.
What I mean with that, is that if you do west build ... -- -DCONFIG_SOMETHING=y today, you still do the same west build after that PR.
And you can opt-in to include mcuboot in that west build with west build ... -- -DCONFIG_SOMETHING=y -DCONFIG_WB_BOOTLOADER_MCUBOOT=y.
And as you see, you are still allowed to use the same flags as always.

If introducing a new snippet workflow including the mcuboot snippet, then I see some risks in using that -DSNIPPETS=mcuboot snippet workflow colliding with the west build mcuboot inclusion.
Let's say #40555 starts applying the same snippet, for example: west build ... -- -DSNIPPETS=mcuboot -DCONFIG_WB_BOOTLOADER_MCUBOOT=y then how should that be treated ?

Therefore I would prefer not to try addressing / creating a new workflow for the MCUboot use-case just now.

I don't consider it much different from the current setup, how do I know that samples/basic/servo_motor will work on anything other than a bbc_microbit? We can't guarantee everything.

No, but the difference here is that for a sample like servo_motor, we do guarantee that it works for that board:

zephyr/samples/basic/servo_motor/sample.yaml

Line 8 in 102ac1a

platform_allow: bbc_microbit

and similar in the docs: https://github.com/zephyrproject-rtos/zephyr/blob/main/samples/basic/servo_motor/README.rst

Whereas as soon as we start placing configs in a common location we must be much clearer on what users may expect.

I do notice that your snippets follows the board, which helps a lot on the concern regarding just a single common location, but it also means a user may not know if -s mcuboot will work (but should find out pretty fast).

mcuboot and usb are good cases.
My concern is more regarding later, when someone wants to have some-snippet merged.
What will be the threshold for new snippets ?

The snippets doesn't state what samples they are guaranteed to work with, and samples doesn't stated which snippets are supported.

Why is the dtc overlay required, and the Kconfig optional.
I see no reasoning for this decision, so could you please elaborate on this.

Because my primary use case was for applying devicetree overlays (fixing flash partitions and chosen nodes). The Kconfig file can easily be made required if that is the consensus view.

I was just curious, cause if devicetree overlay was all you needed, then another option could be to allow DTC_OVERLAY_FILE to lookup relative to board dir, for example DTC_OVERLAY_FILE=overlay/usb.overlay.

And for the application dir we already have the DTC_OVERLAY_FILE and CONF_FILE / OVERLAY_CONFIG if only needing to apply one without the other.

Therefore I was wondering if it would make more sense that a new term like snippets would mean a set with a devicetree overlay + a Kconfig fragment.
We already have a term when applying one of them without the other.

[mbolivar-nordic]

I like this. It's small and seems to address exactly the scope we debated at length after last year's Zephyr Developers Summit. I have a couple of quibbles here and there but I'm generally +1.

[mbolivar-nordic]

I totally get what you're doing here, and although I use this option every day, I would be willing to part ways with it in the name of making progress on this more important issue (assuming "snippet" is chosen).

But there are a few places in the docs that are using still:

doc/guides/west/sign.rst
29:   west build -b YOUR_BOARD -s bootloader/mcuboot/boot/zephyr -d build-mcuboot
30:   west build -b YOUR_BOARD -s zephyr/samples/hello_world -d build-hello-signed -- \

doc/guides/dts/howtos.rst
35:   west build -b qemu_cortex_m3 -s samples/hello_world --cmake-only

Those would need cleanup as well, pending resolution of the bikeshed debate.

[mbolivar-nordic]

I disagree; I think snippets do rise to the level of deserving their own option.

I want to ask what the practical differences between snippets and shields are at this point, in terms of their behavior. Shields seem to be a special case of snippets, morally speaking.

Taking @petejohanson since he's taken an interest in adding more comprehensive shield related options to west build elsewhere.

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

[tejlmand]

@JordanYates @mbolivar-nordic @jfischer-no I have done some thinking on this.

While I believe the snippets concept proposed in this PR makes a lot of sense, then I did some thinking related to the usb use-case.

We have board revisions, and a while back we discussed generalizing this into board variants (on slack as I recall it), where revision is just a specialized kind of a variant.

Now, in addition to a board providing custom variants, Zephyr can provide common variants, like a cdc_acm.
That may allow users to build the same board for a cdc_acm variant, like this:
nrf52840dk_nrf52840 or nrf52840dk_nrf52840@cdc_acm

Just like boards can default to a specific revision, for example nrf9160dk_nrf9160 --> nrf9160dk_nrf9160@0.7.0
Then a board which requires cdc_acm always, for example: ble654_usb, then building for bl654_usb is identical to building bl654_usb@cdc_acm.

And just like with revisions, it can be made visible (and testable by the build system) whether a given variant is supported, similar to what is done for the revision:

zephyr/boards/arm/nrf9160dk_nrf9160/revision.cmake

Lines 6 to 7 in 22c0843

	DEFAULT_REVISION 0.7.0
	VALID_REVISIONS 0.7.0 0.14.0

This further allow users to re-use existing infrastructure where Kconfig snippets and DT overlay may be applied automatically depending on board, variant, (and revision).

I don't see variant support in conflict with snippets, they both serve a purpose.
But the variant will allow to formalize and re-use variants between boards and make it more visible to users what variant type a given board supports.

[jfischer-no]

This neither makes sense nor is it helpful to provide a generic board-wide (or tree-wide) configuration, as it is possible with our shield support, see #40645.

Different configuration on a board level are already possible with board version. On application level DT and *.conf overlays are supported. The example given regarding USB has neither sense nor necessity, worse, it leads some to think that this can be alternative-to / help-with #40645.

Compared to current shield support (even if shield is hardware configuration related) snippets here do not provide support for overall configuration, Kconfig files are not supported, no documentation, no documentation support.

I probably do not quite understand the motivation for these changes, for my part related to #40645, I suggest to rename shield to something more generic and derive from it shield for hardware and foobar for specific configuration.

[tejlmand]

This neither makes sense nor is it helpful to provide a generic board-wide (or tree-wide) configuration, as it is possible with our shield support, see #40645.

@jfischer-no Maybe you haven't noticed, but Zephyr already has different kind of board variants, both formal and informal.

Formal variant is revision.

Informal variant are those used my multi-core SoCs and secure / non-secure targets (cryptocell).

For example, the nRF5340dk_nrf5340 board comes in following variants:

• nrf5340dk_nrf5340_netcore: variant identifying network core in the SoC
• nrf5340dk_nrf5340_appcore: variant identifying application core in the SoC
• nrf5340dk_nrf5340_appcore_ns: configuration variant identifying anon-secure application in the appcore in the SoC.

Now, especially ns is of interest, as it is a configuration variant, but we don't identify it using shield principle.
In which way is a cdc_acm configuration variant of a board significantly different from a ns configuration variant, or other variants mentioned above ?

This is the reason why we should formalize board variant handling in Zephyr, and have revision a specialized type of variant.
And having common baseline variant names ensures the board itself will be able to present if a given variant is supported (something the shield mechanism can never provide)

Other boards with informal variants (non-exhaustive list): lpcxpresso54114, nrf9160dk_nrf9160, mps2_an521

[jfischer-no]

This neither makes sense nor is it helpful to provide a generic board-wide (or tree-wide) configuration, as it is possible with our shield support, see #40645.

@jfischer-no Maybe you haven't noticed, but Zephyr already has different kind of board variants, both formal and informal.

Formal variant is revision.

Informal variant are those used my multi-core SoCs and secure / non-secure targets (cryptocell).

For example, the nRF5340dk_nrf5340 board comes in following variants:

* nrf5340dk_nrf5340_netcore: variant identifying network core in the SoC

* nrf5340dk_nrf5340_appcore: variant identifying application core in the SoC

* nrf5340dk_nrf5340_appcore_ns: configuration variant identifying anon-secure application in the appcore in the SoC.

Now, especially ns is of interest, as it is a configuration variant, but we don't identify it using shield principle. In which way is a cdc_acm configuration variant of a board significantly different from a ns configuration variant, or other variants mentioned above ?

Related to configuration and devicetree, _ns is name agreement only, it does not provide common configuration for all boards, it depends on SoC, and in that case boards have same fragments, see nrf5340dk and bl5340_dvk for example. Of course one could add absolutely the same configuration and devicetree description for all boards that have USB device support and want to use CDC ACM as backend, but that would be stupid, and not maintenance friendly, and also not solve all problems, because it is reasonable to have separate backends for logging and shell, and another one maybe for something else. In contrast, the solution based on shield support like #40645 is omnipotent, it is combinable and works for all boards that have USB device support, whether nrf52840donge or nrf52840dk, downsream and upstream.

This is the reason why we should formalize board variant handling in Zephyr, and have revision a specialized type of variant. And having common baseline variant names ensures the board itself will be able to present if a given variant is supported (something the shield mechanism can never provide)

Other boards with informal variants (non-exhaustive list): lpcxpresso54114, nrf9160dk_nrf9160, mps2_an521

[mbolivar-nordic]

I probably do not quite understand the motivation for these changes, for my part related to #40645, I suggest to rename shield to something more generic and derive from it shield for hardware and foobar for specific configuration.

@jfischer-no I think we may all be in violent agreement here. In particular, I think that we should make sure that "snippets" have the features from "shields" that you require for #40645. Assuming that is fine for you, maybe we can proceed with that plan?

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

[mbolivar-nordic]

@jfischer-no I think we may all be in violent agreement here. In particular, I think that we should make sure that "snippets" have the features from "shields" that you require for #40645. Assuming that is fine for you, maybe we can proceed with that plan?

ping

[JordanYates]

This has been updated for v3.1 with some minor changes since I've been using snippets in my fork for a few months.
The important change is that snippets now allow an optional .cmake file to be run as well. This allows the old "separate board for nonsecure" paradigm to be removed in favour of a trivial snippet.

nonsecure.cmake

list(APPEND DTS_EXTRA_CPPFLAGS "-DARM_TRUSTZONE_NON_SECURE")

nonsecure.conf

# This snippet implies building Non-Secure firmware
CONFIG_TRUSTED_EXECUTION_NONSECURE=y

nonsecure.overlay

/ {
	chosen {
		zephyr,code-partition = &nonsecure_partition;
		zephyr,sram = &sram0_ns;
	};
}

thingy91.dtsi

#ifdef ARM_TRUSTZONE_NON_SECURE
#include <nordic/nrf9160ns_sica.dtsi>
#else
#include <nordic/nrf9160_sica.dtsi>
#endif /* ARM_TRUSTZONE_NON_SECURE */

That is sufficient to build nonsecure applications just by adding -s nonsecure, without any of the _ns files currently in the tree.

[gmarull]

In general, I like the idea, and not against the name, so +1 from my side.

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.