Potential Tool Flow for Project Generation - Discussion

[ReinhardKeil]

This is revised content.

The diagram below shows a potential tool flow for project generation. It basically exports the information from the files described under #7

The CMSIS-Project Manager takes as input the following files:

Input Files	Used for....
DFP Software Packs	... device related information on the tool configuration. May refer an *.rzone file.
BSP Software Packs	... board specific configuration (i.e. debug). May refer to an *.rzone file that defines board components.
*.rzone files	... definition of memory and peripheral resources. If it does not exist, content is created from DFP.
*.ctarget.yml	... defines the complete scope of the application and the build order of sub-projects
*.cproject.yml	... defines the content of an independent build (linker run) - directly relates to a *.cprj file.
*.clayer.yml	... defines a set of pre-configured files and software components for reused in different targets.

The CMSIS-Project Manager has several operating modes:

• verify checks the consistency of input files. For example a *.ctarget.yml file may refer different DFP and a *.clayer.yml that specifies a target. Such inconsistencies are flagged (and perhaps overwrite operations are introduced).
• generate uses the above input files to generate the *.cprj files for CMSIS-Build. Optionally script templates can be used to generate linker scripts and Other Helper files.
• recreate uses one or more *.cprj files to re-create the *.yml input files.

The output of the CMSIS-Project Manager are self-contained *.cprj that refers linker scripts. These files can be copied to a different context (i.e. a CI test environment that generates a always fixed configuration).

We should introduce the notion of provided interfaces and consumed interfaces. This could be used to check for compatibility of projects, layers, and identify the build order of sub-projects. This is already in use here https://github.com/MDK-Packs/CB_Lab4Layer/blob/master/layer/Board/LPCXpresso55S69/Board.clayer#L11

The reason for this complexity is to support the use cases described here #6

Questions:

• Are proposed file extensions OK? Compatible with schema and *.yml editor plugins?
  A: https://github.com/SchemaStore/schemastore/blob/master/src/api/json/catalog.json would allow us to define schema information to the *.yml files. Proposed naming should be OK. This article explains how to configure a yaml schema: https://developers.redhat.com/blog/2020/11/25/how-to-configure-yaml-schema-to-make-editing-files-easier

• Is a BSP *.rzone file additive to a DFP *.rzone file. Does it add board specific resources or defines all resources including DFP again.

• More work is needed on provided interfaces and consumed interfaces in the current *.clayer files.

• Is there a need to review CMSIS-Zone in the wider group?

Next steps:

• Propose an initial format for schema files.
• Define scope of a MVP (minimum viable product) and start to prototype it.
• Review template engines: Freemarker works, but handlebars might be easier to use.

[fred-r]

Hi,

please find enclosed a document introducing some vocabulary and concepts we would like to discuss in the context of this project.

Thanks & Regards,
Multi-Context_Concepts_OpenCMSIS.pdf

[fred-r]

Follow Up here: #31

Hi,

minor question about the file format : why yaml ?
I know it is less important than the data itself but about YAML, I think:

• It is a superset of JSON supporting comments, but this is not key
• It is based on space characters for building a data hierarchy which is error-prone compared to the use of braces and brackets

Besides, with JSON we may validate our data thanks to JSON schema:
https://json-schema.org/

Benefits (taken from the JSON schema web site):
Describes your existing data format(s).
Provides clear human- and machine- readable documentation.
Validates data which is useful for:
Automated testing.
Ensuring quality of client submitted data.

I assume JSON schema can be used to validate YAML documents (or maybe YAML schema exists?) but to me YAML sounds like an add-on we might not need?

Also, I know JSONPath exists but I do not know if YAML has the same tooling available ?

To be discussed in a low priority I guess.

Thanks & Regards,

[fred-r]

Follow Up here: #32

Hi,

another low priority discussion : the template engine.

"Review template engines: Freemarker works, but handlebars might be easier to use."

I am not familiar with Freemarker but with handlebar it is quite "easy" to define your own helpers:
https://handlebarsjs.com/api-reference/helpers.html

It has also a factorization system:
https://handlebarsjs.com/guide/#partials

But the input object format accepted by handlebar might be an issue ?

To be discussed in a low priority I guess.

Thanks & Regards,

[fred-r]

Hi,

I would also like to understand better the concept of clayer.
To me if we want reusable components we have the CMSIS components to do so.
A component can be source code or a libray.

What is clayer bringing on top of a component with dependencies ?
It memorizes pre-resolved dependencies ?
Is that the point ?

Thanks & Regards,

[fred-r]

Hi,

the ctarget concept is not crystal clear to me.
Does it aggregate what we call an execution context and its execution domain ?

So, in a multi-context application project, would we have several ctarget files ?

Thanks & Regards,

[fred-r]

Hi,

regarding .rzone and DFP.
Why would rzone be out of the DFP ?

To me a DFP could provide an rzone (or the equivalent).
Legacy DFP packs would not provide this optional element and then we should accept additional rzone maybe.

Also, I think we need to consider in a central repository MCU and board resources (+ shields if any) : seems like the rzone does not really cover this ?

Thanks & Regards,

[jkrech]

I would also like to understand better the concept of clayer. To me if we want reusable components we have the CMSIS components to do so. A component can be source code or a libray.

What is clayer bringing on top of a component with dependencies ? It memorizes pre-resolved dependencies ? Is that the point ?

For more information about project layers (*.clayer) please read here:
CMSIS-Build
and to see it at work you may want to look at the Lab4Layer repository

Layers are the next level of abstraction between projects and components. The benefit of layers is that it is not only a list of components, but a combination of components that are already pre-configured for a range of use cases.

[jkrech]

regarding .rzone and DFP. Why would rzone be out of the DFP ?

To me a DFP could provide an rzone (or the equivalent). Legacy DFP packs would not provide this optional element and then we should accept additional rzone maybe.

Also, I think we need to consider in a central repository MCU and board resources (+ shields if any) : seems like the rzone does not really cover this ?

The DFP (Device Family Pack) contain device descriptions and ship device deliverables like SVD, Flash Programming Algorithms, Device Header, HAL, Device documentation, etc. Therefore it is a good location for a resource description file (rzone). But certainly we need to support this on a board level (Board Support Pack).
Conceptually it is important that all deliverables are contained and distributed by packs. Once the packs are installed the tools have all required meta information and files required for end to end application development.
New versions of packs will be able to add required meta information and files required by certain tools.

[fred-r]

I would also like to understand better the concept of clayer. To me if we want reusable components we have the CMSIS components to do so. A component can be source code or a libray.
What is clayer bringing on top of a component with dependencies ? It memorizes pre-resolved dependencies ? Is that the point ?

For more information about project layers (*.clayer) please read here: CMSIS-Build and to see it at work you may want to look at the Lab4Layer repository

Layers are the next level of abstraction between projects and components. The benefit of layers is that it is not only a list of components, but a combination of components that are already pre-configured for a range of use cases.

Hi Joachim,

I remember Lab4Layer as I played with it some time back.
But to me a clayer was nothing more than an incomplete cprj.
I do not really get the point about confgiured component : I see it as a list of components you have preselected and by aggregating layers you obtain a cprj.
So to me it is only a way to share pre-assembled components ?
But a component with dependencies can probably do the same ?
I agree that in case several components can serve a dependency then the selection is already made : true.
But for the preconfigured I am not sure to get your point.

Thanks & Regards,

[jkrech]

Hi,

the ctarget concept is not crystal clear to me. Does it aggregate what we call an execution context and its execution domain ?

So, in a multi-context application project, would we have several ctarget files ?

Thanks & Regards,
We don't have specified the details, but there shall only be a single ctarget file with references to cproject and clayer files. What we shall decide is whether resource allocation is described in ctarget for all execution domains.

[jkrech]

I would also like to understand better the concept of clayer. To me if we want reusable components we have the CMSIS components to do so. A component can be source code or a libray.
What is clayer bringing on top of a component with dependencies ? It memorizes pre-resolved dependencies ? Is that the point ?

For more information about project layers (*.clayer) please read here: CMSIS-Build and to see it at work you may want to look at the Lab4Layer repository
Layers are the next level of abstraction between projects and components. The benefit of layers is that it is not only a list of components, but a combination of components that are already pre-configured for a range of use cases.

Hi Joachim,

I remember Lab4Layer as I played with it some time back. But to me a clayer was nothing more than an incomplete cprj. I do not really get the point about confgiured component : I see it as a list of components you have preselected and by aggregating layers you obtain a cprj. So to me it is only a way to share pre-assembled components ? But a component with dependencies can probably do the same ? I agree that in case several components can serve a dependency then the selection is already made : true. But for the preconfigured I am not sure to get your point.

Thanks & Regards,

Please take a look at this board layer:
https://github.com/MDK-Packs/CB_Lab4Layer/tree/master/layer/Board/32F746GDISCOVERY
You will see that a clayer is not only a single description file with references to components but it is a directory contains directories and files and that it also includes the RTE folder. The RTE folder contains the configuration files of the components. These configuration files tailor a component to a specific use case.

[fred-r]

Hi,
the ctarget concept is not crystal clear to me. Does it aggregate what we call an execution context and its execution domain ?
So, in a multi-context application project, would we have several ctarget files ?
Thanks & Regards,
We don't have specified the details, but there shall only be a single ctarget file with references to cproject and clayer files. What we shall decide is whether resource allocation is described in ctarget for all execution domains.

Hi Joachim,

ok so the ctarget is the implementation of the application project concept right ?

Now, the cproject could be the SW project implementing an execution context I guess.
It is then unclear to me how it relates to cprj and ctarget.

• To me we would probably have one cprj per build flavor (or profile like release/debug) or one for all flavors as today the cprj spec allows to define several toolchains and targets as far as I remember.
• So the cproject would reference all the clayers it needs (if any, I assume we may work with only one list of components ?) and the tool would generate a cprj from it

Then regarding resource allocation, as you know, we may envision context segregation.
If so, execution domains shall remain separated so probably we would need to reference them at cproject level ?

Thanks & Regards,
Fred

[fred-r]

I would also like to understand better the concept of clayer. To me if we want reusable components we have the CMSIS components to do so. A component can be source code or a libray.
What is clayer bringing on top of a component with dependencies ? It memorizes pre-resolved dependencies ? Is that the point ?

For more information about project layers (*.clayer) please read here: CMSIS-Build and to see it at work you may want to look at the Lab4Layer repository
Layers are the next level of abstraction between projects and components. The benefit of layers is that it is not only a list of components, but a combination of components that are already pre-configured for a range of use cases.

Hi Joachim,
I remember Lab4Layer as I played with it some time back. But to me a clayer was nothing more than an incomplete cprj. I do not really get the point about confgiured component : I see it as a list of components you have preselected and by aggregating layers you obtain a cprj. So to me it is only a way to share pre-assembled components ? But a component with dependencies can probably do the same ? I agree that in case several components can serve a dependency then the selection is already made : true. But for the preconfigured I am not sure to get your point.
Thanks & Regards,

Please take a look at this board layer: https://github.com/MDK-Packs/CB_Lab4Layer/tree/master/layer/Board/32F746GDISCOVERY You will see that a clayer is not only a single description file with references to components but it is a directory contains directories and files and that it also includes the RTE folder. The RTE folder contains the configuration files of the components. These configuration files tailor a component to a specific use case.

Thanks Joachim.
But I will keep on asking questions...sorry...

What is unclear to me is how this clayer relates to the SW delivery if it references files stored in the RTE folder.
To me files we have in the RTE folders are template files the end-user can further modify.
So today we deliver only templates in the CMSIS components (like startup files for instance in DFP packs having the config attribute).

So how would you deliver this configured file stored in the RTE folder of the project ?

By the way, and I think I already mentioned it in the past in a more specific case (ARM-software/CMSIS_5#1103 (comment)), but I am not a big fan of the RTE folder.
I would prefer having the possibility to choose where to put my files.

Thanks & Regards,

[fred-r]

Hi,

please find enclosed a document introducing some vocabulary and concepts we would like to discuss in the context of this project.

Thanks & Regards, Multi-Context_Concepts_OpenCMSIS.pdf

Please find a minor update of the document : we can consider that a binary can be produced by 1-N software projects and not exactly one (a binary can include a binary or link a library for instance, even if at the end there is one SW project producing the final binary)
Multi-Context_Concepts_OpenCMSIS.pdf

[slhultgren]

It seems like there is quite a big overlap/duplication of information between the .cproject.yml file and the .cprj file.
In this proposal, is the .cprj now "demoted" to being only a temporary output file used for building the project?
(if so, is the intention to get rid of the .cprj file altogether?)

In terms of where to store project settings, is the source-of-truth now instead the .cproject.yml file?

[ReinhardKeil]

Closing as there is no further activity. Feel free to re-open or start a new issue in case that there is still something relevant.