CMSIS-Build Gap Analysis

[jkrech]

The following items are collected from the discussion of the Shadowfax Componentization WG exploration team investigating requirements for a project format and a build management CLI tool.

Discussion was based on CMSIS-Build 0.10.0 https://github.com/ARM-software/CMSIS_5/releases/download/5.7.0/cbuild_install.0.10.0.sh
note: a number of limiting items raised in the discussion have been fixed in version 0.10.2:

Gaps:

• pre- and post-build steps must be specified as part of the build project (item 2)
  • cross platform support
  • specify runtime environment
• define generator operation as part of the build
• define multiple build configurations variants within single project (e.g. Debug vs. Release)
  • need to carefully decide what the supported variations shall be
  • do not allow different board, device, module?
  • only allow difference in component variants?
• hierarchical tool-chain flags (e.g. cflags) do not support an option to "override" in addition to "add" and "remove" flags on a group/file level. (item 11)
• define an additional file that orchestrates multiple dependent projects ("Umbrella") (item 12)
  • multi-core, secure / non-secure, boot-loader + application, etc.
  • build order, cross-project references (item 5)
• define and maintain a dictionary of tool-chain flags that allows to specify command line options for different toolchains in a common way (e.g. optimization levels, debug vs release flags) (item 10)
• extend the files section to allow specifying a physical directory rather than individual files to be included in a build (item 7) (Note: this feature was not agreed, as some people found it "convenient" and others found it "dangerous" and "fuzzy" (e.g. for reproducible builds) - convenience could be achieve through tools)
• specifying the order of static libraries and objects within the scope of cprj as well as multiple cprj (can this be reliably derived from the order the cprj/components/files are listed?)
• add operating mode where component files get copied from packs into the project tree (note: requires to explicitly list all header files that belong to a component)

Limitations

• support nesting of logic file groups in files section (item 3) (fixed)
• support file specification in files section without group (item 4) (fixed)
• extend target section with toolchain independent specification of include path and define (item 9) (fixed)
• multiple toolchain support requires toolchain specific component configuration files to be named uniquely as they otherwise overwrite each other (design requirement ?)

Conclusions from the discussion:

• the cprj file must allow to support multiple compiler tool-chains configuration options concurrently (item 8)
• the cprj file must not store IDE/UI settings. These settings shall be stored in separate files (item 1)
• in order to create self contained copies of projects, all project file references are relative to the project file (item 6)
• hierarchical tool-chain flags: decide whether to record "delta information" (add, remove) or "complete information" but do not mix both approaches. The override should be achieved by combining "remove" and "add" accordingly.

[tarek-bochkati]

thanks @jkrech for summing up CBuild/CPRJ related notes.

I'm just missing the flags override attribute like the 'add' and 'delete' attributes, as far as I remember we were partially OK on this point.

is it possible to add this to the GAPs list ?

[jkrech]

@tarek-bochkati good catch. I did miss it above. I have added it to the list of gaps, however I also added the conclusion from the discussion, that in the rare case of an override, this should be achieved by specifying the 'remove' with the complete flags and specifying the complete set of new flags using the 'add' attribute.

[LMESTM]

Another item that would be useful, probably in cprj, is how to handle the source code of project components.
As of now, AFAIK, in Keil components source code is kept in CMSIS Pack root. But we think that users may prefer to have a copy of the selected source at specific location in their project tree. So cprj should offer an option to specify for each component to either use the cmsis pack root reference, or to make a copy (and where to copy).

[cartu38]

Any initiative to specify static libraries (.a) dependencies order (maybe reading order is way to do today?) ? Kind of corner case but sounds to me required thinking reproducible builds.

[cartu38]

'/cprj/files/.../file' 'name' attribute : do you consider support of some variable describing path to get resource ? Today relative (or absolute path if supported ?) are not helping project structure revisit (resource(s) repository(ies) renaming, move, ...).

[jkrech]

'/cprj/files/.../file' 'name' attribute : do you consider support of some variable describing path to get resource ? Today relative (or absolute path if supported ?) are not helping project structure revisit (resource(s) repository(ies) renaming, move, ...).

Today files are specified relative to cprj file only, since absolute paths do not allow to relocate the project (projects shall be copied out of packs using a recursive copy of a base folder). Renaming and moving of files is probably a feature that a tool shall provide rather than burdening the project format with?

[jkrech]

Another item that would be useful, probably in cprj, is how to handle the source code of project components.
As of now, AFAIK, in Keil components source code is kept in CMSIS Pack root. But we think that users may prefer to have a copy of the selected source at specific location in their project tree. So cprj should offer an option to specify for each component to either use the cmsis pack root reference, or to make a copy (and where to copy).

Added to the description above.

[jkrech]

Any initiative to specify static libraries (.a) dependencies order (maybe reading order is way to do today?) ? Kind of corner case but sounds to me required thinking reproducible builds.

Added to the description above.

[slhultgren]

Another topic is "lockfile" concept.

Since the .cprj file allows working with version ranges, and since the condition resolving of components are non-obvious to users it would be very helpful to record the exact used packs and components in a ".cprj.lock" file.

This will allow for a similar workflow as with "package.json/yarn.lock" from NodeJS ecosystem, where the ".cprj" describes what CAN be used, and the ".cprj.lock" describes what IS used.

This way, there is no fear when moving a project from one computer to another computer with different CMSIS_PACK_ROOT contents, since the .cprj.lock file will guarantee that we still build with the same contents.
The users will then commit the .cprj.lock file, and if there was no changes in the .cprj.lock file they can feel confident that they are still using the same sources in the build.

[brondani]

Another topic is "lockfile" concept.

It seems the "lockfile" would have precisely the content of the file generated with the cbuildgen --update=<CprjFile> option:

Generate <CprjFile> with fixed versions for reproducing the current build.

@slhultgren can you please confirm it?

[0xc0170]

Regarding the lockfile, one feature we had to add to one package manager - related to the lockfile. I could create a new request for it rather than here if interested. Fetch all dependencies that are in the lockfile and store it in the project folder.

When we were releasing software, we used to fetch dependencies, basically local copy of all deps - complete software pack.
The feature is important for people who have restricted access to internet or very slow or another limitations or you just want to forward complete application to a customer. It contains the lockfile and resolved all dependencies.

[slhultgren]

@brondani, --update does solve the reprodicibility of builds, but I'm not sure I understand how to actually implement this in my workflow to support the flexibility I want.

Basically, I want to have my project description slightly loosely defined (using ranges for packs, not specifying everything for the components I use), while still being able to:

• Share this with my colleagues, and be sure that we build the same thing, and they can still see my "loose" intensions.
• If I add a new component to my project, be sure that I am now still building the same components as before (if any condition-evaluation changed due to the new component added it should be clearly highlighted by a changed lockfile)
• Adding new files or compilation options to my "loose.cprj" without worrying about changing component/pack versions used in the build
• Updating used packs at certain manual times in my project development process, regardless of what is present in my CMSIS_PACK_ROOT

In this case I need to have both a "loose.cprj" and a "updated-fixed.cprj", put both of these in GIT to share, and always build the "updated-fixed.cprj".
However, any change made to the loose.cprj would need a re-update of the "update-fixed.cprj", even if I just added a -D flag to the compiler.
This could potentially use newer packs than the previous updated-fixed.cprj.

An issue with this approach to locking (--update) is that it cannot support both reproducible builds in a team and still keeping the project definition fairly loose.
Having the resolved/locked components in a separate file (like package.json vs yarn.lock) would allow both keeping the project definition loose, and also guaranteeing reproducible builds.
Updating specific components is also fairly trivial since it would only require removing those lines from the lockfile.

Also, having the lockfile separate could allow to make it more explicit as well, e.g. for each component record "$PACK_ID $COMPONENT_ID" where the condition of the component is also shown. That would allow a simple way to locate/identify when some components/files change in the build.

@0xc0170 Simplifying offline builds is very interesting also indeed. Did this lockfile also include which components were used? (including the condition that resolved it)

[brondani]

An issue with this approach to locking (--update) is that it cannot support both reproducible builds in a team and still keeping the project definition fairly loose.

@slhultgren I believe you misunderstood the --update=<CprjFile> usage. In your workflow when building a project, to create/update a lock.cprj without touching the loose.cprj you could simply call:

cbuildgen cmake loose.cprj --update=lock.cprj

[slhultgren]

@brondani hmm not sure? This will create possibly different lock.cprj files using different Packs on different computers depending on CMSIS_PACK_ROOT contents? I want to avoid this while still keeping my loose description in the project.

I thought the --update was a pure output operation, or is it also reading some "locking" information from the --update file?

This is pretty standard use case for various package managers where you in the project file describe a range/* for the versions that you want to work with, then the .lock file actually locks to an explicit version, allowing colleagues, CI, to build the same thing.
If we only have a fixed.cprj, then I think updating packs in the project will require more complex tools, and also it will no longer be possible to describe loose projects.

NPM/yarn example again: The yarn.lock is not changed when I just update the scripts section of package.json => similar to I don't want to change the packs used when I just add a -D to my project.

I hope this makes sense :)

[brondani]

I thought the --update was a pure output operation, or is it also reading some "locking" information from the --update file?

@slhultgren It is a pure output operation. In this example lock.cprj is almost a copy of the loose.cprj file, just with fixed versions for packages/components/config files, so you can directly reproduce the build.

Edit: From your example I believe the answer to my very first question is no, the proposed "lockfile" does not have the same content of the generated cprj file. Actually it only stores dependencies version information while the cprj is self-contained. I appreciate there can be vantages/disadvantages even if the scope is similar. Let's discuss it in the meeting.

[brondani]

Fetch all dependencies that are in the lockfile and store it in the project folder.

@0xc0170 Isn't it just a matter of setting the CMSIS_PACK_ROOT to a project subfolder?

[LMESTM]

Fetch all dependencies that are in the lockfile and store it in the project folder.

@0xc0170 Isn't it just a matter of setting the CMSIS_PACK_ROOT to a project subfolder?

I don't know if this this @0xc0170 idea, but my understanding would rather be to get all the dependencies (most probably only the applicable subset of CMSIS_PACK_ROOT content for a given project) into the project folder / file system tree so that user can zip it. That process may require to configure the project to point to those local copies thanks to a project-level local_repository.pidx. We think this is something users will want to do.

[ReinhardKeil]

We agree, we should have additional locations where packs are stored. Just CMSIS_PACK_ROOT seems not sufficient. Specifically during development of new packs there must be a way to work with other repo's that contain a pack. Also it should be possible to add non-public packs.

[cartu38]

Would come back on:

extend the files section to allow specifying a physical directory rather than individual files to be included in a build (item 7) (Note: this feature was not agreed, as some people found it "convenient" and others found it "dangerous" and "fuzzy" (e.g. for reproducible builds) - convenience could be achieve through `tools)

I'm fully sensitive to reproductible builds argument but sounds to me, if some part of sources are not components, today file base .cprj content is not safe enough. If some not components resources, we may guess some local end user files here. Pointing them explicitly as a file is not at all locking their own content. Except if adding to file path description some checksum attribute we have no guarantee about reproductible build. Today current proposal is fine to me considering .cprj PLUS local project resources (.cprj is part of) under some source control. If source control I foresee no issue to reproductible build adding folder based support.

According to me supporting folder based support is a must thinking alive projects end user is playing with Thanks his favorite IDE.