.GPDSC location clarification

[slhultgren]

The specification https://open-cmsis-pack.github.io/Open-CMSIS-Pack-Spec/main/html/pdsc_generators_pg.html#element_generator only specifies that the .gpdsc is located in the specified absolute path or relative to the workingdir.

This is problematic if multiple contexts for different boards/component-sets use the same generator in the same project.
The output-dirs[gendir] property introduced in the csolution / cproject-yml files should solve this problem since this allows specification in a context aware way.

Ideally both the .gpdsc and the generated files created by the generator should end up in the user-configurable output-dirs[gendir]-location.

Backwards compatibility

Since not all generators will be aware of the output-dirs[gendir] from the $G parameter, the tools should look for .gpdsc in a few different places:

1. First look for it in the output-dirs[gendir] folder.
2. If not in 1, then look according to PDSC specified location which is relative to WorkingDir / Absolute path specified

[jkrech]

@slhultgren - you are raising the same generator used by multiple different target-types. How about multiple generators?

[ErikMallbergSTM]

Here are proposed tables for paths in the different cases. @ReinhardKeil What do you think?

Where is gendir specified and what is the result?

csolution	cproject	clayer	component	Resulting folder for generated code and primary GPDSC location (look here for GPDSC first but if not found look in "Resulting GPDSC location" in workingDir table below)
n/a	n/a	n/a	referenced by project	$project/gendir-default
n/a	n/a	n/a	referenced by layer	$layer/gendir-default
gendir	n/a	ignored	referenced by project	$solution/gendir
gendir	n/a	ignored	referenced by layer	$solution/gendir
ignored	gendir	ignored	referenced by project	$project/gendir
ignored	gendir	ignored	referenced by layer	$project/gendir
n/a	n/a	gendir	referenced by project	$project/gendir-default
n/a	n/a	gendir	referenced by layer	$layer/gendir

Priority order:

1. project – most important
2. solution
3. layer

gendir-default: ./generated

Where is workingDir specified and what is the result?

PDSC workingDir	PDSC specified GPDSC location	Resulting workingDir	Resulting GPDSC location	Comment
n/a	n/a	$project	workingDir
n/a	relative	$project	relative to workingDir
relative	n/a	relative to PDSC location	workingDir
relative	relative	relative to PDSC location	relative to workingDir	workingDir cannot be relative to GPDSC since GPDSC is relative to workingDir
relative	absolute	relative to PDSC location	absolute GPDSC loc.
n/a	absolute	$project	absolute GPDSC loc.
absolute	n/a	absolute workingDir	workingDir
absolute	absolute	absolute workingDir	absolute GPDSC loc.
absolute	relative	absolute workingDir	relative to workingDir

The workingDir table does not really change anything compared to the current spec except clarifying the relative-relative that's an infinite recursion

[ReinhardKeil]

The problem with gendir is that it applies to all Cclass names. When a company (i.e. MLOps) provider has an own generator, this may create a conflict. There is not just one *.GPDSC file in such cases.

Have you considered that type of workflows?

I'm still leaning towards a Cclass specific settings as outlined here:
https://github.com/Open-CMSIS-Pack/devtools/blob/main/tools/projmgr/docs/Manual/Proposals%20-%20OutofScope.md#rte-dirs

I had not enough time to work on the aspects of multi-core yet with CMSIS-Toolbox 1.5.0 (as this version should fix the existing CubeMX workflow).

[jkrech]

clarifying /package/generators element the workingDir description:

• If a is described in a file then a relative path in workingDir is relative to the pdsc file
• If a is described in a file a relative path in workingDir is relative to the gpdsc file
• If no workingDir is specified, the project directory is used.
  • this is either the directory where the cproject.yml or clayer.yml file is located, depending on where the components associated with a generator are listed.
  • Note: components of a single generator must not be distributed across cproject and clayer files.

[slhultgren]

Thanks for clarifying the relative workingDir case, makes sense :)

Note: components of a single generator must not be distributed across cproject and clayer files.

Hmm, this seems like a bit problematic, but indeed highlights the issue with placing certain files in the $layer folder (which I don't really like personally)

My belief is that the user is focused on the project. This is where they would expect files to be created. IMO the layer is only a "header-file" that is included by the project. The "build artifacts" and sources are still project focused.

Compare cproject.yml using a clayer.yml with a .c file doing a #include "my_layer.h"

[ReinhardKeil]

Here is a proposal that could simplify the situation:

Definitions:
“gen-attribute” is the string in the component attribute generator.
“generator-component” is a component that defines a “gen-attribute” generator in the *.PDSC file.
“working-directory” is the directory that is used to save the *.GDPSC file.
All files that are defined by a *.GDPSC file have a relative path based on the file location of the *.GDPSC file.

Practical usage (as it exists today) of working directory:

Usage-Restrictions (should be enforce with tools like packchk).

• In PDSC files: the “working-directory” specification must start with $P which makes the directory relative to the project file.
• In GDPSC files: the “working-directory” specification must be identical with the specification in the *.PDSC file.

New behavior of working directory (to be implemented)

Suggestion remove request to define a “working-directory” in PDSC/GDPSC files.
The default “working-directory” is the RTE directory of the “generator-component” (which is dependent on the Cclass that defines the component.

Behavior of $P/%P in PDSC/GDPSC files (already implemented in csolution, add to documentation)
The parameter $P/%P refers to the project YML input file that specifies the "generator-component". In practice it is:

• “*.cproject.yml” if the related “generator-component” is in the *.cproject.yml file.
• “*.clayer.yml” if the related “generator-component” is in the *.clayer.yml file.

Behavior of base directory for RTE files for components (for completeness, already implemented in csolution, add to documentation)
The base directory of RTE files depends on the location of the component. It is:

• base directory of “*.cproject.yml” if the related “component” is in the *.cproject.yml file.
• base directory of “*.clayer.yml” if the related “component” is in the *.clayer.yml file.

Extending YML input files:

• gendir: overwrites the “working-directory” for all “generator-components”. (already implemented, and documented).

• rte: overrides the default path or the RTE directory for specific Cclass name, example: (new feature)

  rte:
    - Device: <working-directory for Cclass Device>
    - Board: <working-directory for Cclass Board>
  

Verification by csolution tool: (new features)

• csolution verifies consistency of “working-directory”

  For a given context, all “generator-components” that have the same “gen-attribute” must refer to the same “working-
  directory”. csolution reports and error when this is violated.

• csolution verifies consistency of “RTE-directory”

  For a given context, all “components” of the same Cclass must refer to the same RTE directory.

[slhultgren]

Some comments on the proposal from #207 (comment)

• The execution directory and output directory are two separate things potentially. (exec_dir != output dir)
  • workingDir → execution directory, this is where the generator potentially expects to read some exe-relative files, like dlls or other hard-coded relative paths
  • output directory → where the generator will create files such as the .gpdsc
• $P is currently defined as "current project folder", is the new %P the resolving to current project file name e.g. 'test.cproject.yml'?
• Changing the default working-dir to RTE folder will potentially break existing generators. I think the default working dir should be kept in the project directory as the current specification states.

The default values of gendir and rtedir to be either layer relative or project relative depending on where the component is specified seems hard to use in practice:

• The requirement that all components of the same Cclass must refer to the same RTE directory effectively makes layers very hard to use in a proper project with default values. E.g. if the cproject uses a Device:startup, then no layer can use ANY Device component since the "Device-rtedir" will be inconsistent for project-components and layer-components.

• The requirement that a generator may only have a single output folder seems like a subtle but major limitation for the usage of generated components between layers and cprojects, since the generator may then only be used either in the cproject OR in the clayer.

These seem like big hard-to-explain-limitations why certain combinations of default values and components/layers won't work.
Effectively in both of these cases, the user will be forced to use an explicit rtedir and gendir to force the files to be placed in ONLY the project.

My vote will always be to simplify and completely remove the layer-specific rtedir/gendir outputs, and say that regardless of where the components are specified, only the outputdirs from the csolution/cproject are relevant.

IMO the simplest rules would be to say (like in #207 (comment)):

1. A generator is launched in the workingDir 

1.1. workingDir can be specified in the .pdsc like today. If left unspecified, the project directory will be used.

2. Output from the generator (which includes the gpdsc) is expected at 3 potential locations:
	Location 1 most important: "csolution/cproject-gendir"
        Location 2: gpdscloc according to the pdsc, if the pdsc specifies it
        Location 3: least important: workingDir according to pdsc if the pdsc specifies it
        Location 4: default to search in the cproject.yml-folder
        If the gpdsc is found in any location, the next locations will not be searched.
        This way newer generators that are gendir aware can start using it, while old generators will still behave like before.  

2.1. Potential <generator> sections in .gpdsc have no impact

3. Whether a component is used from the cproject or clayer makes no difference, the rtefiles and generated files are always centered around the "csolution/cproject gendir/rtedir" values.

[ReinhardKeil]

@slhultgren thanks for your feedback. Let's discuss today what is written below:

Execution Directory:

Issue 626 discusses this problem already (suggests a PATH environment variable). I understand that a CubeMX team is working on a proposal. It should be relative easy to query the execution directory within the tool. When I remember correctly this is retrieved by GetModuleFileName or similar calls within the program itself.

Working Directory:
The definition of $P needs to change in documentation as explained above as this is the current way the tools are working.

Changing the default working-dir to RTE folder will potentially break existing generators.

Yes, I agree. However, my analysis of current pack implementations makes me believe that this will work. It needs however verification.

The default values of gendir and rtedir to be either layer relative or project relative depending on where the component is specified seems hard to use in practice.

Agreed to some extend. However using gendir: or rte: as explained above provides options. Unfortunately I have no idea for a golden default behavior, but I am open for suggestions. My current suggestion is: let's try it in practice on non-trivial examples
that we currently developing as part of the reference application framework. As we agreed to support CubeMX, this could be tested using the CubeMX tool.

If we discover during these tests that other options are required, i.e. your suggested default behavior, we can change the csolution tool accordingly.

Note: If a future Cube2 tool needs a special mode, let's also discuss it. We could add a flag for that, but I believe gendir: or rte: provides enough options for exploration.

[ReinhardKeil]

Consider to change the name of Working Directory to Generator Output Directory

[ReinhardKeil]

I did review my proposal to remove the Working Directory. Unfortunately, this does not work for CubeMX for multi-processor projects or trustzone enabled projects. Here more then one cproject.yml share the same generator settings.

I still would recommend to rename the directory to Generator Output Directory. We could make this directory configurable a csolution.yml level. Default when not specified is as described under #207 (comment)

[ReinhardKeil]

This is the revised proposal for the generators element in the PDSC file.

workingDir is the default directory where generated files are stored. By default the directory is relative to the csolution.yml file (uses csolution.yml as base directory). Using the prefix $P makes the directory relative to the cproject.yml or clayer.yml that defines the component(s) with the related generator-id. workingDir cannot be an absolute path.

workingDir can be overwritten by using the gendirs: node in csolution.yml, cproject.yml, clayer.yml. workingDir is not required, in this case an explicit gendirs: node must be specified.

gpdsc is the name of the GPDSC file that the generator creates and updates.

When using the csolution tool, the cbuild.yml file contains the generators: node that contains the GDPSC filename and the working-dir: as shown below:

generators:
   Cube2:
      working-dir: ../CubeFiles          # from `gendirs:` or if not specified from  PDSD file element <generators>
      gpdsc: MyPackName.gpdsc            # from PDSC file element <generators>

[ReinhardKeil]

Examples that use STM32CubeMX:

• stm32l4_blinky.zip is a simple project that stores the Cube configuration in .\RTE\Device<device-name>
• stm32u5_blinky.zip is a TrustZone example that uses the directory ..\Boards to store the Cube configuration.

[slhultgren]

This is mostly part of the cbuild.yml today, what remains is perhaps just to clarify the purpose of these properties in the cbuild.yml.

Today working-dir in cbuild.yml is the execution directory. This should be dropped.

The gendir: property should be added for the generator in the cbuild.yml and is strictly the generator output directory. This should be added

The result would then be something like

generators:
   <generator id>:
      # from `gendirs:` or if not specified from PDSC file element <generators>.
      # The place where generator should create all files
      gendir: ../CubeFiles

      # from PDSC file element <generators>. Should be created inside the gendir above
      gpdsc: MyPackName.gpdsc

Then perhaps we could drop the gendir from the output-dirs in the cbuild.yml.

[ReinhardKeil]

@slhultgren it looks like we are now aligned. I reflected it in the csolution documentation with this commit:
ReinhardKeil/devtools@74f6693

Before I make a pull request, could you please review.

Have a nice weekend.

[ReinhardKeil]

@slhultgren I suggest to close this item as Open-CMSIS-Pack/devtools#832 covers this now.

[ReinhardKeil]

Replaced by Open-CMSIS-Pack/devtools#880 - suggest to continue discussion there