<packlist>: CMSIS-Pack Inventory file for workspace

[ReinhardKeil]

Development Tools may have a central storage for software packs, which means that there could be potentially many thousand packs installed. As the pack tools read the software packs, (a) the underlying tools may get unresponsive and (b) the user might be overwhelmed with content for example in the "Run-Time Management" dialog of an IDE.

The proposal is therefore to introduce a <packlist> file that limits the packs processed by pack management and build tools as shown in the picture below.

The <packlist> file is automatically extended when a *.cprj file is added to a workspace (the project file has a list of packs). Alternatively there needs to be a way for the user to add more software packs that are considered in the workspace.

Questions:

• Is this an acceptable solution?
• Should we extend the purpose of , for example to refer repositories (during a development phase of a pack) instead of the pack itself?
• Should the content of this file merged into other workspace related files?
• What should be the filename? Any recommendations or suggestions?

[LMESTM]

At first sight, this sounds like a proposal that wouldn't hurt. If no such file then we're back to default / full list of PACKs.

Possible questions are then:

• how would user look for new packs that he needs to add in his projects? The search capability shouldn't be restricted right ?
• maybe we shouldn't limit to list of packs, but allow for some "filtering" (keep all packs with defined keyword, or all packs from given vendor ?)

Other side questions about managing list of Packs.
Do we have the possibility in the *.ctarget or *.cprj to get a snapshot of the current list of packs and their versions, similarly to package-lock-json in node ? That would be helpful to ensure that projects can be re-built with exact same versions of packs.

[ReinhardKeil]

Q: how would user look for new packs that he needs to add in his projects?
A: I believe we need to consider something like a PackInstaller in MDK.

Q: maybe we shouldn't limit to list of packs, but allow for some "filtering" (keep all packs with defined keyword, or all packs from given vendor ?)
A: I believe we need to add top-level categories to packs, i.e. "Graphics", "Cloud Connector", etc.

Q: Do we have the possibility in the *.ctarget or *.cprj to get a snapshot of the current list of packs and their versions, similarly to package-lock-json in node ?
A: *.cprj hast this list of packs) already.

Q: Could you explain a bit better how package-lock-json works?

[LMESTM]

Q: how would user look for new packs that he needs to add in his projects?
A: I believe we need to consider something like a PackInstaller in MDK.

If the file applies to the full "pack management and build tools" it could also impact Pack installer. that can make sense, for instance, if I want to only search in the list of PACKs from arm ... but I think that is not the idea here. The Pack Installer will have its own filtering capability

Q: maybe we shouldn't limit to list of packs, but allow for some "filtering" (keep all packs with defined keyword, or all packs from given vendor ?)
A: I believe we need to add top-level categories to packs, i.e. "Graphics", "Cloud Connector", etc.

Yes indeed..

Q: *.cprj hast this already.

Do you mean it can be specified here in *.cprj packs description ?
I understand that this filed may not specify the versions .... so the need I see: user gets cprj, then he downloads the latest versions of PACKs and components as specified in *.cprj. and wants to make a snapshot of these versions, but he doesn't want to prevent later updates by modifying cprj. Is that possible ?

[jkrech]

Q: *.cprj hast this already.

Do you mean it can be specified here in *.cprj packs description ?
I understand that this filed may not specify the versions .... so the need I see: user gets cprj, then he downloads the latest versions of PACKs and components as specified in *.cprj. and wants to make a snapshot of these versions, but he doesn't want to prevent later updates by modifying cprj. Is that possible ?

In CMSIS-Build this is currently not possible within a single *.cprj file, however cbuildgen can create a copy of the cprj file for a "reproducible build".
https://arm-software.github.io/CMSIS_5/latest/Build/html/cbuildgen.html
cbuildgen my.cprj cmake --update=my_snapshot.cprj | Create fixed version CPRJ file. | Generate with fixed versions for reproducing the current build.

Alternatively you could specify a pack folder that you specifically maintain for this project, which means a lot of duplication of packs on your machine, which is the opposite of what the pack system is designed for.

[ReinhardKeil]

Pack Installer
Yes, the pack installer would discover all packs that are for example available in public pack lists (similar to www.keil.com/pack).

Select Pack Versions for a project
In the *.cprj package element it is already possible to specify a pack version (or a version range).

This is how the MDK uVision IDE selects the pack versions for a project.

To update a pack on a local desktop computer, the user would need to actively download it (for example by using the PackInstaller). There could be a "discover updates for a workspace" feature to assist here.

[cartu38]

If understanding well packlist proposal is to improve tools responsiveness. What's a good concern.

A first question then: "The file is automatically extended when a *.cprj file is added to a workspace" ... I guess some need for packlist update too if some pack dependency is removed from a *.cprj. Same if a .cprj it self removed from workspace. Sounds like sync. effort (what about end user "hacking" *.cprj out of IDE context, ...) so gain/loss to be well balanced.

A second question: should not be such packlist an in memory object relying on IDE session rather than a physical file ? Sounds like to me a non functional file end user doesn't want to be aware about (source control, ...). Ok still paying price to create such object per IDE session but only one time computation per session what's an improvement already.

I like "top-level categories" addon. Sounds to me good way to improve performance is to introduce way to cut as soon a possible some branches within packs tree. Such is runtime computation saver for sure.

[jkrech]

Workspace

What is the definition of a workspace?
What shall be common for ctarget and cprj within a workspace?

• a common physical directory containing N ctarget files with M cprj subdirectories?
• a ctarget can only reference a cprj within the workspacectarget files?
• all projects in a workspace share a single CMSIS_PACK_ROOT folder?

Pack Installer

In the context of CMSIS-Build we have defined the pack installer to be quite restricted. The cli tool shall either add (install) or remove (uninstall) one or more packs from the specified CMSIS_PACK_ROOT folder. Packs are specified by vendor, packname, version and the file extension pack/pdsc. For adding a pack, a download https:// url or file:/// url is required. If a file url referencing a pdsc file is specified, that local repository is added/removed to/from the local_repository.pidx

Pack Manager

This tool has much more advanced capabilities including:

• check for updates of available published, local and installed packs
• discovery: (filtered by preference vs. all)
  • list of available, downloaded and installed pack versions
  • list of devices/boards described in latest packs and locate pack describing a specific device/board
  • list of examples for a selected board/device
  • list of components / filtered list of components
• managing pack preferences

Pack Preferences

Where shall the preferences be located?

• cprj - cprj already maintains a list of required packs by the build (optionally specifying version range or fixed version)
• ctarget ? e.g. .cpref
• workspace ? is there a workspace file or is this implicit? e.g. workspace.cpref
• CMSIS_PACK_ROOT folder? CMSIS_PACK_ROOT/.Preferences/.cpref
  • profiles shall be managed by Pack Manager
  • profiles can be selected for searching and a default profile is stored per ctarget.

[ReinhardKeil]

Definition of Workspace

A workspace gives access to a meaningful set of ctarget files.

In the first implementation it could be a single physical directory containing that contains N ctarget files.

As the ctarget files refer cprj files, the location of the cprj's could be elsewhere.

A workspace does not define the location where packs are located. This is a separate setting and could be part of the <packlist> (the exact definition of <packlist> is ToDo).

[willlordarm]

A second question: should not be such packlist an in memory object relying on IDE session rather than a physical file ? Sounds like to me a non functional file end doesn't want to be aware about (source control, ...). Ok still paying price to create such object per IDE session but only one time computation per session what's an improvement already.

In memory could be the right way to think about this. Conceptually I would expect a user of Keil Studio Cloud (or any other IDE compliant with Open-CMSIS-Pack) to have access to an 'index' of packs that is the union of publicly available CMSIS Packs and local Packs (with local meaning specific to a user and/or workspace not necessarily local to a machine). Within an individual workspace the user should be able to specify which packs are 'active' (i.e. available to the RTE). This active list could be populated from a .cprj or may be manually selected from the index based on a search or filter applied by the user.

[DavidJurajdaNXP]

I am missing purpose of <packlist> file, especially way how it could improve performance.
From my perspective it is kind of hiding problem, not solving it.

If I install some pack into local root pack folder, I am expecting it become visible in pack manager.
I would like to see all locally installed packs in pack manager for purpose of project creation, until I apply some filter.
Consider empty workspace without any project. In this point tool need to deal with all available packs, because the user do not know yet which components become useful for his project.

Instead of filtering of single central root folder by whitelist filter, I would prefer to create new install folder with required packs subset (like Python virtual environment).

Packlist looks like additional manual step during installation:

1. Install pack to local storage by pack installer (install pack in computer scope)
2. Allow usage of pack in workspace via packlist ("install" pack again for workspace scope)
3. Use pack by pack manager.

Filtering should be done in pack manager not in the workspace. User already defined scope by act of pack installation.
Filtering should be highly interactive, which is hard to imagine with packlist.

If we consider tool performance, I would agree the content of installation folder should be automatically indexed, but not filtered.
I my opinion tools like Run-Time Management should not interact with packs directly, but indirectly via indexes.

Ability to add additional location to pack search paths would be useful for accessing local repositories (with packs).

There should be also method for generation of something like Python requirements.txt. It is file which is describing exact content of install folder and allow its replication/re-installation elsewhere.

[jkrech]

Search path for packs:

Today CMSIS-Pack uses the concept of a single reference point for tools to locate packs. This reference point is either specified by the environment variable CMSIS_PACK_ROOT or specified as command line option to the tools. Even though there is no restrictions where and how many of these pack directories exist, this is always independent from the projects. By default all projects will share the same set of installed packs.

Packs are either extracted into a pack vendor, pack name and pack version specific subdirectory or a reference to a pdsc file is listed in the local-repository.pidx file

             .Local/local-repository.pidx
             ARM/CMSIS/5.8.0/...
             STMicroelectronics/STM32F7xx_DFP/2.7.0/...

We are looking to extend this concept by supporting the specification of workspace/solution/project specific packs, which are e.g. stored as part of the project.

Pack Filter

We are looking to extend the concept by support the specification of workspace/solution/project specific pack filters to exclude packs from the search path which are irrelevant in the scope of the project.

exclude packvendor : packname @ version

[0Grit]

I want all sources I end up compiling to be imported into the directory that makes up my projects workspace.
As a mere human I like to have visibility over the sources I am using and to know with certainty that the sources I build with are exactly the sources I want to be building.

In my case a project is generally an embedded application running on an MCU.

[LMESTM]

About file proposal, how does it relate to the lock file proposal that has been discussed here: #11 (comment)

Is it fully separate/complementary discussion or do they overlap ?

[slhultgren]

Can we imagine that we have a new "magical" optional folder as a sibling to any important project file (.csolution or .cproject or .cprj?).
If this new magical folder exists, we search/install packs by default into this one instead of any specified CMSIS_PACK_ROOT.

This lets us use this new folder to override any specified CMSIS_PACK_ROOT env variable.

So I imagine it like this:
$SOME_FOLDER/some_file.cproject.yml
$SOME_FOLDER/.cmsis_pack_root // <-- new "magical" optional folder. Internal structure identical with any CMSIS_PACK_ROOT

We could introduce a preference setting in the $SOME_FOLDER/.cmsis_pack_root that specifies if it inherits from the env CMSIS_PACK_ROOT as well, e.g. with optional filtering:

• inherits: true Means that all packs available in env[CMSIS_PACK_ROOT] will also be used by the tools, but the project-local-".cmsis_pack_root"-installed-ones take precedence over the "global" from env[CMSIS_PACK_ROOT]
• inherits: ["ARM.*", "Vendor.Name.Version"] here only inherits a subset of packs from env[CMSIS_PACK_ROOT]. (All ARM packs, and the specific "Vendor.Name.Version" pack)
• inherits: false If false, then tools should ignore any env[CMSIS_PACK_ROOT]

To me, this is useful for determining how to search, and is complementary but not replacing the need for a lockfile.

[ReinhardKeil]

This is outdated