Feature Request: include package.json

[Jason2866]

for using the builded toolchain with PlatformIO a valid package.json is neccessary.
The builded toolchains from earlephilhower for Arduino ESP8266 do already build and include this file
see esp-quick-toolchain commit
For ESP32 see espressif/arduino-esp32#4714 (comment)

[meltdown03]

You can make your own PlatformIO compatible package using the pio package <path of xtensa32>. Just make your own package.json file, then set platform_packages = toolchain-xtensa32@file:///<path to newly created tar.gz> in your platformio.ini file.
Also, the newest xtensa-esp32-elf version is already available by setting: platform_packages = toolchain-xtensa32@~2.80400.210211

[Jason2866]

I know the Platformio feature to do packages. With adding the package json there would be no need for. For the ESP32-C3 there is no official Platformio package. By adding the JSON the platformio registry is not littered with many user packages with same content.

[antmak]

@Jason2866 BTW maybe you know some doc about package.json syntax? Especially interesting system field values like darwin_x86_64, darwin_arm64, linux_x86_64), where do they come from. It doesn't look like something standard

oh it's kind of python's sys.platform + '_' + platform.machine()

[Jason2866]

Yes probably, since platformio is written in Python. Looked a bit in code and found sys.platform and get_systype() for OS and variant
https://community.platformio.org/t/question-regarding-platform-json/14663

[Jason2866]

@antmak It would help a lot if you add a Platformio compatible package.json.
To use latest GCC build i have to download every package add the package.json and upload to my private repo.

[Jason2866]

@Lapshin Just noticed the newest release still does not contain a package json :-(

@me-no-dev @igrr By adding to the packages the could be used directly from platformio for testing as long the are not uploaded and provided via the Platformio Registry. Furthermore it would save work, since no repacking is necessary for the package upload process to Platformio registry.

[Lapshin]

@Jason2866 , sorry, I haven't heard about package.json until now.

Could you please provide some links to documentation that describes what package.json is, and what it should contain, where it should be placed in an archive?

[Jason2866]

@Lapshin Thx for reacting. The package.json is in the root of the package.
Example:

{
  "name": "toolchain-xtensa-esp32",
  "version": "12.2.0+20230208",
  "description": "GCC Toolchain for Espressif ESP32 Xtensa MCUs",
  "keywords": [
    "build tools",
    "compiler",
    "assembler",
    "linker",
    "preprocessor",
    "espressif",
    "esp32"
  ],
  "license": "GPL-2.0-or-later",
  "repository": {
    "type": "git",
    "url": "https://github.com/espressif/crosstool-NG"
  },
  "system": "linux_x86_64"
}

Mandantory is the correct matching entry of "system"

Code which parses the *.json
https://github.com/platformio/platformio-core/blob/develop/platformio/package/manifest/schema.py

An example to generate the package.json (compiler toolchain for the esp8266)
https://github.com/earlephilhower/esp-quick-toolchain/blob/1cd2ab4a506fbe42be5a6311434ee03064f38b51/Makefile#L299-L310

[Lapshin]

@Jason2866 , please provide more info on the system field. Maybe you have a list of supported values.

I'm asking because it is not in an expected host triplet format...

[Lapshin]

@Jason2866 , if I understood correctly, you have to download all archives to extract and parse package.json. Then, you can identify which archive corresponds to which system...

A question has just crossed my mind. It may not align with the platformio perspective, but I lack knowledge in this area.

Wouldn't it be more efficient to parse all relevant information directly from https://github.com/espressif/esp-idf/blob/master/tools/tools.json? All the required information is available in one place.

[me-no-dev]

@Lapshin i've written a script a while ago to download and repack the toolchains for PIO use. Here is the code that determines what value system should have:

    if(str_contains($asset->name, "linux-arm64") || str_contains($asset->name, "aarch64-linux-gnu")){
        $asset_data["system"] = "linux_aarch64";
    } else if(str_contains($asset->name, "linux-armel") || str_contains($asset->name, "arm-linux-gnueabi")){
        $asset_data["system"] = array("linux_armv6l", "linux_armv7l", "linux_armv8l");
    } else if(str_contains($asset->name, "win32") || str_contains($asset->name, "i686-w64-mingw32")){
        $asset_data["system"] = "windows_x86";
    } else if(str_contains($asset->name, "win64") || str_contains($asset->name, "x86_64-w64-mingw32")){
        $asset_data["system"] = "windows_amd64";
    } else if(str_contains($asset->name, "macos")){
        $asset_data["system"] = str_contains($asset->name, "arm64")?"darwin_arm64":"darwin_x86_64";//array("darwin_x86_64", "darwin_arm64");
    } else if(str_contains($asset->name, "x86_64-apple-darwin")){
        $asset_data["system"] = "darwin_x86_64";
    } else if(str_contains($asset->name, "aarch64-apple-darwin")){
        $asset_data["system"] = "darwin_arm64";
    } else if(str_contains($asset->name, "linux-amd64") || str_contains($asset->name, "x86_64-linux-gn")){
        $asset_data["system"] = "linux_x86_64";
    } else if(str_contains($asset->name, "linux-i686") || str_contains($asset->name, "i686-linux-gnu")){
        $asset_data["system"] = "linux_i686";
    }

[Jason2866]

@me-no-dev Thx for helping here :-)
@Lapshin

Wouldn't it be more efficient to parse all relevant information directly from https://github.com/espressif/esp-idf/blob/master/tools/tools.json? All the required information is available in one place.

That's not how Platformio works. If "stuff" is uploaded in Platformio Registry, Platformio provides the matching package for the Platform. The package.json is used locally when downloaded to verify that a package matches the requirements. If there is no package.json or it does not "fit" the package is withdrawn.
When the package is installed not from Platformio Registry the user has to take care to specify the correct one for his platform. When a false one is used, it will not get installed (checked via the package.json in the package).
So for manually installing (when not available in Platformio Registry) it is mandatory to identify the correct one via the file name.

In short, without a valid package.json included in a package it can not be installed.

[gerekon]

In short, without a valid package.json included in a package it can not be installed.

@Jason2866 Does it mean that package.json should be placed into archive? If yes that could lead to potential conflict if our toolchain will add file with the same name or other guy ask to add file with the same name to support another IDE.
The most universal way from my point of view is have PIO-specific script which users can use to prepare toolchain for that platform.

And all PIO-specific stuff will live there and will be updated when format/scheme changes.

[Jason2866]

@gerekon Yes, the package.json needs to be into the archive. I wouldn't have asked here to add if there is an other way. Since the package.json does not have strict reglementations about it contents, i see no problems if other IDE needs additional infos.
Platformio does not support what you suggest.
To be clear not having package.json inside will force every user wanting to use the toolchain to repack the package and save it "somewhere". Everything else than a nice user experience.

[gerekon]

Since the package.json does not have strict reglementations about it contents, i see no problems if other IDE needs additional infos.

Keeping info for multiple IDEs in one file will need some kind of coordination, at least we will have to place data for every IDE into separate structs (with own key name) to avoid keys names collisions. See below

{
 "PIO": {
 # PIO data go here
 },
 "CLion": { # just for example
 # CLion data go here
 }
}

Since PlatformIO does not support this structuring it makes no sense to go this way IMHO.

Platformio does not support what you suggest.

I am talking about supporting this in the script kept in separate repo which will handle all specifics of that platform and can be updated accordingly. There can be GH workflow jobs which will be triggered upon every Espressif toolchain release and produce archives for standalone usage with PlatformIO.
Now we have to duplicate archives in tar.gz format for Arduino support, I hope that we will remove that duplication in future when Arduino will support tar.xz.

So adding the 3rd version of the archives will make release a bit messy.

[Jason2866]

I see no problem for Platformio for this structure

{
 "PIO": {
 "name": "toolchain-xtensa-esp32",
  "version": "12.2.0+20230208",
  "description": "GCC Toolchain for Espressif ESP32 Xtensa MCUs",
  "keywords": [
    "build tools",
    "compiler",
    "assembler",
    "linker",
    "preprocessor",
    "espressif",
    "esp32"
  ],
  "license": "GPL-2.0-or-later",
  "repository": {
    "type": "git",
    "url": "https://github.com/espressif/crosstool-NG"
  },
  "system": "linux_x86_64"
 },
 "CLion": { # just for example
 # CLion data go here
 }
}

Since Platformio is searching for the entrys.

Duplication is what i want to avoid. Exactly this is actually needed when using the new toolchains. Copy them add the package.json and upload "somewhere"

I am talking about supporting this in the script kept in separate repo which will handle all specifics of that platform and can be updated accordingly. There can be GH workflow jobs which will be triggered upon every Espressif toolchain release and produce archives for standalone usage with PlatformIO.

A solution which is fine too. Imho not needed, because i think issues with conflicting entrys in package.json will not happen.

EDIT: Would be very happy to see a solution, every way is fine.

[Lapshin]

@Jason2866 , I wonder why you guys want to have system and version in package.json

Just run xtensa-esp32-elf-gcc --version from the package. And you will know:

1. If binary is built for the correct system
2. Version of toolchain

I mean, package.json is useless from my point of view at all

[Jason2866]

@Lapshin Dont ask me this, i am not a Platformio maintainer. It is done this way. No user choice to do different. I agree 100% Not happy how it is done.
No package.json ZERO possibility to install the package

[Jason2866]

I think package.json makes sense at all. Platformio does not only install toolchains. All kind of tools, frameworks and what else is needed to get a Multi Embed Platform IDE running.

[gerekon]

Since Platformio is searching for the entrys.

If another IDE will use entries with the same name it will lead to problems.

To tell the truth I am not fun of introducing dependency on 3rd party (higher level) tool into to the release archive. Looks like asking screwdriver's manufacturer to supply a special adapter for it to fit into custom toolbox :-). If I were PIO maintainer I'd provide such script for the developers who wants to use their IDE with not-registered tools.

Ok. To avoid keeping separate repo I think we can add a job to GH which will produce archive for PIO with that file and users will be able to run it manually and download the results. So no need to keep a lot of archives.
@Lapshin WDYT?

[Lapshin]

Sorry, but I don't fully understand the use case and motivation of having package.json in an archive.

I just executed strace pio pkg install --global --tool "espressif/toolchain-xtensa-esp32" and see from logs that pio is created package.json by itself...

openat(AT_FDCWD, "/home/jhordyess/.platformio/.cache/tmp/pkg-installing-dzbgq9zi/package.json", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 4
newfstatat(4, "", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0
ioctl(4, TCGETS, 0x7fff05682460)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR)                   = 0
write(4, "{\n  \"name\": \"toolchain-xtensa-es"..., 431) = 431
close(4)                                = 0

From this point of view, I have the following questions:

• Do you maintain your own pio-registry?
  • If yes, why don't you ask PlatformIO to do it properly?

If you want to use custom packages (not from the pio-registry), of course, it requires some extra action from the user. I don't see any advantage in providing a package.json for every toolchain distro because users must choose the correct one.

All related info could be obtained from the toolchain filename (system architecture + toolchain version). With this context, I see a better solution to have a small python script to generate the right package.json. (With using toolchain's filename as input parameter).
After package.json is generated, a user may continue repacking with instruction https://docs.platformio.org/en/latest/core/userguide/pkg/cmd_pack.html

BTW, you also may ask Platformio guys to implement automatic package.json generation from the archive filename with the command pio pkg pack [<source directory, tar.gz or zip>] [OPTIONS].

Or to ask PlatformIO to support new esp toolchains after they released (1-2 times a year).

[Jason2866]

@Lapshin Platformio does not upload new toolchains to the Platformio registry anymore. There is no afaik partnership contract anymore between espressif and Platformio.
To have the toolchains in the Platformio registry, the owner of the toolchains has to do.
This is espressif. To successfully upload the package to the Platformio registry the package.json has to be in the package. There is no auto generated package.json!
@me-no-dev uploaded the last toolchains and had to repack.
If you don't want have toolchains from "someone" uploaded in Platformio registry, you (espressif) should do.
A clear word from my side, i do not understand your behaviour here. Many developers of great projects with espressif MCUs do use Platformio. Why are you against making handling for all participants easier?

[gerekon]

Why are you against making handling for all participants easier?

@Jason2866 I understand your concerns. We are not against this. Above I wrote my concerns about having package.json in archives.
We will try to solve this in universal way.

[Lapshin]

@Jason2866 , please check the latest release. Is it good now or some changes are required?

[Jason2866]

@Lapshin Thank you. It does work :-)