Cannot use generic platform_packages = $URL with different OS

[mcspr]

Configuration

Operating system:
Any

PlatformIO Version (platformio --version):
abc0489

Description of problem

Original issues:
platformio/platform-espressif8266#224
esp8266/Arduino#6294

From the platform/espressif8266 issue, this is the local configuration for the current development version for esp8266 Core:
platformio.ini

[env]
platform = espressif8266
board = d1_mini
framework = arduino
platform_packages =
    framework-arduinoespressif8266 @ https://github.com/esp8266/Arduino.git
    toolchain-xtensa @ https://github.com/earlephilhower/esp-quick-toolchain/releases/download/3.0.0-gnu12/x86_64-linux-gnu.xtensa-lx106-elf-0474ae9.200706.tar.gz

[env:test]
src_build_flags = -DTEST

While this configuration is valid on Linux x86_64, it is not possible to re-use the same environment options on Windows.

My proposal is to either:

• add platform_repositories = ... setting with file:///... (or just filename, same directory), https:///... syntax, allowing the user to create the package manifest manually (as described in the https://docs.platformio.org/en/latest/platforms/creating_platform.html), thus allowing to specify multiple package options which could be filtered by OS.
• allow 'framework' package to somehow override package versions. For example, esp8266/Arduino already maintains Arduino IDE manifests, which could be read by PIO (or, read directly). Current implementation uses tools/get.py external script to read it's contents, which it turn downloads OS-specific package to be used with the arduino-cli
• edit: allow to also specify the "system" clause (linux_x86_64, darwin_x86_64 etc.). idk how this would happen in .ini though

As discussed in the esp8266/Arduino issue, this also creates problems with their CI system while building development versions. Current implementation simply re-uses tools/get.py result and replaces the ~/.platformio/packages/toolchain-xtensa with the up-to-date version, preserving the old package.json to avoid version conflicts
esp8266/Arduino#6294 (comment)

Additional info

It is possible to create a custom platform and provide a correct manifest.json. However this would also require maintaining differences between the real platform and the fork, as there is also python code shipped with the platform:
https://github.com/Jason2866/platform-espressif8266/tree/new_gcc

[mcspr]

As a POC:
develop...mcspr:feat/platform-repos

With manifest.json from @Jason2866 placed into the project dir or used via url:
https://github.com/Jason2866/platform-espressif8266/releases/download/3.0.0-gnu12/manifest.json

platformio.ini

[env:test]
platform = espressif8266
framework = arduino
board = d1_mini
platform_repositories =
    https://github.com/Jason2866/platform-espressif8266/releases/download/3.0.0-gnu12/manifest.json
platform_packages =
    toolchain-xtensa @ 5.10000.200706

There seems to be a typo: 5.10000.200706 is declared in the manifest, but the actual package version is 5.01010.200706

[Jason2866]

@mcspr not a typo. I got a error in platformio when using 5.01010.200706.
Seems this build numbering is not valid. So i did a (ugly) workaround to cheat Platformio

[mcspr]

Yeah, replacing the exact version match with the >=5 (as it is here https://github.com/Jason2866/platform-espressif8266/blob/b0167d26f3708396942ce6b663544e4984490ca5/platform.json#L39), I do see an error:

  File "/home/builder/git/platformio-core/platformio/managers/package.py", line 118, in max_satisfying_repo_version
    specver = semantic_version.Version(v["version"])
  File "/home/builder/.platformio/penv/lib64/python3.8/site-packages/semantic_version/base.py", line 105, in __init__
    major, minor, patch, prerelease, build = self.parse(version_string, partial)
  File "/home/builder/.platformio/penv/lib64/python3.8/site-packages/semantic_version/base.py", line 318, in parse
    raise ValueError("Invalid leading zero in minor: %r" % version_string)
ValueError: Invalid leading zero in minor: '5.01010.200706'

But it is a typo... from the esp-quick-toolchain side? Meaning 01010 should be 100100, same as the version uploaded into the PIO registry and conforming with semver:
https://semver.org/#spec-item-2 (quoting ...MUST NOT contain leading zeroes...)
https://bintray.com/platformio/dl-packages/toolchain-xtensa/2.100100.200706

[Jason2866]

Will this feature be added? If yes is there a time target?
There are new (beta) xtensa build chains for the ESP8266. For the ESP32 there are newer xtensa32 build chains too, which can not be used without OS dependency. We are a little bit stuck in Tasmota32 development because we would need features of the esp32s2 branch which needs newer/different xtensa32 toolchains
As interims solution, is there a possibility to do a custom Toolchain Package build?
The old way i did before is not working anymore or has this been changed only with v.5.?

[mcspr]

Bump? Effectively, espressif8266 docs are invalid when user wants to use git version of the framework:
https://docs.platformio.org/en/latest/platforms/espressif8266.html#using-arduino-framework-with-staging-version

There's also was a semi-recent version bump of the toolchain to 10.2:
https://github.com/earlephilhower/esp-quick-toolchain/releases/tag/3.0.0-gnu13

I don't quite get the removal of packageRepositories :/ Is it expected that esp8266/Arduino team maintains the package, but then we need a different author-prefix there? Or do we need a volunteer to track & upload the package(s), when needed? (although, no-one from the esp8266/Arduino core team actually uses PIO, so... I guess I can volunteer?)

[Jason2866]

@mcspr Do you know if there is a way to do a user toolchain package?

[mcspr]

@Jason2866 my guess would be by uploading all xtensa-lx106 tarballs from the link above, then the package would be installed via platform_packages = ... mcspr/toolchain-xtensa@5.100200.200918, since it would resolve the correct OS requirement by then?

I have tried uploading the linux one:

$ pio package pack x86_64-linux-gnu.xtensa-lx106-elf-2f311fd.200918.tar.gz
Wrote a tarball to ".../toolchain-xtensa-linux_x86_64-5.100200.200918.tar.gz"
$  pio package publish toolchain-xtensa-linux_x86_64-5.100200.200918.tar.gz
The package has been accepted for moderation. We will notify you on email when process it.

pio access list does not show anything, yet, so right now I just wait. In the mean time, I also added the Windows one

edit: not sure if the pack step is required though, since we can use the original .tar.gz already

[Jason2866]

You will get a mail if it is accepted. Before nothing can be seen.
Mhh, didnt try just to upload one tarball. If it would be so easy to do one by one for every OS
We will see. Moderation took a day for my last package

[Jason2866]

@mcspr Good and bad news Linux does work platform_packages = mcspr/toolchain-xtensa@5.100200.200918.
Windows does not

Error: Could not find the package with 'mcspr/toolchain-xtensa @ 5.100200.200918' requirements for your system 'windows_amd64'

Maybe the problem is using the same version number 5.100200.200918 PlatformIO docs say not allowed to use twice for package publish
@ivankravets How to publish the needed different OS toolchains with the same version number?

[ivankravets]

Why do you use strict patch version? Please use rules described in our docs for platform install and lib install commands.

You need to use ~

[Jason2866]

Thanks for the hint. I want to use this specific version no other. This should work?
For the Linux variant is does work.

[ivankravets]

platform_packages = mcspr/toolchain-xtensa@~5.100200.200918

[Jason2866]

Same error

Error: Could not find the package with 'mcspr/toolchain-xtensa @ ~5.100200.200918' requirements for your system 'windows_amd64' ```

[ivankravets]

Sorry, this
platform_packages = mcspr/toolchain-xtensa@~5.100200.0

Or

platform_packages = mcspr/toolchain-xtensa@^5

[mcspr]

re. the above - I accidentally only pushed the i686 version, I published the x86_64 Windows though so there's another waiting time :oops:

Since bintray mirror is also gone, can't actually see the uploads to track system
(+1 to the #3711)

[mcspr]

...but I can use the json API directly though (maybe I am missing some other CLI command there? can't seem to find any other use of list_packages):

diff --git a/platformio/commands/access.py b/platformio/commands/access.py
index d9fb3970..be00b8d7 100644
--- a/platformio/commands/access.py
+++ b/platformio/commands/access.py
@@ -122,7 +122,11 @@ def access_revoke(client, urn, urn_type):
 @click.option("--json-output", is_flag=True)
 def access_list(owner, urn_type, json_output):
     reg_client = RegistryClient()
-    resources = reg_client.list_resources(owner=owner)
+    #resources = reg_client.list_resources(owner=owner)
+    result = reg_client.list_packages("toolchain-xtensa", filters={"owners":["mcspr"]})
+    print(result)
+    import sys
+    sys.exit()
     if json_output:
         return click.echo(json.dumps(resources))
     if not resources:

Right now there are:

... after `pio access list` ...
>>> res = {'page': 1, 'limit': 10, 'total': 1, 'items': [{'id': 11401, 'type': 'tool', 'owner': {'username': 'mcspr'}, 'name': 'toolchain-xtensa', 'description': 'xtensa-gcc', 'keywords': [], 'added_at': '2020-11-03T13:11:27Z', 'updated_at': '2020-11-03T13:14:02Z', 'unique_downloads_count': 2, 'version': {'id': 30985, 'name': '5.100200.200918', 'unpacked_size': 253788019, 'released_at': '2020-11-03T13:11:27Z', 'files': [{'name': 'toolchain-xtensa-linux_x86_64-5.100200.200918.tar.gz', 'size': 73973292, 'checksum': {'sha256': '7c82507b6dce67bdd991d399ea6382660e7fd7d09a9d3a10ddcc4c68a87ca525'}, 'system': ['linux_x86_64'], 'download_url': 'https://dl.registry.platformio.org/download/mcspr/tool/toolchain-xtensa/5.100200.200918/toolchain-xtensa-linux_x86_64-5.100200.200918.tar.gz'}, {'name': 'toolchain-xtensa-windows_x86-5.100200.200918.tar.gz', 'size': 69444241, 'checksum': {'sha256': '669c83813688c215fb449d24599b3227a2b9c10d2d72058616d37ae65a2b6dd4'}, 'system': ['windows_x86'], 'download_url': 'https://dl.registry.platformio.org/download/mcspr/tool/toolchain-xtensa/5.100200.200918/toolchain-xtensa-windows_x86-5.100200.200918.tar.gz'}, {'name': 'toolchain-xtensa-darwin_x86_64-5.100200.200918.tar.gz', 'size': 74645844, 'checksum': {'sha256': '0cecee528cbae807ce50f2bdc25b03d69932b2f3c9478f5382dde2c2f114f896'}, 'system': ['darwin_x86_64'], 'download_url': 'https://dl.registry.platformio.org/download/mcspr/tool/toolchain-xtensa/5.100200.200918/toolchain-xtensa-darwin_x86_64-5.100200.200918.tar.gz'}]}}]}
>>> [x['system'] for x in res['items'][0]['version']['files']]
[['linux_x86_64'], ['windows_x86'], ['darwin_x86_64']]

[Jason2866]

The are listed here too
@mcspr For my use i see no need for this feature request anymore. Can be solved with publish packages.

[ivankravets]

The package mcspr/toolchain-xtensa for x86_64 has been approved.

[Jason2866]

Can confirm. It does work :-)
Thx @ivankravets for your help (again!) and @mcspr for the doing (i thought to much complicated...)

[mcspr]

True, feels like the original issue is resolved.

@ivankravets from the qs' above:

• do we need to pio package pack the package archives from the releases at esp-quick-toolchain? at a glance, only difference I see between original and repacked is removal of 'xtensa-lx106-elf/' subdir before the files
• can espressif8266 docs reference these packages? (to at least avoid additional google search as noted here https://gitter.im/esp8266/Arduino?at=5f829da899e1ab4dd1f0cb92)

[ivankravets]

do we need to pio package pack the package archives from the releases at esp-quick-toolchain? at a glance, only difference I see between original and repacked is removal of 'xtensa-lx106-elf/' subdir before the files

Is this a bug? Our tools are open source. See https://github.com/platformio/platformio-core/blob/develop/platformio/package/pack.py

can espressif8266 docs reference these packages?

Sorry, I'm outdated from the latest dev-platform news. @valeros will help.

[mcspr]

I meant packing does this to the file:

diff --git a/orig.txt b/packed.txt
index 9160320..94ed7f7 100644
--- a/orig.txt
+++ b/packed.txt
@@ -1,1613 +1,1517 @@
...
-xtensa-lx106-elf/package.json
+package.json
...
-xtensa-lx106-elf/bin/xtensa-lx106-elf-addr2line
+bin/xtensa-lx106-elf-addr2line
... and etc...

Just removing the additional subdir. I meant, does it matter for the publishing if subdir stays there?

[ivankravets]

The root of package is a place where manifest exists.