etl outdated on platformio registry

[positron96]

ETLCPP has 2 versions on platformio registry: arduino one (which is up-to-date), and non-arduino one. The latter is 19.3.5 and has not been updated for a while. Is it possible to update it?
I'm using it for CubeMX project, and seemingly I don't want an arduino version of the library (though I don't exactly know what the differences are).

[jwellbelove]

I will try to look at the PlatformIO registry issue this weekend. The difference between the normal and Arduino versions is a special header file required to get the Arduino IDE to recognise the library, and a slightly different directory structure that is mandated by the Arduino library system. The Arduino version does not include the unit tests.

[PetteriAimonen]

As a workaround until the registry is updated, one can use this in platformio.ini:

lib_deps =
     "Embedded Template Library"=https://github.com/ETLCPP/etl.git

[jwellbelove]

I'm having a little trouble updating the PlatformIO registry at the moment.
The email I am able to log on with has no projects associated with it.
The email that is listed on the 'etlcpp/Embedded Template Library' page does not result in a password reminder email when I try pio account forgot.
It's such a long time since I interacted with PlatformIO, I can't remember how I accessed it.

[PetteriAimonen]

Maybe @ivankravets can help. He did help me when I had the same trouble with nanopb :)

[ivankravets]

@jwellbelove I don't see an account with jwellbelove username in the registry. Could you try to login to PlatformIO with your Github account? So, we can make you an owner of etlcpp organization.

How to log in via Github? The easiest way is to start pio home, click on "account" icon (top right corner), and select Github.

[jwellbelove]

@ivankravets
OK, done.
The two registry entries are etlcpp/Embedded Template Library - Arduino` and 'etlcpp/Embedded Template Library'.

[ivankravets]

@jwellbelove thanks! Added you as an owner of etlcpp organization. You can check this via pio org list command.

To solve this issue we need to decide what to do with a library name. It has Embedded Template Library in the registry, the library.manifest has Embedded Template Library ETL which makes problems for our package processor. It will not process a package that has manifests with a different name or version fields.

So, what is the final library name? If you try to publish https://github.com/ETLCPP/etl/tree/20.22.0 now, it will not update https://registry.platformio.org/libraries/etlcpp/Embedded%20Template%20Library and will create a totally new package due to a different name.

[jwellbelove]

The general ETL library is named Embedded Template Library and the version for Arduino is named
Embedded Template Library - Arduino.

https://registry.platformio.org/search?q=author%3A%22John+Wellbelove%22

[ivankravets]

There are 2 manifests in the repository. Could I ask you to normalize names? We need fixes for both manifests.

You can then clone 20.22.0 branch, remove library.properties file, update name in library.json, use pio package publish —owner etlcpp.

[jwellbelove]

I'm not sure what you mean by 'normalising the names'.
One is named Embedded Template Library and the other Embedded Template Library - Arduino.
Are you saying that the non-Arduino version should not have the library.properties file.
Where is the manifest file? Is it editable by me?

[jwellbelove]

I noticed that pio org list only shows one library.

etlcpp
------
Display Name:  Embedded Template Library
Owners:        jwellbelove

[ivankravets]

I see, you updated manifests in the development branch. It looks great now. When do you plan to bump 20.23.0?

[jwellbelove]

Today

[ivankravets]

pio org list shows companies that you own. You can change the company name “ Embedded Template Library” via pio org update command.

To list owned packages, please use pio access list

[jwellbelove]

OK, I get it now. I'll have to try to familiarise myself with PlatformIO a bit more in the future.

[ivankravets]

It seems the issue can be closed. The legacy crawler on the PlatformIO side updated library to the latest version https://registry.platformio.org/libraries/etlcpp/Embedded%20Template%20Library

@jwellbelove, thanks for the help.

[positron96]

This is great! Thank you guys!

[jwellbelove]

One small question.
When I push a new release, do I have to pio package publish each time, or does the PIO registry automatically update?

[PetteriAimonen]

@jwellbelove There are two ways:

1. Using the "legacy library crawler" which automatically checks the library.json file in git repository and automatically updates registry if it sees the version number has increased.
2. Using pio package publish every time.

I think the 2. version is what is recommended nowadays.

One can setup some github workflow to do it automatically on some condition (such as pushing new tag). I've been meaning to do the same for nanopb, @hacker-cb set up some rules here which may be helpful: https://github.com/nanopb/nanopb_cpp/blob/master/.github/workflows/deploy-platformio.yaml

[ivankravets]

@PetteriAimonen thanks for the advanced comment 🙏

@jwellbelove yes, legacy crawler fetches new releases automatically. Once use one-time pio package publish (manual publishing, recommended), "legacy crawler" will be disabled for this library. You will need to re-run this command on the next release.

pio package publish --owner etlcpp

[jwellbelove]

It appears that there are two different owners for the normal and Arduino registry entries at the moment.
I'm sure they didn't have different owners before.

jwellbelove/Embedded Template Library - Arduino
ETL. A C++ template library tailored for embedded systems. Directories formatted for Arduino

etlcpp/Embedded Template Library
ETL. A C++ template library tailored for embedded systems.

I'm currently having issues with the normal (non Arduino) ETL library publish.
"Error: File size exceeds your limit: 50 MB"
Is there a way to tell it to ignore certain sub-directories?

[jwellbelove]

I think that the etlcpp one is incorrect as when I publish from here, it states this...

Type:     library
Owner:    jwellbelove
Name:     Embedded Template Library
Version:  20.24.1

[ivankravets]

I'm currently having issues with the normal (non Arduino) ETL library publish.

We have a related feature request, platformio/platformio-core#4188
Let me quickly implement it. I'll be back soon.

Meanwhile, you can ignore unnecessary files with https://docs.platformio.org/en/latest/librarymanager/config.html#export

[ivankravets]

@jwellbelove support for .gitignore was added in platformio/platformio-core@6f11f81

Please run pio upgrade --dev. You will be upgraded to the latest development version of PlatformIO Core. Change current working dir in a shell to the library root and run pio pkg pack -o path/to/exported.tar.gz. Did PlatfomIO exclude correctly all files?

[jwellbelove]

Thanks, I'll give that a try.

[jwellbelove]

I get

  error: [Errno 13] Permission denied: 'c:\\python\\python39\\Scripts\\pio.exe'
  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

Encountered error while trying to install package.

platformio

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
WARNING: Ignoring invalid distribution -latformio (c:\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution -latformio (c:\python\python39\lib\site-packages)

I've tried with administrator rights, but the same result.
I've updated pip.

[ivankravets]

I see. Please use pip install https://github.com/platformio/platformio-core/archive/refs/heads/develop.zip

[jwellbelove]

Same result.

[ivankravets]

Do you use PlatformIO IDE (VSCode) or bare CLI?

[jwellbelove]

I've been updating through the terminal.

[ivankravets]

It looks like there is python.exe process that blocks updating process. Please kill all python.exe processes and repeat pip install https://github.com/platformio/platformio-core/archive/refs/heads/develop.zip

[jwellbelove]

Well, I've checked to instances of python running (none) and I've rebooted, but nothing has changed.

[ivankravets]

Could you manually remove c:\\python\\python39\\Scripts\\pio.exe and pip install https://github.com/platformio/platformio-core/archive/refs/heads/develop.zip?

Sorry, I'm a macOS user and do not know all security details regarding Windows OS.

[jwellbelove]

Worked!

[ivankravets]

Does pio pkg pack produce stripped tar.gz?

[positron96]

Sorry to say that, but there appeared another problem with platformio. When pio downloads recent 20.24.1 version, (https://registry.platformio.org/libraries/etlcpp/Embedded%20Template%20Library) it's missing library.json file for some reason.
That file contains important line "includeDir": "include", and without it all cpp files that have #include <etl/anything.h> (e.g. etl/vector.h) fail because the compiler doesn't add include folder to search path. These includes worked before in 19.3.5 version.

20.24.0 and 20.23.0 are affected as well, while 19.3.5 works as expected.

[jwellbelove]

The tar.gz file is stripped of some files, the problem is that it is missing all of the files in the test directory and every sub-directory beneath it.

[jwellbelove]

library.json does appear in my tar.gz file.

[positron96]

Is it in the root? When pio downloads from registry, there is one library.json inside arduino folder, but I assume it's used for arduino variant of the library.

If this is any help, here is what's in the library after download:

$ du -hs * .[!.]* | sort -k2
4.0K    .circleci
4.0K    .clang-format
4.0K    .gitattributes
13K     .github
8.0K    .gitignore
1.0K    .gitlab-ci.yml
0       .gitmodules
1.0K    .piopm
4.0K    CMakeLists.txt
104K    Doxyfile
4.0K    LICENSE
8.0K    README.md
1.0K    appveyor.yml
19K     arduino
385K    examples
0       file.txt
40K     images
5.2M    include
1.0K    library.properties
4.0K    meson.build
1.0K    meson_options.txt
1.0K    subprojects
108K    support
5.0K    temp
1.0K    todo.txt
20K     uml

[jwellbelove]

This is my local directory.

[jwellbelove]

Could it be picking up the Arduino version over the one in the product root?

[ivankravets]

Guys, sorry for the delay. I could not imagine that will wake up in my life from the rocket strikes. I’ll be back soon with ideas. Glad to fix any issues related to the PlatformIO as soon as possible.

🙏

[jwellbelove]

Stay safe!

[ivankravets]

@jwellbelove sorry for the delay, see this PR #514

The idea is to declare manually what PlatformIO should export when packaging a library. Please merge it and run pio pkg pack. Does it look good now?

[Compizfox]

Sorry for bumping this, but I'm a bit confused about which library to use.

I had etlcpp/Embedded Template Library - Arduino in my lib_deps, but this library does not exist anymore.

Looking at the PIO registry, I see the following three libraries:

• etlcpp/Embedded Template Library
• jwellbelove/Embedded Template Library - Arduino
• jwellbelove/Embedded Template Library

What is the difference between the first and the third? And do I need the Arduino version if I'm using the Arduino framework, but on a different board (Teensy) with PlatformIO (and thus not with the Arduino IDE)?

Thanks in advance!

[jwellbelove]

There's been a bit of confusion over which one was definitive. Somehow I ended up with two accounts, though I don't seem to be able to access the etlcpp one anymore (even to delete it, though I would prefer it to be the default).
At the moment the jwellbelove libraries are current.

[Compizfox]

Thanks for clearing that up!

For my other question, am I right that the Arduino version is only required when using the Arduino IDE?

[jwellbelove]

Yes, the Arduino version has the directory structure mandated by the Arduino IDE.
It's also missing the unit tests.

[Compizfox]

Thanks!

[jwellbelove]

@ivankravets Any idea how I can get access to the other account 'etlcpp'
I try poi account forgot, but I get no email.
I think PlatformIO has got confused somehow about the ownership.

Publishing to the jwellbelove account, I of course get Warning! A package with the name Embedded Template Library is already published by the next owners: etlcpp

[ivankravets]

@jwellbelove the etlcpp is an organization, not a separate account. You are the owner of this organization. Could you check it via pio org list? To manage org's packages, please use pio package publish --owner etlcpp.

[ivankravets]

@Compizfox we have just removed jwellbelove/Embedded Template Library from the registry, please use etlcpp/Embedded Template Library or https://registry.platformio.org/libraries/jwellbelove/Embedded%20Template%20Library%20-%20Arduino

[jwellbelove]

OK, I thought the jwellbelove and etlcpp were both accounts. I didn't know there was a difference.
Right, what I am going to do is (what I originally intended) remove the packages from the jwellbelove account and republish them under the etlcpp organisation.

[ivankravets]

@jwellbelove we will move them, no need to republish. Next time please use pio package publish --owner etlcpp. If you don't specify organization, the package will be published from your personal account - jwellbelove

[jwellbelove]

Thanks, I understand now.

[ivankravets]

It seems now all packages look good in the registry https://registry.platformio.org/search?q=owner%3Aetlcpp
They are owned by etlcpp.

@jwellbelove thanks for the amazing library and your contribution to the OSS 🙏
Please don't hesitate to ping me for any help.

[jwellbelove]

Awesome, thanks for all your help.