mkbuildoptglobals.py: assert python version

[mcspr]

resolve #8877
also fix spurious warning when getdefaultencoding() is used with versions ≥3.11

[blue-wind-25]

Hello,

I was thinking if is it possible to make Arduino ESP8266 able to detect the required Python3 executable name; for example, in my system by executing:

compgen -c python | grep -P '^python(\d(.\d+)?)?$' | sort | uniq

I got:

python
python2
python2.7
python3
python3.11
python3.4
python3.6

The problem of course, it does not mean that python3 is actually a symbolic link to python3.11 because in my case python3 is the default Python3 used by the system (and it is not python3.11).

Anyway, it is just an idea because I am not sure how difficult it is to implement the detection in Arduino.

Thank you!

[mcspr]

Depends on whether we want to do this magic auto-detection of latest version vs. having user simply replace the PATH temporarily to give a specific one. From venv, custom script launcher, different shell environment, etc.

Current approach is packaging this launcher
https://github.com/earlephilhower/esp-quick-toolchain/blob/master/blobs/python3-via-env.tar.gz

compgen is shell, while we do this thing with execv call

#!/usr/bin/env python3
import os
import sys

args = ["env", "python3"] + sys.argv[1:];
os.execv("/usr/bin/env", args)

We can't (?) modify menu of IDE dynamically based on available python versions, so I'd rather leave the python3 as the only entrypoint

[blue-wind-25]

Hmmm, yes I think you are correct. There seems to be no method to modify menu of IDE dynamically based on available python versions and because the python executable names can be anything we also cannot list all the possible combinations in the menu.

Perhaps, using a special environment variable can also help. Lets say if ARDUINO_PYTHON is defined, the launcher can use that as the entry point name instead of python3, but I am not sure if this approach can simply be used with this ESP8266 core or should actually be specified further as a more generic Arduino IDE feature.

[mcspr]

So before replacing any scripts / adding our env we have either

• python3.11 -mvenv somewhere -> using script that sources somewhere/bin/activate and calls IDE. PATH is prepended with somewhere/bin so python3 calls python3.11
• runtime.tools.python3.path build property replacement through cli, where {runtime.tools.python3.path}/python3 is the correct version e.g. arduino-cli compile -b esp8266com:esp8266:generic:eesz=autoflash --build-property runtime.tools.python3.path=/my/python/path/prefix
• same as above, platform.local.txt placed near existing platform.txt that has a runtime.tools.python3.path=/my/python/path/prefix line

python executable is our private thing here, so maybe ESP8266_ARDUINO_PYTHON_PATH if environment tweak is more coherent solution to document

[blue-wind-25]

• So by using python3.11 -mvenv somewhere, essentially the whole IDE will run with a modified PATH? Can this introduce problems in the future if somehow the modified path interfere with something else later?
• In my opinion using runtime.tools.python3.path may not be easy/convenient for all users, but it might be better (safer?) than modifying path for the whole IDE.
• Using platform.local.txt may be easier than the the 2nd solution above, assuming any modification to the file does not get deleted when the ESP8266 package is updated, but still in POSIX, the .arduino15 directory is meant to be a hidden directory and in Windows the actual locations is long (C:\Users\Name\AppData\so_on_so_forth). It may also not endearing to some users.

Finally because the env var ESP8266_ARDUINO_PYTHON_PATH will be for this package only, it may be the safest and easiest for users. It can be (semi) permanently set in bashrc in Linux, or its equivalent if using zsh in MacOS and in Windows should be from Control Panel. Because the env var is for our private use and our python launcher will revert back topython3 as the entry point if the env var is not defined, I think this can be the safest solution and simplest to document.

Yes I know in Windows, the package also carries its own copy of python3 executable and DLL. I tested them and I found that they do not run on Win7 without SP1. I also remembered that I somehow manage to download python3 >= 3.7 that will work on plain Win7 (surprisingly Arduino IDE 2.0.x can be made to run in plain Win7).

I am pretty sure that plain Win7 is no longer used widely, but using env var still may provide better flexibility for users with custom python installation path (it should still work in Win10 and 11, right?)

[mcspr]

Another funny way to override is to make a custom tool directory. It even overrides the local git installation tools/

> arduino-cli compile -b esp8266com:esp8266:generic --show-properties | grep tools.python3.path=
runtime.tools.python3.path={runtime.platform.path}/tools/python3
> mkdir -p ~/.arduino15/packages/esp8266com/tools/python3/9.9.9
> touch ~/.arduino15/packages/esp8266com/tools/python3/9.9.9/python3
> arduino-cli compile -b esp8266com:esp8266:generic --show-properties | grep tools.python3.path=
runtime.tools.python3.path=/home/runner/.arduino15/packages/esp8266com/tools/python3/9.9.9

Not sure if IDE / CLI would care some time later when updating things, since this package version would not be referenced by any platform.

https://github.com/earlephilhower/esp-quick-toolchain has to be updated for env var change
(or our packaging .json modified to point some place else for python)

[blue-wind-25]

Well, so it seems the IDE and CLI do have some feature to detect (compare?) version by directory name. Of course, we can not be sure that this feature (bug?) will be consistent for future releases.

The python launcher is only a small file. Is it necesarry to make a full new release after modifications to https://github.com/earlephilhower/esp-quick-toolchain for env var change? Or maybe only one of the release assets can be updated? It seems a bit weird and time consuming to re-release the whole quick toolchain again with only some very small changes. If so, maybe changing the json file is the better method, at least for now?

[mcspr]

Spoke too soon, above is no longer the case for latest -cli versions
https://arduino.github.io/arduino-cli/0.31/package_index_json-specification/#tools-definitions
arduino/arduino-cli@b894e12#diff-e7150dca3b87d28bf9177589951b5ac21829768a9d1d621bc2cf69c32a3524f5

No need to re-release everything, just a tag for the commit that updates python archive which then can be fetched with a permanent link. Replacing existing assets breaks old versions.

[blue-wind-25]

No need to re-release everything, just a tag for the commit that updates python archive which then can be fetched with a permanent link. Replacing existing assets breaks old versions.

That is good news. Thank you for your help and attention!

[blue-wind-25]

I have tested the env var using manually installed https://github.com/earlephilhower/esp-quick-toolchain/releases/download/3.2.0-gcc10.3/python3-via-env.tar.gz. It is working properly in my machine. Thank you again for your help and attention.