Binaries on PyPI only support Python 3.9 #128

BenjaminRodenberg · 2021-05-12T10:38:57Z

I installed pythonfmu from PyPI via pip3 install --user python3fmu. However, when I tried to use the generated FMU OMEdit crashed. After some investigations I found out that the binaries provided on PyPI are only compatible with Python 3.9 (as far as I understand). According to the documentation pythonfmu should work with any python3 distribution, if I don't understand it wrong.

Why only python 3.9?

I downloaded pythonfmu-0.6.2-py3-none-any.whl from here, unpacked it and ran ldd on libpythonfmu-export.so:

/pythonfmu-0.6.2/pythonfmu/resources/binaries/linux64# ldd libpythonfmu-export.so 
	linux-vdso.so.1 (0x00007ffdb51e8000)
	libpython3.9.so.1.0 => not found
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0b76d8e000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0b76d73000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0b76b81000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0b76a32000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0b771aa000)

Further details

I assume that this is due to the way how the binaries of pythonfmu are generated in this action.

I'm not sure whether this is something that one can change easily. Distributing binaries for all python platforms is probably out of scope.

I finally fixed this issue by cloning this repository, building the binaries myself via cmake & make in pythonfmu/pythonfmu-export and installing pythonfmu via setup.py.

The text was updated successfully, but these errors were encountered:

markaren · 2021-05-12T10:41:38Z

It should work in Python 3.6 <= as the ABI is kept binary compatible. I for one, use Python 3.7.
However, I'm on Windows, so there might be a difference there.

BenjaminRodenberg · 2021-05-12T10:58:21Z

Thanks for the reply! On Ubuntu 20.04 I ran into trouble with the system python (3.8) when trying to use the FMU in OMEdit. I can try to reproduce the issue on my system and post the error here.

BenjaminRodenberg · 2021-05-12T11:38:54Z

I generate the example PythonFMU/examples/demoslave.py. Once with pythonfmu from PyPI, once with the version I have built using this repository.

In both cases I ran

~/PythonFMU/examples$ pythonfmu build -f demoslave.py
Extracted Resistor.fmu
cd Resistor_{github,pypi}/binaries/linux64

I will describe the behavior below.

PyPI

Importing in OMEdit gives many errors and I cannot "simulate".

[1] 13:33:47 Scripting Error
[/home/benjamin/.openmodelica/libraries/index.json: 0:0-0:0]: The package index /home/benjamin/.openmodelica/libraries/index.json could not be parsed.

[2] 13:33:47 Scripting Error
Failed to load package Modelica (default) using MODELICAPATH /usr/bin/../lib/omlibrary:/home/benjamin/.openmodelica/libraries/

[3] 13:33:47 Scripting Error
[/home/benjamin/.openmodelica/libraries/index.json: 0:0-0:0]: The package index /home/benjamin/.openmodelica/libraries/index.json could not be parsed.

[4] 13:33:47 Scripting Error
Failed to load package ModelicaReference (default) using MODELICAPATH /usr/bin/../lib/omlibrary:/home/benjamin/.openmodelica/libraries/.

[5] 13:34:47 Scripting Error
[fmiLogger] module FMICAPI: Could not load the FMU binary: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

[6] 13:34:47 Scripting Error
[instantiate] Could not load "resources/0001_Resistor_pypi.fmu" which is associated with "test.Root.Resistor_pypi"; it may be corrupted or may not support your platform

[7] 13:34:47 Scripting Error
[initialize] Model "test" is in wrong model state

[8] 13:34:47 Scripting Error
[terminate] Model "test" is in wrong model state

libpython3.9.so.1.0 is not found by ldd

~/PythonFMU/examples/Resistor_pypi/binaries/linux64$ ldd Resistor.so 
	linux-vdso.so.1 (0x00007fff7e796000)
	libpython3.9.so.1.0 => not found
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc210351000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc210336000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc210144000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc20fff3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc2107b7000)

This Repo (github)

Importing in OMEdit gives fewer errors, "simulate" works:

[1] 13:36:08 Scripting Error
[/home/benjamin/.openmodelica/libraries/index.json: 0:0-0:0]: The package index /home/benjamin/.openmodelica/libraries/index.json could not be parsed.

[2] 13:36:08 Scripting Error
Failed to load package Modelica (default) using MODELICAPATH /usr/bin/../lib/omlibrary:/home/benjamin/.openmodelica/libraries/.

[3] 13:36:08 Scripting Error
[/home/benjamin/.openmodelica/libraries/index.json: 0:0-0:0]: The package index /home/benjamin/.openmodelica/libraries/index.json could not be parsed.

[4] 13:36:08 Scripting Error
Failed to load package ModelicaReference (default) using MODELICAPATH /usr/bin/../lib/omlibrary:/home/benjamin/.openmodelica/libraries/.

[5] 13:36:35 Scripting Notification
Result file: test_res.mat (bufferSize=10)

libpython3.8.so.1.0 is found by ldd

~/PythonFMU/examples/Resistor_github/binaries/linux64$ ldd Resistor.so 
	linux-vdso.so.1 (0x00007ffcd016e000)
	libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f06e0cdc000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f06e0afb000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f06e0ae0000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f06e08ee000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f06e08be000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f06e08a2000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f06e087f000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f06e0879000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f06e0874000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f06e0725000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f06e1367000)

markaren · 2021-05-12T11:55:28Z

Yeah, I can replicate through WSL (Ubuntu 18.04).
I guess the same issue will appear when using e.g. 3.6 to build the binary.

markaren · 2021-05-12T12:00:30Z

This may be the reason why it work on different Python versions on Windows:

if (WIN32)
  set(Python3_LIBRARIES ${Python3_LIBRARY_DIRS}/python3.lib)
endif ()

I dunno if something similar is possible to do for Linux.

BenjaminRodenberg · 2021-05-12T12:52:06Z

I think libpython3.so has a similar purpose on UNIX systems (see here: https://stackoverflow.com/a/42388242). I unsuccessfully tried the following modification:

# Force to use stable Python ABI https://docs.python.org/3/c-api/stable.html
add_compile_definitions(Py_LIMITED_API)
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
if (WIN32)
  set(Python3_LIBRARIES ${Python3_LIBRARY_DIRS}/python3.lib)
else (UNIX)
  set(Python3_LIBRARIES ${Python3_LIBRARY_DIRS}/libpython3.so)
endif ()

I got stuck at the point that libpython3.so could not be found and I also don't know how to get it.

markaren · 2021-05-12T17:57:14Z

I ran the locate command on my Ubuntu installation and there is no libpython3.so..
So I'm kind of blank as to facilitate this for linux users. It is clearly an issue as the FMUs generated for that platform will have major portability issues.

markaren · 2021-05-13T15:35:37Z

Ok, so now I'm a little bit confused... You see, the CI compiles the binaries once. Presumably using 3.9..
Tests are then carried out on 3.7, 3.8 and 3.9 using the same binary. They all run and they all pass (Ubuntu and Windows)..

BenjaminRodenberg · 2021-05-18T07:57:41Z

Ok, so now I'm a little bit confused... You see, the CI compiles the binaries once. Presumably using 3.9..
Tests are then carried out on 3.7, 3.8 and 3.9 using the same binary. They all run and they all pass (Ubuntu and Windows)..

Do you know whether on the CI instance executing the tests libpython3.9.so.1.0 is present or not? I can imagine that if a test is carried out with (for example) python 3.7 while both 3.7 and 3.9 are installed on the system it might still work. However, if libpython3.9.so.1.0 is not present (which is what I would assume for the general case when somebody runs python 3.7) I assume the tests should fail.

I once also was in the situation that both python 3.8 and 3.9 were installed on my system and everything worked nicely with python 3.8, if I remember correctly.

markaren · 2021-05-18T09:06:23Z

Yeah, the virtual environment has:

2.7.18
3.5.10
3.6.13
3.7.10
3.8.10
3.9.5

HenningScheufler · 2021-07-22T07:25:50Z

I could also reproduce this issue on ubuntu 20.04 using a miniconda with a python version 3.8:

Following output is produced:

pyfmi.fmi.InvalidBinaryException: The FMU could not be loaded. Error loading the binary. Could not load the FMU binary: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

After creating a new conda environment 3.9:

conda create -n py39 python=3.9
conda activate py39
pip install pythonfmu
conda create -n py39 python=3.9

The problem persist despite using the correct python version. However, if we add python 3.9 to our the library path this issue can be resolved

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:ANACONDA_PATH/envs/py39/lib/

Why is python 3.9 linked by default on ubuntu?

markaren · 2021-07-22T09:14:56Z

As far as I can tell, there is no way for PythonFMU to support more than one python distribution on Ubuntu as it links to a specific version of Python. Windows does not.. I guess Python 3.9 is the version used by default on GitHub Actions (Python 3.x) so that is the one that is linked. The only way to make it work with a different Python on Linux is to manually build the library and replace the .so.

HenningScheufler · 2021-09-24T20:00:58Z

The python interpreter gets found by:
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
However, this line (tested with cmake version 3.21.3) always find the newest interpreter which explains why it always links to 3.9.

After adding following line

set(Python3_FIND_STRATEGY LOCATION) # added requires cmake 3.15
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)

the activated interpreter is used

ubuntu 1804 only comes with cmake 3.10 upgrading to ubuntu 2004 should change the cmake version to 3.16

2018-Summer · 2021-12-10T06:22:17Z

I also have this problem, how to solve this problem finally, do you need to upgrade python version to 3.9? I'm currently using version of Python =3.6.9

markaren · 2021-12-10T07:40:09Z

To resolve the issue locally, you can

Clone the repo
Run pythonfmu/pythonfmu-export/build_unix.sh
Copy the generated .so from pythonfmu/resources into your FMU

2018-Summer · 2021-12-13T02:53:48Z

[FMI native bridge] Fatal: Failed to load library 'C:\Users******\AppData\Local\Temp\fmi4j_dnn3437415721011360530\binaries\win64/dnn.dll', error: 126

Fmu generated under Windows, but in Java environment to call down my compiled FMU, the above error, is the compilation binary file error?

markaren · 2021-12-13T05:24:48Z

@2018-Summer Windows is not related to this issue, please keep the posts relevant. But that error is stemming from a missing dll on PATH.

…663c102e24fd521f7 seems to be a workable fix to NTNU-IHB#128

jschueller · 2023-11-09T07:44:32Z

this should be fixed now with #187

jschueller · 2023-12-01T16:32:15Z

@markaren I dont see new binaries on pypi, why did you close this ?

markaren · 2023-12-01T16:42:14Z

You are right, this issue is specifically about pypi

jschueller · 2023-12-01T16:45:03Z

do you plan on tagging a new version ?

markaren · 2023-12-01T17:15:46Z

https://pypi.org/project/pythonfmu/ (v0.6.3)

jschueller · 2023-12-01T17:38:45Z

great, thanks!

jschueller · 2023-12-04T08:25:00Z

I guess it can be closed now

markaren added the bug Something isn't working label May 12, 2021

fcollonval mentioned this issue Oct 12, 2021

ENH: pip install . will call cmake before packaging #136

Merged

charelstoncrabb added a commit to Geminus-AI/PythonFMU that referenced this issue Apr 5, 2023

This https://github.com/bfut/PythonFMU/commit/915226c3bf2287fa9d83aba…

d12414e

…663c102e24fd521f7 seems to be a workable fix to NTNU-IHB#128

markaren closed this as completed Dec 1, 2023

markaren reopened this Dec 1, 2023

markaren closed this as completed Dec 4, 2023

StephenSmith25 mentioned this issue May 23, 2024

Can you confirm if python3.11 is mandatory for using pythonfmu3 StephenSmith25/PythonFMU3#8

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Binaries on PyPI only support Python 3.9 #128

Binaries on PyPI only support Python 3.9 #128

BenjaminRodenberg commented May 12, 2021

markaren commented May 12, 2021 •

edited

Loading

BenjaminRodenberg commented May 12, 2021 •

edited

Loading

BenjaminRodenberg commented May 12, 2021 •

edited

Loading

markaren commented May 12, 2021

markaren commented May 12, 2021

BenjaminRodenberg commented May 12, 2021

markaren commented May 12, 2021

markaren commented May 13, 2021 •

edited

Loading

BenjaminRodenberg commented May 18, 2021

markaren commented May 18, 2021

HenningScheufler commented Jul 22, 2021

markaren commented Jul 22, 2021

HenningScheufler commented Sep 24, 2021 •

edited

Loading

2018-Summer commented Dec 10, 2021 •

edited

Loading

markaren commented Dec 10, 2021

2018-Summer commented Dec 13, 2021

markaren commented Dec 13, 2021

jschueller commented Nov 9, 2023 •

edited

Loading

jschueller commented Dec 1, 2023

markaren commented Dec 1, 2023

jschueller commented Dec 1, 2023

markaren commented Dec 1, 2023 •

edited

Loading

jschueller commented Dec 1, 2023

jschueller commented Dec 4, 2023

Binaries on PyPI only support Python 3.9 #128

Binaries on PyPI only support Python 3.9 #128

Comments

BenjaminRodenberg commented May 12, 2021

Why only python 3.9?

Further details

markaren commented May 12, 2021 • edited Loading

BenjaminRodenberg commented May 12, 2021 • edited Loading

BenjaminRodenberg commented May 12, 2021 • edited Loading

PyPI

This Repo (github)

markaren commented May 12, 2021

markaren commented May 12, 2021

BenjaminRodenberg commented May 12, 2021

markaren commented May 12, 2021

markaren commented May 13, 2021 • edited Loading

BenjaminRodenberg commented May 18, 2021

markaren commented May 18, 2021

HenningScheufler commented Jul 22, 2021

markaren commented Jul 22, 2021

HenningScheufler commented Sep 24, 2021 • edited Loading

2018-Summer commented Dec 10, 2021 • edited Loading

markaren commented Dec 10, 2021

2018-Summer commented Dec 13, 2021

markaren commented Dec 13, 2021

jschueller commented Nov 9, 2023 • edited Loading

jschueller commented Dec 1, 2023

markaren commented Dec 1, 2023

jschueller commented Dec 1, 2023

markaren commented Dec 1, 2023 • edited Loading

jschueller commented Dec 1, 2023

jschueller commented Dec 4, 2023

markaren commented May 12, 2021 •

edited

Loading

BenjaminRodenberg commented May 12, 2021 •

edited

Loading

BenjaminRodenberg commented May 12, 2021 •

edited

Loading

markaren commented May 13, 2021 •

edited

Loading

HenningScheufler commented Sep 24, 2021 •

edited

Loading

2018-Summer commented Dec 10, 2021 •

edited

Loading

jschueller commented Nov 9, 2023 •

edited

Loading

markaren commented Dec 1, 2023 •

edited

Loading