Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

symengine will conflict with libsbml on Python 3.9 and Linux #1056

Closed
cdiener opened this issue Mar 18, 2021 · 15 comments
Closed

symengine will conflict with libsbml on Python 3.9 and Linux #1056

cdiener opened this issue Mar 18, 2021 · 15 comments
Labels
bug SBML Related to reading and writing SBML models.

Comments

@cdiener
Copy link
Member

cdiener commented Mar 18, 2021

Problem description

This will hit us when we support Python 3.9. There seems to be serious bug in libSBML that breaks reading of SBO terms completely. This will make several of our tests fail.

Code Sample

Create a minimal, complete, verifiable example.

This uses the E. coli model from our repo.

from cobra.io import read_sbml_model

mod = read_sbml_model("iJO1366.xml.gz")
mod.reactions.LDH_D.annotation

On Python < 3.9

{'sbo': 'SBO:0000375', 'bigg.reaction': 'LDH_D', 'biocyc': 'META:DLACTDEHYDROGNAD-RXN', 'ec-code': '1.1.1.28', 'kegg.reaction': 'R00704', 'metanetx.reaction': 'MNXR101037', 'rhea': ['16369', '16370', '16371', '16372']}

On Python 3.9

{'sbo': 'SBO:',  # <- see here
 'bigg.reaction': 'LDH_D',
 'biocyc': 'META:DLACTDEHYDROGNAD-RXN',
 'ec-code': '1.1.1.28',
 'kegg.reaction': 'R00704',
 'metanetx.reaction': 'MNXR101037',
 'rhea': ['16369', '16370', '16371', '16372']}

This is due to a bug in libsbml where libsbml.SBase.getSBOTermID() always returns "SBO:" (without the ID). libsbml.SBase.getSBOTerm() still works (returns 375 in this case).

Context

I think depinfo is a bit buggy as well. Here is the correct libsbml version:

ython 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libsbml
>>> libsbml.__version__
'5.19.0'
System Information
==================
OS                  Linux
OS-release 5.10.0-4-amd64
Python              3.9.2

Package Versions
================
appdirs~=1.4                  not installed
black                                20.8b1
bumpversion                           0.6.0
cobra                                0.21.0
depinfo                               1.6.0
diskcache~=5.0                not installed
future                               0.18.2
httpx~=0.14                   not installed
importlib_resources                   3.3.0
isort; extra == "development" not installed
numpy~=1.13                   not installed
optlang<1.4.6                 not installed
pandas~=1.0                   not installed
pip                                  21.0.1
pydantic~=1.6                 not installed
python-libsbml==5.19.0        not installed
rich~=6.0                     not installed
ruamel.yaml~=0.16             not installed
scipy                                 1.6.1
setuptools                           54.1.2
six                                  1.15.0
swiglpk                               5.0.3
tox                                  3.23.0
wheel                                0.36.2
@cdiener cdiener added SBML Related to reading and writing SBML models. bug labels Mar 18, 2021
@Midnighter
Copy link
Member

So is this something we should address or do we need an upstream fix in libSBML?

(Please open an issue on depinfo when you can say clearly what is going wrong.)

@cdiener
Copy link
Member Author

cdiener commented Mar 18, 2021

I think this is upstream but there is a potential workaround by using libsbml.SBase.getSBOTerm() instead and assembling the ID from that.

No idea about depinfo. It shows the correct versions for some deps but not others. Maybe it's something with my setup.

@cdiener
Copy link
Member Author

cdiener commented Mar 22, 2021

Okay, it is not upstream per se, but there is something weird going on with our imports because the following fixes it:

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import libsbml

In [2]: from cobra.io import read_sbml_model

In [3]: mod = read_sbml_model("src/cobra/test/data/iJO1366.xml.gz")

In [4]: mod.reactions.LDH_D.annotation
Out[4]: 
{'sbo': 'SBO:0000375',
 'bigg.reaction': 'LDH_D',
 'biocyc': 'META:DLACTDEHYDROGNAD-RXN',
 'ec-code': '1.1.1.28',
 'kegg.reaction': 'R00704',
 'metanetx.reaction': 'MNXR101037',
 'rhea': ['16369', '16370', '16371', '16372']}

So just by importing libsbml before cobrapy it has no issues anymore. I could also reproduce this in a fresh Python env.

@Midnighter
Copy link
Member

Strange, I don't have that issue at all in the following environment:

Python 3.9.0 (default, Nov 29 2020, 14:58:16)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from cobra.io import read_sbml_model

In [2]: mod = read_sbml_model("src/cobra/test/data/iJO1366.xml.gz")

In [3]: mod.reactions.LDH_D.annotation
Out[3]:
{'sbo': 'SBO:0000375',
 'bigg.reaction': 'LDH_D',
 'biocyc': 'META:DLACTDEHYDROGNAD-RXN',
 'ec-code': '1.1.1.28',
 'kegg.reaction': 'R00704',
 'metanetx.reaction': 'MNXR101037',
 'rhea': ['16369', '16370', '16371', '16372']}
System Information
==================
OS                      Linux
OS-release 5.8.0-7642-generic
Python                  3.9.0

Package Versions
================
appdirs                                      1.4.4
black ; extra == 'development'       not installed
bumpversion ; extra == 'development' not installed
cobra                                       0.21.0
depinfo                                      1.7.0
diskcache                                    5.2.1
future                                      0.18.2
httpx                                       0.17.1
importlib-resources                          5.1.2
isort ; extra == 'development'       not installed
numpy                                       1.20.1
optlang                                      1.4.4
pandas                                       1.2.3
pip                                         20.2.3
pydantic                                     1.8.1
python-libsbml                              5.19.0
rich                                         6.2.0
ruamel.yaml                                0.16.13
scipy ; extra == 'array'             not installed
setuptools                                  49.2.1
six                                         1.15.0
swiglpk                                      5.0.3
tox ; extra == 'development'         not installed

@cdiener
Copy link
Member Author

cdiener commented Mar 22, 2021

Here is a more minimal version that triggers the error for me:

Python 3.9.0 (default, Nov 15 2020, 14:28:56) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import cobra

In [2]: import libsbml

In [3]: doc = libsbml.readSBML("iJO1366.xml.gz")

In [4]: doc.getModel().getReaction("R_LDH_D").getSBOTermID()
Out[4]: 'SBO:'

@cdiener
Copy link
Member Author

cdiener commented Mar 22, 2021

The problem is a conflict between libsbml and symengine. Uninstalling symengine fixes the bug.

@Midnighter
Copy link
Member

Python 3.9.0 (default, Nov 29 2020, 14:58:16) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import libsbml

In [2]: doc = libsbml.readSBML("cobrapy/src/cobra/test/data/iJO1366.xml.gz")

In [3]: doc.getModel().getReaction("R_LDH_D").getSBOTermID()
Out[3]: 'SBO:0000375'

😆 which libSBML is that using?

@Midnighter
Copy link
Member

The problem is a conflict between libsbml and symengine. Uninstalling symengine fixes the bug.

Wow, what?

@cdiener
Copy link
Member Author

cdiener commented Mar 22, 2021

I had those issues before. Something in the symengine wheels breaks other packages, I'll open a report there.

See for instance: numba/llvmlite#587

@cdiener
Copy link
Member Author

cdiener commented Mar 22, 2021

Minimal example:

In [1]: import symengine

In [2]: import libsbml

In [3]: doc = libsbml.readSBML("iJO1366.xml.gz")

In [4]: doc.getModel().getReaction("R_LDH_D").getSBOTermID()
Out[4]: 'SBO:'

Removing the symengine import will change the output to:

Python 3.9.0 (default, Nov 15 2020, 14:28:56) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import libsbml

In [2]: doc = libsbml.readSBML("iJO1366.xml.gz")

In [3]: doc.getModel().getReaction("R_LDH_D").getSBOTermID()
Out[3]: 'SBO:0000375'

@cdiener cdiener changed the title SBO term reading broken in Python 3.9 symengine will conflict with libsbml on Python 3.9 and Linux Mar 22, 2021
@Midnighter
Copy link
Member

I was wondering if symengine causes libsbml to load a different shared library but as far as I can tell that is not the case.

❯ ldd python3.9/site-packages/libsbml/_libsbml.cpython-39-x86_64-linux-gnu.so                                                                                
        linux-vdso.so.1 (0x00007ffe9fb8a000)                                                   
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5b691ce000)            
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5b6907f000)                      
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5b69064000)              
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5b68e72000)                      
        /lib64/ld-linux-x86-64.so.2 (0x00007f5b6a986000)
❯ ldd python3.9/site-packages/symengine.libs/libsymengine-054f8ecc.so.0.7.0                                                                                  
        linux-vdso.so.1 (0x00007ffd109f5000)
        libgmp-b6b552a9.so.10.4.1 => not found
        libflint-b75bea04.so.15.0.1 => not found                                               
        libmpc-d5c9e4c1.so.3.1.0 => not found
        libmpfr-0d3b9336.so.6.0.2 => not found
        libz-1f4891ad.so.1.2.11 => not found
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4557b2e000)                    
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4557b0b000)          
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f45579bc000)                      
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f45579a1000)              
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f45577af000)                                                                                                                     
        /lib64/ld-linux-x86-64.so.2 (0x00007f4559bc6000)

Unless it's a local linux-vdso.so.1? Not really my area of strength.

@cdiener
Copy link
Member Author

cdiener commented Mar 22, 2021

But one is a manylinux1 image and the other manylinux2010, so there may be different versions of the same lib. But if they are built cleanly they should not export overlapping symbols.

@matthiaskoenig
Copy link
Contributor

Most likely this is due to C++ symbols being overwritten within the swig packages.
Basically all swig C++ symbols are imported in the same namespace so that conflicting namespaces will overwrite symbols from other packages. So the last package import wins basically. Something similar happened between libsbml and libsedml which have many similar names.
That the import order is fixing this is a strong indicator. As a workaround a reload of the library works within the respective code areas. E.g. just do a reload(libsbml) or reload(symengine) before accessing objects from the library.

@cdiener
Copy link
Member Author

cdiener commented Mar 24, 2021

Symengine is not a swig package though. But I do think you are right, ergo same symbols. I thought that the point of the new many linux images was to isolate this better but that does not seem to be the case.

@cdiener
Copy link
Member Author

cdiener commented Mar 25, 2021

Fixed in symengine 0.7.1.

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import symengine

In [2]: import libsbml

In [3]: doc = libsbml.readSBML("iJO1366.xml")

In [4]: doc.getModel().getReaction("R_LDH_D").getSBOTermID()
Out[4]: 'SBO:0000375'

@cdiener cdiener closed this as completed Mar 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug SBML Related to reading and writing SBML models.
Projects
None yet
Development

No branches or pull requests

3 participants