ImportError: dynamic module does not define module export function (PyInit_ydk_)

[vipaoqun]

Expected Behavior

YDK should work well with python2 and python3 in Centos7.5

Current Behavior

I have built and installed YDK-gen and YDK-py using python2 and python3 in my Centos 7.5 VM , it works well in python2 but not in python3, and I notice that both of them are built using the same GCC 4.8.5.More details you can also refer to the community discussion

Steps to Reproduce

[root@localhost ydk-gen]# 
[root@localhost ydk-gen]# cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core) 
[root@localhost ydk-gen]# 
[root@localhost ydk-gen]# 
[root@localhost ydk-gen]# 
[root@localhost ydk-gen]# 
[root@localhost ydk-gen]# python -V
Python 3.6.2
[root@localhost ydk-gen]# 
[root@localhost ydk-gen]# 
[root@localhost ydk-gen]# python -m pip list
Package                 Version    
----------------------- -----------
alabaster               0.7.12     
Babel                   2.6.0      
docutils                0.14       
gitdb                   0.6.4      
GitPython               1.0.1      
imagesize               1.1.0      
Jinja2                  2.10       
MarkupSafe              1.1.0      
pbr                     5.1.1      
pip                     18.1       
pyang                   1.6        
pybind11                2.2.4      
Pygments                2.2.0      
pytz                    2018.7     
rstr                    2.2.6      
setuptools              28.8.0     
six                     1.11.0     
smmap                   0.9.0      
snowballstemmer         1.2.1      
Sphinx                  1.4a1      
sphinx-rtd-theme        0.1.9      
stevedore               1.30.0     
virtualenv              16.1.0     
virtualenv-clone        0.4.0      
virtualenvwrapper       4.8.2      
ydk                     0.7.3      
ydk-models-cisco-ios-xr 6.5.1      
ydk-models-ietf         0.1.5.post2
[root@localhost ydk-gen]# 
[root@localhost ydk-gen]# 
[root@localhost ydk-gen]# python
Python 3.6.2 (default, Nov 21 2018, 21:44:58) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> from ydk import *
>>> 
>>> 
>>> from ydk.types import Empty
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/types/__init__.py", line 17, in <module>
    from .py_types import Entity, EntityCollection, Config, Filter, YList, YLeafList
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/types/py_types.py", line 35, in <module>
    from ydk_ import is_set
ImportError: dynamic module does not define module export function (PyInit_ydk_)
>>> 
>>> 
>>> from ydk.services import CRUDService
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/services/__init__.py", line 17, in <module>
    from .codec_service import CodecService
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/services/codec_service.py", line 21, in <module>
    from ydk.entity_utils import get_data_node_from_entity as _get_data_node_from_entity
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/entity_utils/__init__.py", line 17, in <module>
    from ydk.ext.entity_utils import get_data_node_from_entity
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/exthook.py", line 87, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named ydk.ext.entity_utils
>>> 
>>> 

and

[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# sudo yum install epel-release
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.nethub.com.hk
 * epel: mirror2.totbb.net
 * extras: centos.nethub.com.hk
 * updates: centos.nethub.com.hk
rabbitmq_erlang/x86_64/signature                                                                      |  836 B  00:00:00     
rabbitmq_erlang/x86_64/signature                                                                      | 1.0 kB  00:00:00 !!! 
rabbitmq_erlang-source/signature                                                                      |  836 B  00:00:00     
rabbitmq_erlang-source/signature                                                                      |  951 B  00:00:00 !!! 
Package epel-release-7-11.noarch already installed and latest version
Nothing to do
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# sudo yum install libssh-devel gcc-c++ python-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.nethub.com.hk
 * epel: mirror2.totbb.net
 * extras: centos.nethub.com.hk
 * updates: centos.nethub.com.hk
Package libssh-devel-0.7.1-3.el7.x86_64 already installed and latest version
Package gcc-c++-4.8.5-28.el7_5.1.x86_64 already installed and latest version
Package python-devel-2.7.5-69.el7_5.x86_64 already installed and latest version
Nothing to do
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# sudo yum install https://devhub.cisco.com/artifactory/rpm-ydk/0.7.3/libydk-0.7.3-1.x86_64.rpm
Loaded plugins: fastestmirror
libydk-0.7.3-1.x86_64.rpm                                                                             | 2.1 MB  00:00:01     
Examining /var/tmp/yum-root-j0oOQI/libydk-0.7.3-1.x86_64.rpm: libydk-0.7.3-1.x86_64
/var/tmp/yum-root-j0oOQI/libydk-0.7.3-1.x86_64.rpm: does not update installed package.
Error: Nothing to do
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# sudo yum install centos-release-scl -y > /dev/null
[root@localhost ~]# 
[root@localhost ~]# sudo yum install devtoolset-4-gcc* -y > /dev/null
[root@localhost ~]# 
[root@localhost ~]# sudo ln -sf /opt/rh/devtoolset-4/root/usr/bin/gcc /usr/bin/cc
[root@localhost ~]# 
[root@localhost ~]# sudo ln -sf /opt/rh/devtoolset-4/root/usr/bin/g++ /usr/bin/c++
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# python2
Python 2.7.5 (default, Jul 13 2018, 13:06:57) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import ydk
>>> from ydk.types import Empty
>>> from ydk.services import CRUDService
>>> 
>>> exit()
[root@localhost ~]# 
[root@localhost ~]# python
Python 3.6.2 (default, Nov 21 2018, 21:44:58) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import ydk
>>> from ydk.types import Empty
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/types/__init__.py", line 17, in <module>
    from .py_types import Entity, EntityCollection, Config, Filter, YList, YLeafList
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/types/py_types.py", line 35, in <module>
    from ydk_ import is_set
ImportError: dynamic module does not define module export function (PyInit_ydk_)
>>> 
>>> from ydk.services import CRUDService
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/services/__init__.py", line 17, in <module>
    from .codec_service import CodecService
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/services/codec_service.py", line 21, in <module>
    from ydk.entity_utils import get_data_node_from_entity as _get_data_node_from_entity
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/entity_utils/__init__.py", line 17, in <module>
    from ydk.ext.entity_utils import get_data_node_from_entity
  File "/usr/local/python3.6/lib/python3.6/site-packages/ydk/exthook.py", line 87, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named ydk.ext.entity_utils
>>> 
>>> exit()
[root@localhost ~]# 

Your Script

Logs

Enable logging and post the logs below

System Information

[ygorelik]

The GCC 4.8.5 is not sufficient to compile YDK C++ code. It does not support some of the GCC version 5 C++ constructs, which are widely used in YDK code. The documentation explicitly defines installation of GCC 5.x in System Requirements. All published C/C++ artifacts are compiled with GCC 5.4.0 and most likely are not compatible with other GCC versions. The error appears due to this incompatibility. If customer's system has different installation of GCC, the YDK libraries must be recompiled from source.

[ghost]

Issue should be

• documented in YDK doc
• fixed in cmake

[ygorelik]

Added the following statement in documentation:

YDK supports both Python2 and Python3 versions.  At least Python2.7 or Python3.4 along with corresponding utilities pip and pip3 must be installed on your system. 

It is also required for Python installation to include corresponding shared library. As example: 

 - python2.7  - /usr/lib/x86_64-linux-gnu/libpython2.7.so
 - python3.5m - /usr/lib/x86_64-linux-gnu/libpython3.5m.so

[ghost]

For cmake, need to investigate a bit.

currently, to determine the python library, we are using the routine here: https://github.com/CiscoDevNet/ydk-gen/blob/master/sdk/python/core/setup.py#L110
this is passed to cmake here: https://github.com/CiscoDevNet/ydk-gen/blob/master/sdk/python/core/setup.py#L92
cmake uses this here: https://github.com/CiscoDevNet/ydk-gen/blob/master/sdk/python/core/CMakeLists.txt#L31

The fix (if any) will have to be in one of the above.

See the documentation here: https://cmake.org/cmake/help/v3.2/module/FindPythonLibs.html.
Python_ADDITIONAL_VERSIONS should be set before calling find_package(PythonLibs).

But somehow whatever we are passing to it seems inaccurate or incomplete in some environments

[ghost]

The issue can be fixed by following the guidelines in YDK documentation: http://ydk.cisco.com/py/docs/getting_started.html