Python module does not load ... #1782

linas · 2018-06-26T21:10:26Z

This is a followup to issue opencog/opencog#3226 --
Starting the cogserver results in the following message being printed:

[2018-06-26 20:39:40:204] [INFO] Starting CogServer loop.
[2018-06-26 20:39:55:415] [INFO] Using config file found at: /home/ubuntu/run/./dj-en.conf

[2018-06-26 20:39:55:428] [INFO] [global_python_initialize] Start
[2018-06-26 20:39:55:439] [INFO] [global_python_initialize] Adding OpenCog sys.path directories
[2018-06-26 20:39:55:440] [WARN] PythonEval::try_to_load_modules Failed to load the opencog.atomspace module
[2018-06-26 20:39:55:440] [INFO] [global_python_initialize] Finish
[2018-06-26 20:39:55:440] [INFO] [global_python_initialize] Start
[2018-06-26 20:39:55:440] [WARN] PythonEval::atomspace_py_object Failed to load theopencog.atomspace module
[2018-06-26 20:39:55:440] [INFO] PythonEval::initialize_python_objects_and_imports Finished initialising python evaluator.

The text was updated successfully, but these errors were encountered:

linas · 2018-06-26T21:15:23Z

Above is after a clean install:

rm -r /usr/local/include/opencog
rm -r /usr/local/lib/opencog
rm -r /usr/local/share/opencog

and then rebuilding/installing cogutils, atomspace, opencog.

The python module gets installed here, on my system:

/usr/local/lib/python3.5/dist-packages/opencog/atomspace.so

and ldd -r shows that its fully linked, so that is not the issue ...

linas · 2018-06-26T21:20:41Z

My current guess is that python is not looking in the /usr/local directory, and/or is looking it site-packages instead of dist-packages. This is a very old generic python issue: ubuntu uses site-packages (or is it Debian??) but the mainstream python maintainers do not. Plus, this policy is different between python 2.7 and 3.5 so its very confusing to figure out the correct install location.

ChehST · 2018-09-26T23:41:53Z

Hi, is there "$ python -m site" your path ?

/usr/local/lib/python3.5/dist-packages/opencog/atomspace.so

linas · 2018-09-29T00:49:14Z

$ python -m site
sys.path = [
    '/home/linas',
    '/usr/lib/python2.7',
    '/usr/lib/python2.7/plat-x86_64-linux-gnu',
    '/usr/lib/python2.7/lib-tk',
    '/usr/lib/python2.7/lib-old',
    '/usr/lib/python2.7/lib-dynload',
    '/usr/local/lib/python2.7/dist-packages',
    '/usr/local/lib/python2.7/dist-packages/owyl-0.3.dev-py2.7.egg',
    '/usr/local/lib/python2.7/dist-packages/cocos2d-0.6.0-py2.7.egg',
    '/usr/local/lib/python2.7/dist-packages/distribute-0.7.3-py2.7.egg',
    '/usr/lib/python2.7/dist-packages',
    '/usr/lib/python2.7/dist-packages/PILcompat',
    '/usr/lib/python2.7/dist-packages/gtk-2.0',
    '/usr/lib/pymodules/python2.7',
]
USER_BASE: '/home/linas/.local' (exists)
USER_SITE: '/home/linas/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

linas · 2018-09-29T00:54:53Z

Hang on; the above python -m site is on a machine that does not generate that error. Let me try again on the machine that does generate the error

linas · 2018-09-29T00:58:41Z

On the failing system, I get:

$ python -m site
sys.path = [
    '/home/ubuntu/run',
    '/usr/lib/python2.7',
    '/usr/lib/python2.7/plat-x86_64-linux-gnu',
    '/usr/lib/python2.7/lib-tk',
    '/usr/lib/python2.7/lib-old',
    '/usr/lib/python2.7/lib-dynload',
    '/home/ubuntu/.local/lib/python2.7/site-packages',
    '/usr/local/lib/python2.7/dist-packages',
    '/usr/lib/python2.7/dist-packages',
    '/usr/lib/python2.7/dist-packages/PILcompat',
]
USER_BASE: '/home/ubuntu/.local' (exists)
USER_SITE: '/home/ubuntu/.local/lib/python2.7/site-packages' (exists)
ENABLE_USER_SITE: True

linas · 2018-09-29T01:00:24Z

and also:

$ python3 -m site
sys.path = [
    '/home/ubuntu/run',
    '/usr/lib/python35.zip',
    '/usr/lib/python3.5',
    '/usr/lib/python3.5/plat-x86_64-linux-gnu',
    '/usr/lib/python3.5/lib-dynload',
    '/usr/local/lib/python3.5/dist-packages',
    '/usr/lib/python3/dist-packages',
]
USER_BASE: '/home/ubuntu/.local' (exists)
USER_SITE: '/home/ubuntu/.local/lib/python3.5/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

ChehST · 2018-09-29T01:45:52Z

Maybe you've different versions of python? For example: 3.4; 3.5; .3.7? And system runs 3.4 version via this call "$ python3 m -site". Try point more precisely, like "python3.5 -m site" I think for each version, system uses different paths.

linas · 2019-02-02T04:22:28Z

The only way I can get python to work with opencog is to explicitly specify the path.

PYTHONPATH=/usr/local/lib/python3/dist-packages/ python3

vsbogd · 2019-02-04T08:20:04Z

The only way I can get python to work with opencog is to explicitly specify the path.

yes, it looks like the python issue, atomspace uses get_python_lib call to get the destination for new modules:

		EXECUTE_PROCESS(
			COMMAND ${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import *;print(get_python_lib())"
			OUTPUT_VARIABLE PYTHON_DEST
		)

But at the moment get_python_lib returns /usr/local/lib/python3/dist-packages path which is not in sys.path surprisingly. But at the same time sys.path contains /usr/local/lib/python3.5/dist-packages

linas · 2019-02-06T22:57:35Z

There's already code in PythonEval.cc that hacks sys.path and maybe we need one more hack. It's just endlessly annoying to have to always try things 3 or 4 times, before finding the correct magic spell to make python work.

vsbogd · 2019-02-07T08:07:56Z

Yes, fix for sys.path should solve the issue in GogServer. But it will not solve problem for opencog libraries which are used from Python interpreter. @stellarspot raised an issue on Ubuntu about this problem https://bugs.launchpad.net/ubuntu/+source/python3-defaults/+bug/1814653

linas · 2019-06-10T16:26:29Z

Ah, so on failing system, python3 -m site includes /usr/local/lib/python3.5/dist-packages whereas the atomspace is installed into /usr/local/lib/python3/dist-packages .. so the question is, why is the atomspace being installed where python can't find it?

linas · 2019-06-10T16:33:32Z

well, CMake prints:

-- Python destination dir found: /usr/local/lib/python3/dist-packages
-- Python install dir: /usr/local/lib/python3/dist-packages/opencog

and that is the path provided by

from distutils.sysconfig import *
print(get_python_lib(True, False, '/usr/local'))

So I conclude: python3 distutils.sysconfig is broken: it tells us to use a path that python itself does not actually use.

vsbogd · 2019-06-10T16:36:38Z

Yes, it is what https://bugs.launchpad.net/ubuntu/+source/python3-defaults/+bug/1814653 is about

vsbogd · 2019-06-10T16:39:13Z

I also added https://bugs.launchpad.net/ubuntu/+source/python3-stdlib-extensions/+bug/1832215 and proposed patch to package maintainers.

linas · 2019-06-10T16:53:14Z

after some experiments:

>>> print(get_python_lib(True, False, '/usr/local'))
/usr/local/lib/python3/dist-packages
>>> print(get_python_lib(True, True, '/usr/local'))
/usr/local/lib/python3.5

linas · 2019-06-10T16:56:10Z

@vsbogd you may want to email Matthias Klose directly (and/or report it as a Debian bug, which Ubuntu is picking up) (per your "root cause" patch https://salsa.debian.org/cpython-team/python3-stdlib/blob/master/debian/patches/3.6/distutils-install-layout.diff )

vsbogd · 2019-06-10T17:42:20Z

I wrote to Matthias. I am not sure it is need to be reported as Debian, as Matthias has an e-mail at ubuntu.com, so I will wait for his response.

ferrouswheel · 2019-06-12T00:56:38Z

This is a very old generic python issue: ubuntu uses site-packages (or is it Debian??) but the mainstream python maintainers do not.

It's the opposite - Ubuntu and Debian use dist-packages. If you build python from source, python packages will go into site-packages.

This hasn't changed between python2.7/python3.5/python3.6, but the distutils bug does seem to be introduced moving from 2 to 3.

An alternative way to find the correct location is:

$ python3.5 -c 'import site; print(site.getsitepackages())'
['/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.5/dist-packages']

(then either grab the first item, or search for the one prefixed by /usr/local/lib)

This works for python versions 2.7, 3.5, 3.6 on Ubuntu 18.04. I assume it works elsewhere too.

ferrouswheel · 2019-06-12T01:07:57Z

I spoke too soon, virtualenv doesn't have a recent site module so wouldn't work with my proposed alternative: pypa/virtualenv#355 :-(

linas added the python label Jun 26, 2018

linas added the good first issue Good for newcomers label Jul 8, 2018

vsbogd mentioned this issue Jun 10, 2019

ImportError: No module named 'opencog.atomspace' opencog/opencog#3528

Closed

ferrouswheel mentioned this issue Jun 12, 2019

It's not possible to get a TV unless the atom is in an atomspace #2216

Merged

ferrouswheel mentioned this issue Jun 12, 2019

Workaround for distutils bug in Ubuntu/Debian's Python3 versions #2220

Merged

linas closed this as completed in #2220 Jun 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python module does not load ... #1782

Python module does not load ... #1782

linas commented Jun 26, 2018

linas commented Jun 26, 2018

linas commented Jun 26, 2018

ChehST commented Sep 26, 2018

linas commented Sep 29, 2018

linas commented Sep 29, 2018

linas commented Sep 29, 2018

linas commented Sep 29, 2018

ChehST commented Sep 29, 2018

linas commented Feb 2, 2019

vsbogd commented Feb 4, 2019

linas commented Feb 6, 2019

vsbogd commented Feb 7, 2019

linas commented Jun 10, 2019

linas commented Jun 10, 2019

vsbogd commented Jun 10, 2019

vsbogd commented Jun 10, 2019

linas commented Jun 10, 2019

linas commented Jun 10, 2019 •

edited

Loading

vsbogd commented Jun 10, 2019

ferrouswheel commented Jun 12, 2019 •

edited

Loading

ferrouswheel commented Jun 12, 2019

Python module does not load ... #1782

Python module does not load ... #1782

Comments

linas commented Jun 26, 2018

linas commented Jun 26, 2018

linas commented Jun 26, 2018

ChehST commented Sep 26, 2018

linas commented Sep 29, 2018

linas commented Sep 29, 2018

linas commented Sep 29, 2018

linas commented Sep 29, 2018

ChehST commented Sep 29, 2018

linas commented Feb 2, 2019

vsbogd commented Feb 4, 2019

linas commented Feb 6, 2019

vsbogd commented Feb 7, 2019

linas commented Jun 10, 2019

linas commented Jun 10, 2019

vsbogd commented Jun 10, 2019

vsbogd commented Jun 10, 2019

linas commented Jun 10, 2019

linas commented Jun 10, 2019 • edited Loading

vsbogd commented Jun 10, 2019

ferrouswheel commented Jun 12, 2019 • edited Loading

ferrouswheel commented Jun 12, 2019

linas commented Jun 10, 2019 •

edited

Loading

ferrouswheel commented Jun 12, 2019 •

edited

Loading