fix for "python -c 'import nest'" readline issues with Anaconda Python #209

Merged
merged 2 commits into from Feb 15, 2016

Conversation

Projects
None yet
5 participants
@espenhgn
Contributor

espenhgn commented Jan 19, 2016

importing nest with Anaconda Python may be broken for some due to some mismatch between the system and Anaconda-provided readline library files, such that importing nest in a running python interpreter would work, while "python -c 'import nest'" would not. Importing readline before pynestkernel takes care of that.

@apeyser

This comment has been minimized.

Show comment
Hide comment
@apeyser

apeyser Jan 19, 2016

Contributor

Does the pynest code assume that readline gets imported? If not, we shouldn't create a forced dependency on readline. Why not use python -c 'import readline, nest'?

Contributor

apeyser commented Jan 19, 2016

Does the pynest code assume that readline gets imported? If not, we shouldn't create a forced dependency on readline. Why not use python -c 'import readline, nest'?

@JanneM

This comment has been minimized.

Show comment
Hide comment
@JanneM

JanneM Jan 19, 2016

Contributor

It should perhaps depend on whether you use --with-readline during
configuration or not.

2016-01-19 20:45 GMT+09:00 Alexander Peyser notifications@github.com:

Does the pynest code assume that readline gets imported? If not, we
shouldn't create a forced dependency on readline. Why not use python -c
'import readline, nest'?


Reply to this email directly or view it on GitHub
#209 (comment).

Jan Morén
http://sites.google.com/site/janmoren/
http://janneinosaka.blogspot.com

Contributor

JanneM commented Jan 19, 2016

It should perhaps depend on whether you use --with-readline during
configuration or not.

2016-01-19 20:45 GMT+09:00 Alexander Peyser notifications@github.com:

Does the pynest code assume that readline gets imported? If not, we
shouldn't create a forced dependency on readline. Why not use python -c
'import readline, nest'?


Reply to this email directly or view it on GitHub
#209 (comment).

Jan Morén
http://sites.google.com/site/janmoren/
http://janneinosaka.blogspot.com

@apeyser

This comment has been minimized.

Show comment
Hide comment
@apeyser

apeyser Jan 19, 2016

Contributor

@JanneM : right. configure.ac currently has a --with/out-readline option, and on bgq it disables readline entirely. Doing a sed over an import file would seem overkill, as well as having a 'import nest_with_readline' option. There's /GNUreadline in systemdict, but that's available after the pynestkernel is imported, and anything more is overkill.

@espenhgn : so I go back to asking, why not just do 'import readline, nest' manually for the Anaconda Python case? Or using your own anaconda-python script that does that?

side note: Anaconda Python -- what an awful, awful name. Why name something intentionally ambiguously?

Contributor

apeyser commented Jan 19, 2016

@JanneM : right. configure.ac currently has a --with/out-readline option, and on bgq it disables readline entirely. Doing a sed over an import file would seem overkill, as well as having a 'import nest_with_readline' option. There's /GNUreadline in systemdict, but that's available after the pynestkernel is imported, and anything more is overkill.

@espenhgn : so I go back to asking, why not just do 'import readline, nest' manually for the Anaconda Python case? Or using your own anaconda-python script that does that?

side note: Anaconda Python -- what an awful, awful name. Why name something intentionally ambiguously?

@espenhgn

This comment has been minimized.

Show comment
Hide comment
@espenhgn

espenhgn Jan 19, 2016

Contributor

@apeyser I don't think the average Nest user would quickly understand that importing readline before nest would fix that ImportError. Note that import nest in a py-script started from the command line crashes similarly, and why should people have to adapt their codes to run with a particular python environment? It's not like the init.py file do not already contain several workarounds for other issues already.

Another less neat alternative could be to put the pynestkernel import within a try-except ImportError statement.

Contributor

espenhgn commented Jan 19, 2016

@apeyser I don't think the average Nest user would quickly understand that importing readline before nest would fix that ImportError. Note that import nest in a py-script started from the command line crashes similarly, and why should people have to adapt their codes to run with a particular python environment? It's not like the init.py file do not already contain several workarounds for other issues already.

Another less neat alternative could be to put the pynestkernel import within a try-except ImportError statement.

@apeyser

This comment has been minimized.

Show comment
Hide comment
@apeyser

apeyser Jan 19, 2016

Contributor

@espenhgn Does this issue appear only with Anaconda Python? Do we know what's going on here?

Contributor

apeyser commented Jan 19, 2016

@espenhgn Does this issue appear only with Anaconda Python? Do we know what's going on here?

@espenhgn

This comment has been minimized.

Show comment
Hide comment
@espenhgn

espenhgn Jan 19, 2016

Contributor

@apeyser Tested only with anaconda so far, could occur with e.g., Enthought. @tammoippen tried figuring out what libreadline is used when, but the conclusion was not clear to me. Traceback:

espen@espen-VirtualBox:~$ python -c "import nest"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/espen/.local/nest/lib/python2.7/site-packages/nest/__init__.py", line 52, in <module>
    from . import pynestkernel as _kernel
ImportError: /home/espen/anaconda2/bin/../lib/libreadline.so.6: undefined symbol: PC
´´´
Contributor

espenhgn commented Jan 19, 2016

@apeyser Tested only with anaconda so far, could occur with e.g., Enthought. @tammoippen tried figuring out what libreadline is used when, but the conclusion was not clear to me. Traceback:

espen@espen-VirtualBox:~$ python -c "import nest"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/espen/.local/nest/lib/python2.7/site-packages/nest/__init__.py", line 52, in <module>
    from . import pynestkernel as _kernel
ImportError: /home/espen/anaconda2/bin/../lib/libreadline.so.6: undefined symbol: PC
´´´
@apeyser

This comment has been minimized.

Show comment
Hide comment
@apeyser

apeyser Jan 20, 2016

Contributor

@espenhgn
The following issue seems similar: ContinuumIO/anaconda-issues#152
So anaconda has ongoing issues with library linking because (I think) it tries to replace dynamically linked libraries with it's own, that then have secondary issues with links.

Anaconda seems to be a "bad thing". Who is using this kludge and for what? What does this add over pip and virtualenv's?

Contributor

apeyser commented Jan 20, 2016

@espenhgn
The following issue seems similar: ContinuumIO/anaconda-issues#152
So anaconda has ongoing issues with library linking because (I think) it tries to replace dynamically linked libraries with it's own, that then have secondary issues with links.

Anaconda seems to be a "bad thing". Who is using this kludge and for what? What does this add over pip and virtualenv's?

@espenhgn

This comment has been minimized.

Show comment
Hide comment
@espenhgn

espenhgn Jan 20, 2016

Contributor

@apeyser I'm not gonna survey who's using anaconda on their own machines, but at least in our lab everyone using Python on the Blaustein cluster is. This import issue do not occur there though. The main advantage I guess is that users won't have to deal with all dependencies e.g., for pip-installing scipy etc.

Contributor

espenhgn commented Jan 20, 2016

@apeyser I'm not gonna survey who's using anaconda on their own machines, but at least in our lab everyone using Python on the Blaustein cluster is. This import issue do not occur there though. The main advantage I guess is that users won't have to deal with all dependencies e.g., for pip-installing scipy etc.

@apdavison

This comment has been minimized.

Show comment
Hide comment
@apdavison

apdavison Jan 20, 2016

@apeyser The main advantage of Anaconda is that it provides pre-built binaries for many Python packages. This saves a lot of time, compared to pip, when installing large packages such as Numpy, Scipy, Matplotlib. It is also a huge gain on OS X and Windows, where compiling such packages is sometimes tricky. (This advantage may disappear if/when Python wheels take off).

@apeyser The main advantage of Anaconda is that it provides pre-built binaries for many Python packages. This saves a lot of time, compared to pip, when installing large packages such as Numpy, Scipy, Matplotlib. It is also a huge gain on OS X and Windows, where compiling such packages is sometimes tricky. (This advantage may disappear if/when Python wheels take off).

@apeyser

This comment has been minimized.

Show comment
Hide comment
@apeyser

apeyser Jan 20, 2016

Contributor

@espenhgn @apdavison

Thanks for the info. So the problem here is essentially that people are putting pre-built binaries from one set-up on to their local distribution and there will be inevitable clashes in the dynamic libraries.

So the solutions are:

  • We try to hack around this by adding imports to get the correct linking order
  • We complain to anaconda to improve their system to handle linking issues
  • We push it back on to the users to properly compile nest using the right link options to fit anaconda
  • Are there anaconda recipes or package specs that could add the right link options for users to build nest under anaconda?
Contributor

apeyser commented Jan 20, 2016

@espenhgn @apdavison

Thanks for the info. So the problem here is essentially that people are putting pre-built binaries from one set-up on to their local distribution and there will be inevitable clashes in the dynamic libraries.

So the solutions are:

  • We try to hack around this by adding imports to get the correct linking order
  • We complain to anaconda to improve their system to handle linking issues
  • We push it back on to the users to properly compile nest using the right link options to fit anaconda
  • Are there anaconda recipes or package specs that could add the right link options for users to build nest under anaconda?
@apeyser

This comment has been minimized.

Show comment
Hide comment
@apeyser

apeyser Jan 26, 2016

Contributor

I think the final solution to this, is that this is a problem of improper linking using anaconda. The build is picking up a different environment than the run.
The solution is thus to build under anaconda, using an anaconda recipe, which would also allow the inclusion of releases in anaconda.
So, someone using anaconda could put together a proper build and share it...

If I'm missing something, please let me know. Otherwise, I'll close the issue until someone fulfills request #210, which imho is the correct solution.

Contributor

apeyser commented Jan 26, 2016

I think the final solution to this, is that this is a problem of improper linking using anaconda. The build is picking up a different environment than the run.
The solution is thus to build under anaconda, using an anaconda recipe, which would also allow the inclusion of releases in anaconda.
So, someone using anaconda could put together a proper build and share it...

If I'm missing something, please let me know. Otherwise, I'll close the issue until someone fulfills request #210, which imho is the correct solution.

@espenhgn

This comment has been minimized.

Show comment
Hide comment
@espenhgn

espenhgn Feb 1, 2016

Contributor

@tammoippen suggested to import readline only if 'linux' in sys.platform and 'Anaconda' in sys.version.

Contributor

espenhgn commented Feb 1, 2016

@tammoippen suggested to import readline only if 'linux' in sys.platform and 'Anaconda' in sys.version.

@apeyser

This comment has been minimized.

Show comment
Hide comment
@apeyser

apeyser Feb 2, 2016

Contributor

Much less ugly... I wonder whether there should be a Makefile 'platform hacks' module created that gets loaded here...

but I wonder, if NEST in Anaconda is so popular, why hasn't there been a volunteer to make a linux anaconda recipe?

Contributor

apeyser commented Feb 2, 2016

Much less ugly... I wonder whether there should be a Makefile 'platform hacks' module created that gets loaded here...

but I wonder, if NEST in Anaconda is so popular, why hasn't there been a volunteer to make a linux anaconda recipe?

@apdavison

This comment has been minimized.

Show comment
Hide comment
@apdavison

apdavison Feb 11, 2016

I think Anaconda is more popular with OS X and Windows users; it is not needed so much on Linux distros as they have fully-integrated package managers, and most open source tools are developed primarily for Linux and so build more reliably.

I think Anaconda is more popular with OS X and Windows users; it is not needed so much on Linux distros as they have fully-integrated package managers, and most open source tools are developed primarily for Linux and so build more reliably.

@jougs

This comment has been minimized.

Show comment
Hide comment
@jougs

jougs Feb 15, 2016

Contributor

During the NEST Core meeting on 15th of February 2016, we decided to merge this as we can't expect Anaconda to fix the problem in a general way and a Anaconda recipe for NEST is not in sight.

Contributor

jougs commented Feb 15, 2016

During the NEST Core meeting on 15th of February 2016, we decided to merge this as we can't expect Anaconda to fix the problem in a general way and a Anaconda recipe for NEST is not in sight.

jougs added a commit that referenced this pull request Feb 15, 2016

Merge pull request #209 from espenhgn/anaconda_fix
fix for "python -c 'import nest'" readline issues with Anaconda Python

@jougs jougs merged commit d010039 into nest:master Feb 15, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@espenhgn espenhgn deleted the espenhgn:anaconda_fix branch Apr 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment