Skip to content
This repository

add dirty trick for readline import on OSX #937

Merged
merged 2 commits into from over 2 years ago

2 participants

Min RK Fernando Perez
Min RK
Owner

also made the libedit warning extremely loud, so people don't miss it.

We still get reports of people never having noticed the warning, and getting confused when readline is broken on OSX.

The libedit test is also simplified to just check libedit in readline.__doc__, which the official Python docs recommend, rather than the elaborate subprocess/otool method that was not especially reliable.

The reason for the dirty trick

pip installs to site-packages by default, but site-packages dirs always come after lib-dynload (and extras, etc.), which is where the system readline is installed.

That means that a non-setuptools install (pip or setup.py install) cannot override any package that ships with OSX, including:

  • numpy
  • readline
  • twisted
  • pyobjc

without installing to a non-standard path (not even user site-packages via --user).

Note that this path issue not unique to OSX - lib-dynload, and patform packages always come before site-packages, including --user. It's just that OSX seems to be the only platform where there's anything in those locations that you would actually want to supplant.

The method for the dirty trick

  1. remove lib-dynload from sys.path before trying to import readline the first time
  2. after import, restore lib-dynload to its place in sys.path
  3. if import failed without lib-dynload, try it one more time, to get the default module

This doesn't solve everything

It has also come to my attention that EPD and Fink Pythons both ship with broken readline (logical for EPD, given readline's license). However, neither of these report as being anything other than GNU readline, so we can't detect that they are broken.

It's possible that we could print a warning if we detect that we are in EPD and we get to step 3 without readline, but if EPD's readline ever does work, then that message would be misleading, and we would have no real way to know that the issue has been fixed.

The answer remains, in all cases (except for possibly Python.org Python and macports, only because they are untested), that the first command OSX Python users should ever do is easy_install readline. Not just the System Python. After this merge, pip does work to override readline in IPython, but not anywhere else in your Python world, so the warning message does note that pip install will likely not work, even though it does within IPython.

Min RK add dirty trick for readline import on OSX
also made the libedit warning extremely loud, so people don't miss it.
We still get reports of people never having noticed the warning, and
getting confused when readline is broken on OSX.

The reason for the dirty trick:

    pip installs to site-packages by default, but site-packages dirs always
    come *after* lib-dynload (and extras, etc.), which is where the system
    readline is installed.

    That means that a non-setuptools install (pip or setup.py install) *cannot*
    override any package that ships with OSX, including:
        numpy, readline, twisted, pyobjc
    without installing to a non-standard path (not even user site-packages via `--user`).

The method for the dirty trick:

    1. remove lib-dynload from sys.path before trying to import readline the first time
    2. after import, restore lib-dynload to its place in sys.path
    3. if import failed without lib-dynload, try it one more time
387dcd6
IPython/utils/rlineimpl.py
@@ -17,6 +17,18 @@ import warnings
17 17
 
18 18
 from subprocess import Popen, PIPE
19 19
 
  20
+if sys.platform == 'darwin':
  21
+    # dirty trick, to skip the system readline, because pip-installed readline
  22
+    # will never be found on OSX, since lib-dynload always comes ahead of site-packages
  23
+    from distutils import sysconfig
  24
+    lib_dynload = sysconfig.get_config_var('DESTSHARED')
  25
+    del sysconfig
  26
+    try:
  27
+        dynload_idx = sys.path.index(lib_dynload)
  28
+    except ValueError:
  29
+        dynload_idx = -1
2
Fernando Perez Owner
fperez added a note October 27, 2011

I'd use None instead of -1 for the sentinel, since it's a bit more obvious that a None index can never be actually correct. That will make the check below read if dynload_idx is not None, which also reads better (to me).

Min RK Owner
minrk added a note October 27, 2011

Sure, I'll do that. It's definitely more Pythonic, I have just gotten used to c-style '-1 index means not found'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
IPython/utils/rlineimpl.py
((43 lines not shown))
93  
-            "including some crashes on tab completion, and incorrect history navigation. "
94  
-            "It is highly recommended that you install readline, "
95  
-            "which is easy_installable with: 'easy_install readline'",
96  
-            RuntimeWarning)
97  
-        uses_libedit = True
98  
-    # cleanup names
99  
-    del dt,p,out,err
  86
+if have_readline:
  87
+    # Official Python docs state that 'libedit' is in the docstring for libedit readline:
  88
+    uses_libedit = 'libedit' in _rl.__doc__
  89
+    # Note that many non-System Pythons also do not use proper readline,
  90
+    # but do not report libedit at all, nor are they linked dynamically against libedit.
  91
+    # known culprits of this include: EPD, Fink
  92
+    # There is not much we can do to detect this, until we find a specific failure
  93
+    # case, rather than relying on the readline module to self-identify as broken.
  94
+if uses_libedit and sys.platform == 'darwin':
1
Fernando Perez Owner
fperez added a note October 27, 2011

I'd add a line of whitespace above 94 to separate these two big if blocks for readability

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Fernando Perez
Owner

Other than the minor edits above, I have nothing against the approach. We should probably ping Enthought to ask them to make their readline identifiable in some manner though, I'm sure they could just add a flag to their module to allow us to recognize it.

I haven't run the code myself, but given your OSX expertise and how much you've dealt with this issue, I'm OK if you want to merge once the above tiny things are fixed. Though if you want a second pair of OSX-specific eyes, that's OK too.

Min RK
Owner

I do worry about mucking with the path, but I was helping someone out today, explaining why pip doesn't work, and they asked if I could just instruct Python to ignore the system one, allowing the site-packages to be searched for readline, and I realized it actually is possible. I tested it, and it certainly does work - pip-installed readline will be found by IPython, at least with System Python and EPD on my machine.

I honestly don't know why EPD and/or fink readline are broken - we should ask Ilan, or someone. Just this week, I have had people with EPD at py4science and Fink (in #915) report weird readline behavior (not the same), and both found that their respective weirdnesses were resolved by easy_install readline. If it is that they are linked against libedit (fink is not, but EPD might be) and just not reporting it, that would make sense, and it is a bug that their readline.__doc__ is wrong. But I don't know what the default behavior is, when you build Python and readline is not present.

Min RK STY: rlineimpl review from @fperez
* add some line breaks
* use None to indicate dynload not found, instead of -1
d86e772
Min RK
Owner

style comments addressed

Fernando Perez
Owner

I'd say go for this one, but a ping to Ilan on enthought-dev would be probably a good idea. Sorry to dump that one on you, but I simply don't know enough about osx to say anything intelligent on such a discussion...

Fernando Perez
Owner

Awesome, merging now!

Fernando Perez fperez merged commit 90b0139 into from October 27, 2011
Fernando Perez fperez closed this October 27, 2011
Fernando Perez fperez referenced this pull request from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Min RK minrk referenced this pull request in ludwigschwardt/python-gnureadline December 08, 2013
Merged

put readline.so in python_readline.readline #26

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 2 unique commits by 1 author.

Oct 27, 2011
Min RK add dirty trick for readline import on OSX
also made the libedit warning extremely loud, so people don't miss it.
We still get reports of people never having noticed the warning, and
getting confused when readline is broken on OSX.

The reason for the dirty trick:

    pip installs to site-packages by default, but site-packages dirs always
    come *after* lib-dynload (and extras, etc.), which is where the system
    readline is installed.

    That means that a non-setuptools install (pip or setup.py install) *cannot*
    override any package that ships with OSX, including:
        numpy, readline, twisted, pyobjc
    without installing to a non-standard path (not even user site-packages via `--user`).

The method for the dirty trick:

    1. remove lib-dynload from sys.path before trying to import readline the first time
    2. after import, restore lib-dynload to its place in sys.path
    3. if import failed without lib-dynload, try it one more time
387dcd6
Min RK STY: rlineimpl review from @fperez
* add some line breaks
* use None to indicate dynload not found, instead of -1
d86e772
This page is out of date. Refresh to see the latest.

Showing 1 changed file with 56 additions and 44 deletions. Show diff stats Hide diff stats

  1. 100  IPython/utils/rlineimpl.py
100  IPython/utils/rlineimpl.py
@@ -17,6 +17,18 @@
17 17
 
18 18
 from subprocess import Popen, PIPE
19 19
 
  20
+if sys.platform == 'darwin':
  21
+    # dirty trick, to skip the system readline, because pip-installed readline
  22
+    # will never be found on OSX, since lib-dynload always comes ahead of site-packages
  23
+    from distutils import sysconfig
  24
+    lib_dynload = sysconfig.get_config_var('DESTSHARED')
  25
+    del sysconfig
  26
+    try:
  27
+        dynload_idx = sys.path.index(lib_dynload)
  28
+    except ValueError:
  29
+        dynload_idx = None
  30
+    else:
  31
+        sys.path.pop(dynload_idx)
20 32
 try:
21 33
     from readline import *
22 34
     import readline as _rl
@@ -29,6 +41,26 @@
29 41
     except ImportError:
30 42
         have_readline = False
31 43
 
  44
+if sys.platform == 'darwin':
  45
+    # dirty trick, part II:
  46
+    if dynload_idx is not None:
  47
+        # restore path
  48
+        sys.path.insert(dynload_idx, lib_dynload)
  49
+        if not have_readline:
  50
+            # *only* have system readline, try import again
  51
+            try:
  52
+                from readline import *
  53
+                import readline as _rl
  54
+                have_readline = True
  55
+            except ImportError:
  56
+                have_readline = False
  57
+            else:
  58
+                # if we want to warn about EPD / Fink having bad readline
  59
+                # we would do it here
  60
+                pass
  61
+    # cleanup dirty trick vars
  62
+    del dynload_idx, lib_dynload
  63
+
32 64
 if have_readline and hasattr(_rl, 'rlmain'):
33 65
     # patch add_history to allow for strings in pyreadline <= 1.5:
34 66
     # fix copied from pyreadline 1.6
@@ -51,52 +83,32 @@ def add_history(line):
51 83
 # Test to see if libedit is being used instead of GNU readline.
52 84
 # Thanks to Boyd Waters for the original patch.
53 85
 uses_libedit = False
54  
-if sys.platform == 'darwin' and have_readline:
55  
-    # Previously this used commands.getstatusoutput, which uses os.popen.
56  
-    # Switching to subprocess.Popen, and exponential falloff for EINTR
57  
-    # seems to make this better behaved in environments such as PyQt and gdb
58  
-    dt = 1e-3
59  
-    while dt < 1:
60  
-        try:
61  
-            p = Popen(['otool', '-L', _rl.__file__], stdout=PIPE, stderr=PIPE)
62  
-        except OSError:
63  
-            try:
64  
-                # otool not available (no XCode), use lsof instead.
65  
-                # This *could* have a false positive
66  
-                # if another package that uses libedit explicitly
67  
-                # has been imported prior to this test.
68  
-                p = Popen(['lsof', '-p', str(os.getpid())], stdout=PIPE, stderr=PIPE)
69  
-            except OSError:
70  
-                # This is highly unlikely, but let's be sure
71  
-                # we don't crash IPython just because we can't find lsof
72  
-                p = out = err = None
73  
-                warnings.warn("libedit detection failed")
74  
-                break
75  
-
76  
-        out,err = p.communicate()
77 86
 
78  
-        if p.returncode == 4:
79  
-            # EINTR
80  
-            time.sleep(dt)
81  
-            dt *= 2
82  
-            continue
83  
-        elif p is None or p.returncode:
84  
-            warnings.warn("libedit detection failed: %s"%err)
85  
-            break
86  
-        else:
87  
-            break
  87
+if have_readline:
  88
+    # Official Python docs state that 'libedit' is in the docstring for libedit readline:
  89
+    uses_libedit = 'libedit' in _rl.__doc__
  90
+    # Note that many non-System Pythons also do not use proper readline,
  91
+    # but do not report libedit at all, nor are they linked dynamically against libedit.
  92
+    # known culprits of this include: EPD, Fink
  93
+    # There is not much we can do to detect this, until we find a specific failure
  94
+    # case, rather than relying on the readline module to self-identify as broken.
88 95
 
89  
-    if p is not None and p.returncode == 0 and re.search(br'/libedit[\.\d+]*\.dylib\s', out):
90  
-        # we are bound to libedit - new in Leopard
91  
-        _rl.parse_and_bind("bind ^I rl_complete")
92  
-        warnings.warn("Leopard libedit detected - readline will not be well behaved "
93  
-            "including some crashes on tab completion, and incorrect history navigation. "
94  
-            "It is highly recommended that you install readline, "
95  
-            "which is easy_installable with: 'easy_install readline'",
96  
-            RuntimeWarning)
97  
-        uses_libedit = True
98  
-    # cleanup names
99  
-    del dt,p,out,err
  96
+if uses_libedit and sys.platform == 'darwin':
  97
+    _rl.parse_and_bind("bind ^I rl_complete")
  98
+    warnings.warn('\n'.join(['', "*"*78,
  99
+        "libedit detected - readline will not be well behaved, including but not limited to:",
  100
+        "   * crashes on tab completion",
  101
+        "   * incorrect history navigation",
  102
+        "   * corrupting long-lines",
  103
+        "   * failure to wrap or indent lines properly",
  104
+        "It is highly recommended that you install readline, which is easy_installable:",
  105
+        "     easy_install readline",
  106
+        "Note that `pip install readline` generally DOES NOT WORK, because",
  107
+        "it installs to site-packages, which come *after* lib-dynload in sys.path,",
  108
+        "where readline is located.  It must be `easy_install readline`, or to a custom",
  109
+        "location on your PYTHONPATH (even --user comes after lib-dyload).",
  110
+        "*"*78]),
  111
+        RuntimeWarning)
100 112
 
101 113
 # the clear_history() function was only introduced in Python 2.4 and is
102 114
 # actually optional in the readline API, so we must explicitly check for its
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.