loading TCL / Tk symbols dynamically

[matthew-brett]

This is an attempt to load the symbols we need from the Tkinter.tkinter
module at run time, rather than by linking at build time.

It is one way of building a manylinux wheel that can build on a basic
Manylinux docker image, and then run with the TCL / Tk library installed
on the installing user's machine. It would also make the situation
better on OSX, where we have to build against ActiveState TCL for
compatibility with Python.org Python, but we would like to allow
run-time TCL from e.g. homebrew.

I have tested this on Debian Jessie Python 2.7 and 3.5, and on OSX 10.9
with Python 2.7.

Questions:

• Would y'all consider carrying something like this approach in
  the matplotlib source, but not enabled by default, to help building
  binary wheels?
• Do you have any better suggestions about how to do this?
• My C fu is weak; is there a way of collecting the typedefs I need from
  the TCL / Tk headers rather than copying them into the _tkagg.cpp
  source file (typdefs starting around line 52)?
• My fu for Python C extension modules is also weak; did I configure
  exceptions and handle references correctly?

[mdboom]

Thanks for this. This is really helpful.

@matthew-brett wrote:

Would y'all consider carrying something like this approach in the matplotlib source, but not enabled by default, to help building binary wheels?

I'd prefer to give this loads of testing and just have this as the "one way to do it" if we can. We could eliminate tons of hacks in the build script and tons of pain and support using this approach, at the expense of the very minor dynamic linking at import time.

My C fu is weak; is there a way of collecting the typedefs I need from the TCL / Tk headers rather than copying them into the _tkagg.cpp source file (typdefs starting around line 52)?

I don't think so.

My fu for Python C extension modules is also weak; did I configure exceptions and handle references correctly?

I'll read through and try to verify.

[mdboom]

This isn't exception proof -- PyUnicode_EncodeFSDefault could raise an exception here. Maybe just make a regular old function for this purpose?

[mdboom]

I think this only needs to be called if func == NULL.

[mdboom]

This is fine, but is a sort of unidiomatic pattern. Rather than using a static variable has_error here, why not just return NULL if there was an error and then call it differently from _func_loader, e.g.:

if (!((TCL_CREATE_COMMAND = (tcl_cc) _dfunc(tkinter_lib, "Tcl_CreateCommand")) &&
      (TCL_APPEND_RESULT = (tcl_cc) _dfunc(tkinter_lib, "Tcl_AppendResult")) &&
    ... etc ..

[mdboom]

It's a matter of personal preference, but I generally like to avoid the excessive nesting by:

1. setting all PyObject * to NULL in their declaration
2. doing:

if (pModule == NULL) {
  goto exit;
}

3. having

exit:
   Py_XDECREF(pString);
   Py_XDECREF(pSubmodule);
   Py_XDECREF(pModule);

   return ret;

at the bottom

[QuLogic]

My C fu is weak; is there a way of collecting the typedefs I need from the TCL / Tk headers rather than copying them into the _tkagg.cpp source file (typdefs starting around line 52)?

What's wrong with #includeing the header?

[mdboom]

What's wrong with #includeing the header?

Because the point of this change is to replace actual direct function calls with function pointers. Numpy, for example, has a special generated header for this exact purpose (__multiarray_api.h), but most C libraries don't.

[QuLogic]

I was hoping since it's C++ you could use some auto magic, but that might be C++11 only.

[dopplershift]

auto is static (i.e. compile-time) typing only, so there's no way that could work for runtime type inference. That's the realm of C++ virtual functions/polymorphism.

[QuLogic]

We don't need runtime typing; we just need the type of the function at compile time so that we can set up a pointer to it.

[dopplershift]

The return type of dlsym() (and hence _dfunc) is void *; there is absolutely no way for the compiler to help you out here.

[QuLogic]

I didn't really mean auto, just some "C++ magic" like it. And that something would be typeof or decltype, but both of these come with their own issues (one's a GNU-ism, the other's C++11 only).

[ogrisel]

This include fails when using Visual C++ for Python 2.7:

src/_tkagg.cpp(270) : fatal error C1083: Cannot open include file: 'dlfcn.h': No such file or directory

https://ci.appveyor.com/project/mdboom/matplotlib/build/1.0.1545/job/724qv5hcfngqa2r9#L1033

[ogrisel]

Actually more recent versions of Visual C++ also fail, e.g. in the Python 3.5 Windows build on AppVeyor:

src/_tkagg.cpp(270): fatal error C1083: Cannot open include file: 'dlfcn.h': No such file or directory 
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\amd64\\cl.exe' failed with exit status 2
Command exited with code 1

https://ci.appveyor.com/project/mdboom/matplotlib/build/1.0.1545/job/rk6safsaid63i7ch#L1053

[matthew-brett]

@mdboom - addressed your comments - I think. I've tested disabling the linker flags for TCL / Tk libraries in setupext.py - works on OSX Python 2.7, 3.5, Debian Jessie 2.7 and 3.5.

[mdboom]

Cool. If this works, we might as well take most of the crazy header-file-hunting out of setupext.py as well.

As for the Windows-specific issues, @cgohlke, any thoughts?

[matthew-brett]

I've done some speculative changes to support Windows, trying to work out how to test them now.

[matthew-brett]

OK - Windows changes are compiling and passing tests on Python 2.7 at least.

I propose to remove the not-dynamic branch, and disable linking the TCL / Tk libs in setupext.py to see if this can work without build-time linking.

[matthew-brett]

Build-time linking disabled, Appveyor tests passing, failure on travis I believe is unrelated, so from my point of view, I think this is ready.

[cgohlke]

Maybe I am missing something, but how can this work on Windows where the Tcl/Tk symbols are not exported from _tkinter.pyd? Is appveyor testing TkAgg?

>>> from ctypes import *
>>> from ctypes.wintypes import *
>>> windll.kernel32.LoadLibraryW.restype = HMODULE
>>> windll.kernel32.LoadLibraryW.argtypes = [LPCWSTR]
>>> windll.kernel32.GetProcAddress.argtypes = [HMODULE, LPCSTR]
>>> windll.kernel32.GetProcAddress.restype = c_void_p
>>> from tkinter import _tkinter
>>> hdll = windll.kernel32.LoadLibraryW(_tkinter.__file__)
>>> print(windll.kernel32.GetProcAddress(hdll, b"PyInit__tkinter"))
140732219092336
>>> print(windll.kernel32.GetProcAddress(hdll, b"Tcl_CreateCommand"))
None

[efiring]

Quick test with a conda python 3.5 environment on OS X 10.9.5: it works! I was not able to get a working Tk backend without this.
Maybe unrelated to this, but I get lots of warnings while executing:

2016-05-18 10:51:04.230 python[18741:507] setCanCycle: is deprecated.  Please use setCollectionBehavior instead

[matthew-brett]

Christoph - sorry - I'm behind a fierce proxy in Cuba, and I can't compile myself. Google is blocking me from downloading the Appveyor build artifacts with We're sorry, but this service is not available in your location (Google code blocks Cuba too).

The code at https://github.com/matplotlib/matplotlib/pull/6442/files#diff-c4f5727f595ec7b41e86800bb717a0efR323 should be checking for null pointers at run time, and raising an exception.

What do you get if you try and make and show a plot with the tkagg backend using one the build artifacts on Appveyor - e.g. https://ci.appveyor.com/project/mdboom/matplotlib/build/1.0.1553/job/c4lm62wyn1541509/artifacts ?

[cgohlke]

Using matplotlib-1.5.1+1973.g7257404-cp27-cp27m-win_amd64.whl:

>>> import matplotlib
>>> matplotlib.use('TkAgg')
>>> matplotlib.__version__
'1.5.1+1973.g7257404'
>>> from matplotlib import pyplot
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "matplotlib\pyplot.py", line 113, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
  File "matplotlib\backends\__init__.py", line 55, in pylab_setup
    [backend_name], 0)
  File "matplotlib\backends\backend_tkagg.py", line 13, in <module>
    import matplotlib.backends.tkagg as tkagg
  File "matplotlib\backends\tkagg.py", line 9, in <module>
    from matplotlib.backends import _tkagg
RuntimeError: Cannot load function Tcl_CreateCommand

Also, the artifact is missing zlib.dll:

>>> from matplotlib import pyplot
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "matplotlib\pyplot.py", line 29, in <module>
    import matplotlib.colorbar
  File "matplotlib\colorbar.py", line 34, in <module>
    import matplotlib.collections as collections
  File "matplotlib\collections.py", line 27, in <module>
    import matplotlib.backend_bases as backend_bases
  File "matplotlib\backend_bases.py", line 63, in <module>
    import matplotlib.textpath as textpath
  File "matplotlib\textpath.py", line 20, in <module>
    from matplotlib.mathtext import MathTextParser
  File "matplotlib\mathtext.py", line 62, in <module>
    import matplotlib._png as _png
ImportError: DLL load failed: The specified module could not be found.

[matthew-brett]

The png failure must be unrelated. I suppose the GetProcAddress failure is related to this

Windows allows programmers to access symbols exported by the main executable. 
Windows does not use a global symbol table, and has no API to search across multiple
modules to find a symbol by name.

So, the next question is - do you (Christoph) or does anyone know how to query _tkinter.pyd at run time for its library dependencies?

[matthew-brett]

Rather than analyze the DLL file and add an extra dependency, I am trying to search through the loaded modules in the current process, looking for the TCL / Tk routines. This seems to be very fast on simple tests.

[matthew-brett]

Christoph - would you mind checking https://ci.appveyor.com/project/mdboom/matplotlib/build/1.0.1558/job/snxq695d8h4sgqli/artifacts

[matthew-brett]

I finally got a Windows test install working - this code works for me.

[mdboom]

Do we need any of this stuff (including finding header files) anymore if we've duplicated the function signatures (as function pointer signatures) in our own code now anyway?

[matthew-brett]

Well, we still need the definitions for these data types:

    Tcl_Interp
    Tcl_Command
    Tk_PhotoHandle
    Tk_PhotoImageBlock
    Tk_Window
    Tcl_CmdProc
    ClientData
    Tcl_CmdDeleteProc

I guess we could copy those out of some TCL / Tk headers and then be entirely free of the system TCL / Tk.

[matthew-brett]

I adding a minimal set of defines from TCL / Tk in the latest commit. It seems to compile and run correctly on OSX, Linux and Windows, but I would appreciate some eyes on the minimal header, because of my aforementioned C-fu weakness.

If this new header is OK, then I'll go ahead and delete all the TCL / Tk finding logic in setupext.py.

[matthew-brett]

I did some timing on the tcl / tk symbol search on Windows. It's very fast. Package / function to replicate the search at https://github.com/matthew-brett/tktest:

In [1]: import numpy
In [2]: from scipy import io, stats, ndimage, optimize, sparse
In [3]: from sklearn import *
In [4]: from skimage import *
In [5]: from tktest import search_for_tk
In [6]: timeit search_for_tk()
10000 loops, best of 3: 106 µs per loop

[matthew-brett]

Test failures seem to be due to freetype source server, not these changes.

[matthew-brett]

OK - I checked current Tcl / Tk trunk, and I've edited the header excerpts, so they are compatible from Tcl / Tk 8.4 through current Tcl / Tk trunk. I think we should get away with these header excerpts for a long time.

I've removed all the Tcl / Tk discovery stuff as no longer needed, and tested on OSX, Linux and Windows. I think this one is ready to merge.

[matthew-brett]

Github seems to be having trouble today - the latest commit for this branch is de99a76 - I don't see that yet in this interface. When it arrives, you should have the changes I referred to in my last message.

[tacaswell]

👍 on this getting backported to 1.5.2rc1. As soon as that is done I will do an rc2, still with the target of a final next weekend.

[efiring]

Travis passed. Appveyor stalled in the build stage on the Py3.5 job. I didn't see anything indicating the build failure was related to the PR, so I am closing and reopening to try again.

[efiring]

Well, I don't have any idea what the problem is, but Appveyor failed to build on Python 3.5 just like last time. Very little output is generated. It seems to stall before actually doing any conda installations.

[matthew-brett]

Appveyor now passing on all Pythons for some reason. Travis failure looks unrelated. Appveyor and Travis tests now testing tkagg import and succeeding. Tests of this branch on OSX at MacPython/matplotlib-wheels finding and importing tkagg correctly (unrelated failures but tests passing on Pythons 2.7 3.4 3.5).

[efiring]

Everything passed and Appveyor showed the tkagg test result correctly, but it looks like Travis either didn't execute the tkagg import test, or swallowed its output:

The command "echo Testing import of tkagg backend
MPLBACKEND="tkagg" python -c 'import matplotlib.pyplot as plt; print(plt.get_backend())'
echo Testing using $NPROC processes

[matthew-brett]

OSX builds now green - successful tkagg import checks for Python 2.7 Python 3.4 Python 3.5.

[matthew-brett]

Eric - I think the missing output is just because the command is in a block with other commands. The output is further down after the whole command input block - e.g. https://travis-ci.org/matplotlib/matplotlib/jobs/132588520#L1018

[efiring]

backported to v1.5.x as b80e0f1

[cournape]

When updating matplotlib for canopy, I am seeing the error "Could not find TCL routines" with 1.5.3.

At first, I suspect an issue in how we build python for canopy, but I have been able to reproduce the issue with python 2.7.12 from python.org and just installing wheels from Gholke website. The failure appears for both 1.5.3 and 2.0.0.

I could also reproduce the issue w/ 1.5.3 using conda (2.7.13 this time)

[cournape]

Hm, that should really be an issue on its own, sorry