Investigate the use of python wheels for python wrapping

[chrisdembia]

https://github.com/ogrisel/python-appveyor-demo

http://pythonwheels.com/

@msdemers do you know anything about wheels?

[moorepants]

Wheels are binary distributions of python packages. They are very suitable for packages that are not pure Python packages that have C extensions. It is likely possible to build binary versions of opensim+python wrappings and distribute them as wheels. The typical way this is handled though, is that opensim should be installed via a standard package manager. You could then build Python wheels that link against binaries of opensim that are installed into typical places. But in general, I don't think Python wheels are suitable for distributing big C/C++ apps that have Python wrappers. Opensim is more than just a Python C extension.

There is a better route though!

First...the "best" route is to just get opensim into various distributions so they can be installed with standard package managers: apt, yum, homebrew, etc. The problem with this approach is that you have to support many package managers and these are typically slow to update because the various distributions have lengthy release cycles.

But there is a savior that exists just for applications like opensim!

conda!

I would highly recommend investing in building simbody/opensim packages for the conda package manager. conda is cross platform and language agnostic (python wheels are not language agnostic). I started building a simbody package for conda a while back. I think all of the simbody dependencies already have conda packages. It shouldn't be too hard to get a simbody and opensim package built for conda for all architectures. Once that is done all you have to do is type (same on all OSs):

$ conda install opensim
$ python
>>> import opensim

You could even distribute the GUI this way and there would be one simple command to install the GUI for all OSs. And it doesn't require root privileges on any system!

[moorepants]

Also, the binstar service is available that you can automate building all of your conda binaries on all major architectures. binstar also acts as the main binary hosting service.

[chrisdembia]

thanks @moorepants . If we decide to distribute via conda, https://github.com/libdynd/libdynd might be a good example.

[moorepants]

FYI: https://github.com/rmcgibbo/python-appveyor-conda-example Might be useful, as it is a fork of Oliver's demo that you listed above but for conda packages.

[chrisdembia]

Thank you for that. I feel that a key factor in deciding which distribution system to use is how likely it is that users already have it. For this reason, something like "biicode" may become important as well.

[moorepants]

That is a very important factor and of course the most popular package mangers are apt, yum, homebrew, macports, etc [1]. But there are no package managers that both all OS's have access to and are popular. I mentioned conda because it is likely now the most popular way to install the scientific python stack and if you are targeting scientific Python users with the Opensim Python API, then it is probably the right choice. If you choose conda for making your Python support super easy for users, then you automatically get cross platform builds of Opensim as a freebie and your C++ and Matlab users can also benefit from this. Conda is not popular outside the scientific Python world yet, so you may need to support other package managers for popularity reasons (and maybe some technical too). But to address the, assumed, point of this issue, i.e. how can we provide opensim easily to Python users?, I believe investment in conda is both the correct technical choice and the popular choice.

I'll see if I can finish up my conda packages for simbody and opensim sometime and I'll share them with you all.

[1] not sure if there are any super popular ones for windows

[chrisdembia]

There's chocolately for windows, but I don't know if it's "popular." Thanks for refocusing this on python. Do you think conda is now more popular than pip?

[moorepants]

No, conda is definitely not more popular than pip for the general Python audience. The general Python audience is dominated by web devs, sys admins, etc that do not typically require software that is complicated to build. But I think it is likely the most popular way to install Python for the scientific Python audience. If your target audience is scientific users, then most of them will have conda and/or at have least used it.

The Python packaging ecosystem (pypi, pip, easy_install, setuptools, wheels, etc) are all centered around distributing Python packages. A Python package is typically a pure python source distribution or one with some C extensions (i.e. small bits of C code that are compiled on install that typically only require a compiler and no compilation dependencies besides Python). The Python system can be used to distribute Python packages that have huge "C extensions" which have big dependency lists and is even able to distribute wrappers for large projects in other languages. But if you investigate these projects you'll find that they have a really hard shoving their build system into the Python system. For projects like those, most people install from binary packages. For example, try installing wxpython, VTK python bindings, or simply try installing the scipy stack on windows with pip and you'll typically find that you have to become fluent at building complex software from source. As mentioned earlier, to install Python bindings for large software packages you typically install binaries of the base package and then use the Python packaging tools to dynamically link the wrappers during the wrapper complication to your core lib. So, that's why binary based package managers exist and conda is a relatively new binary based, language and os agnostic, package manager that grew out of scientific python users hate of having to compile all this software all the time to use in Python.

[chrisdembia]

Wow! Very helpful! Thank you for that.

[moorepants]

I'll take that as a "thanks, now shut up!" 😄

[chrisdembia]

Nooo! I just don't know when I'll find time to work on this.

[moorepants]

I'm super busy the next 3 weeks, but maybe I'll find some time to play with this.

[chrisdembia]

I think we'll just go with conda.

[aymanhab]

@chrisdembia What distribution options would help @kidzik ?

[chrisdembia]

@kidzik has already set up his own conda package but he has to build them locally. If we get our software into conda-forge, then it is built automatically for all 3 major platforms. I've chatted with @kidzik a little bit and we are attempting to update the simbody conda package here: conda-forge/simbody-feedstock#23

[moorepants]

It is worth noting that several C++ packages are now distributed via wheels. For example VTK can be installed with pip: https://pypi.org/project/vtk/.

[chrisdembia]

Ah thanks. It would be nice not to have to build the packages locally. Seems like we could use https://github.com/joerick/cibuildwheel to build the wheels on CI.

[moorepants]

It may be possible to create a wheel from a conda package too. I've heard allusions to that.

[chrisdembia]

We can have it all!