Building 0.4.x on Python 3 breaks for Python 2 and vice versa

[benjaoming]

Summary

This happens already in 0.4.2 (and probably also previous editions)

When built with Python 3, the dist only works for Python 3 (and fails on Python 2.7). I haven't tested the converse.

I'm pretty sure this affects Pex files as well since they're built from .whl files. We already have an issue similar to this in #1710 - namely that Pex is broken on Python 3 environments with namespaced packages. This narrows the window regarding how many (few!) Python environments that the Pex builds will work with.

Traceback or relevant snippet from server.log or browser console

Failure in Py2 when built on Py3:

➜  ~ kolibri start             
Traceback (most recent call last):
  File ".virtualenvs/kolibri_pip/bin/kolibri", line 11, in <module>
    sys.exit(main())
  File ".virtualenvs/kolibri_pip/local/lib/python2.7/site-packages/kolibri/utils/cli.py", line 283, in main
    django.setup()
  File ".virtualenvs/kolibri_pip/lib/python2.7/site-packages/kolibri/dist/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File ".virtualenvs/kolibri_pip/lib/python2.7/site-packages/kolibri/dist/django/apps/registry.py", line 115, in populate
    app_config.ready()
  File ".virtualenvs/kolibri_pip/lib/python2.7/site-packages/kolibri/dist/django/contrib/admin/apps.py", line 22, in ready
    self.module.autodiscover()
  File ".virtualenvs/kolibri_pip/lib/python2.7/site-packages/kolibri/dist/django/contrib/admin/__init__.py", line 26, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File ".virtualenvs/kolibri_pip/lib/python2.7/site-packages/kolibri/dist/django/utils/module_loading.py", line 50, in autodiscover_modules
    import_module('%s.%s' % (app_config.name, module_to_search))
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File ".virtualenvs/kolibri_pip/lib/python2.7/site-packages/kolibri/dist/django_q/admin.py", line 5, in <module>
    from .tasks import async
  File ".virtualenvs/kolibri_pip/lib/python2.7/site-packages/kolibri/dist/django_q/tasks.py", line 11, in <module>
    import cluster
  File ".virtualenvs/kolibri_pip/lib/python2.7/site-packages/kolibri/dist/django_q/cluster.py", line 11, in <module>
    standard_library.install_aliases()
  File ".virtualenvs/kolibri_pip/lib/python2.7/site-packages/kolibri/dist/future/standard_library/__init__.py", line 451, in install_aliases
    __import__(newmodname)
ImportError: No module named reprlib

Failure in Py3 when built on Py2:

➜  ~ kolibri start --foreground
Traceback (most recent call last):
  File ".virtualenvs/kolibri_pip/bin/kolibri", line 7, in <module>
    from kolibri.utils.cli import main
  File ".virtualenvs/kolibri_pip/local/lib/python3.5/site-packages/kolibri/utils/cli.py", line 27, in <module>
    from . import server  # noqa
  File ".virtualenvs/kolibri_pip/local/lib/python3.5/site-packages/kolibri/utils/server.py", line 6, in <module>
    import cherrypy
  File ".virtualenvs/kolibri_pip/lib/python3.5/site-packages/kolibri/dist/cherrypy/__init__.py", line 64, in <module>
    from cherrypy._cperror import HTTPError, HTTPRedirect, InternalRedirect
  File ".virtualenvs/kolibri_pip/lib/python3.5/site-packages/kolibri/dist/cherrypy/_cperror.py", line 118, in <module>
    from cgi import escape as _escape
  File "/usr/lib/python3.5/cgi.py", line 42, in <module>
    import html
  File ".virtualenvs/kolibri_pip/lib/python3.5/site-packages/kolibri/dist/html/__init__.py", line 7, in <module>
    raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

How to reproduce

1. make dist from a Python 3 env (meaning that python command points to Python 3)
2. Install dist/x in a Python 2 env.

[benjaoming]

From the docs:

Pure Python Wheels that are not “universal” are wheels that are pure python (i.e. contains no compiled extensions), but don’t natively support both Python 2 and 3.

https://packaging.python.org/tutorials/distributing-packages/#pure-python-wheels

If your code supports both Python 2 and 3, but with different code (e.g., you use “2to3”) you can run setup.py bdist_wheel twice, once with Python 2 and once with Python 3. This will produce wheels for each version.

I'd rather eliminate the problem that causes the dist to only work on Py3.

[benjaoming]

Confirming that creating the dist in a Python 2 env, works for another Python 2 env, but fails in Python 3!

[benjaoming]

Building 0.4.3 with Python 2 once again, I can confirm that the problem persists.

The pex file will because of an option given to the Pex builder in make pex (--python-shebang=/usr/bin/python) by default run with Python 2 (I don't know about any systems but Arch Linux that doesn't have Python 2 as default?)

The problem seems to be unnecessary though:

from __future__ import absolute_import
import sys

if sys.version_info[0] < 3:
    from future.moves.html import *
else:
    raise ImportError('This package should not be accessible on Python 3. '
                      'Either you are trying to run from the python-future src folder '
                      'or your installation of python-future is corrupted.')

(example is from kolibri/dist/html installed when the dist future is installed under Python 2, the package html is not installed when future's setup.py gets called in Python 3!)

We could have a package called html that just gracefully allows import from Python 3 system modules... but that requires messing with kolibri/dist after installing dependencies, which is... a mess.

[benjaoming]

This all boils down to django-q being dependent on future, a package which thinks it's great to build a compatibility layer that blows up when loaded on Python 3. So much for compatibility.

I suggest we fix it my ridding django-q from this dependency, since they already advertise as py2.py3 on PyPi, so it could be considered a slight problem.

[benjaoming]

I think this needs doing for 0.4:

We release two separate builds manually on PyPi. I'll do that.

For 0.5, I could try fixing django-q in our own build and send a PR upstream.

This is very vital to have fixed, I predict a lot of fuzz about not know whether something was built for Py2 or Py3, since this is not part of our labelling of builds.

[aronasorman]

This is intentional for all builds not containing barbequeue. django-q, the previous queueing system, depended on future. futureis a compatibility library, but it sets things up during build time, not runtime.

[aronasorman]

Might be better to just release two version of kolibri for 4.x on PyPI, if we are planning to do that.

[benjaoming]

Fixed in #1810