Silent fail when use use_2to3 but not available

[buxx]

Hello,

I recently had issues with an hand made docker image. When installing some package with python3, installed code was python2 code instead python3 code. Explanation of this issue is:

• Installed package use use_2to3 parameter to convert it's code into python3 code.
• 2to3 command was not present in the system

Problem here is pip install successfully the package while 2to3 is not available. It should crash no ?

Thank's,
Bux.

[twschiller]

I'm seeing behavior consistent with this when trying to install the anyjson 0.3.3 package on Python 3.6.1 x64 on Azure App Service (Kudu) with setuptools 36.2.7 and pip 9.0.1

The __init__.py of the anyjson package contains Python 2-style exception handling.

Here's the pip log. What should the log look like if 2to3 runs successfully?

 Running setup.py (path:D:\local\Temp\pip-build-4kv5aaxb\anyjson\setup.py) egg_info for package anyjson
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info\anyjson.egg-info
    writing pip-egg-info\anyjson.egg-info\PKG-INFO
    writing dependency_links to pip-egg-info\anyjson.egg-info\dependency_links.txt
    writing top-level names to pip-egg-info\anyjson.egg-info\top_level.txt
    writing manifest file 'pip-egg-info\anyjson.egg-info\SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pip-egg-info\anyjson.egg-info\SOURCES.txt'
  Source in d:\local\temp\pip-build-4kv5aaxb\anyjson has version 0.3.3, which satisfies requirement anyjson from https://pypi.python.org/packages/c3/4d/d4089e1a3dd25b46bebdb55a992b0797cff657b4477bc32ce28038fdecbc/anyjson-0.3.3.tar.gz#md5=2ea28d6ec311aeeebaf993cb3008b27c
Installing collected packages: anyjson
  Running setup.py install for anyjson: started
    Running command D:\home\python361x64\python.exe -u -c "import setuptools, tokenize;__file__='D:\\local\\Temp\\pip-build-4kv5aaxb\\anyjson\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record D:\local\Temp\pip-yv1g9ufz-record\install-record.txt --single-version-externally-managed --compile
    running install
    running build
    running build_py
    creating build
    creating build\lib
    creating build\lib\anyjson
    copying anyjson\__init__.py -> build\lib\anyjson
    Fixing build\lib\anyjson\__init__.py
    Fixing build\lib\anyjson\__init__.py
    running install_lib
    creating D:\home\python361x64\Lib\site-packages\anyjson
    copying build\lib\anyjson\__init__.py -> D:\home\python361x64\Lib\site-packages\anyjson
    byte-compiling D:\home\python361x64\Lib\site-packages\anyjson\__init__.py to __init__.cpython-36.pyc
      File "D:\home\python361x64\Lib\site-packages\anyjson\__init__.py", line 88
        except self._encode_error, exc:
                                 ^
    SyntaxError: invalid syntax

    running install_egg_info
    running egg_info
    writing anyjson.egg-info\PKG-INFO
    writing dependency_links to anyjson.egg-info\dependency_links.txt
    writing top-level names to anyjson.egg-info\top_level.txt
    reading manifest template 'MANIFEST.in'
    writing manifest file 'anyjson.egg-info\SOURCES.txt'
    Copying anyjson.egg-info to D:\home\python361x64\Lib\site-packages\anyjson-0.3.3-py3.6.egg-info
    running install_scripts
    writing list of installed files to 'D:\local\Temp\pip-yv1g9ufz-record\install-record.txt'
    Running setup.py install for anyjson: finished with status 'done'

[benoit-pierre]

Running rm -rf venv && python3.6 -m venv venv && ./venv/bin/pip install -q 'setuptools==36.2.7' 'pip==9.0.1' && ./venv/bin/pip install -v anyjson:

    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/anyjson
    copying anyjson/__init__.py -> build/lib/anyjson
    Fixing build/lib/anyjson/__init__.py
    Skipping optional fixer: buffer
    Skipping optional fixer: idioms
    Skipping optional fixer: set_literal
    Skipping optional fixer: ws_comma
    Fixing build/lib/anyjson/__init__.py
    Skipping optional fixer: buffer
    Skipping optional fixer: idioms
    Skipping optional fixer: set_literal
    Skipping optional fixer: ws_comma
    running install_lib
    creating /home/bpierre/progs/src/setuptools/venv/lib/python3.6/site-packages/anyjson
    copying build/lib/anyjson/__init__.py -> /home/bpierre/progs/src/setuptools/venv/lib/python3.6/site-packages/anyjson
    byte-compiling /home/bpierre/progs/src/setuptools/venv/lib/python3.6/site-packages/anyjson/__init__.py to __init__.cpython-36.pyc
    running install_egg_info
    running egg_info
    writing anyjson.egg-info/PKG-INFO
    writing dependency_links to anyjson.egg-info/dependency_links.txt
    writing top-level names to anyjson.egg-info/top_level.txt
    reading manifest file 'anyjson.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'anyjson.egg-info/SOURCES.txt'
    Copying anyjson.egg-info to /home/bpierre/progs/src/setuptools/venv/lib/python3.6/site-packages/anyjson-0.3.3-py3.6.egg-info
    running install_scripts
    writing list of installed files to '/tmp/pip-a_hl86t6-record/install-record.txt'

[benoit-pierre]

I think setuptools directly uses lib2to3, which is a core Python 3 module.

[twschiller]

Thanks, @benoit-pierre. Is there a chance that setuptools is detecting a different Python version to determine which fixes should be run?

Trying to run install without a venv as D:\home\python361x64\python.exe -m pip install anyjson in the following environment:

D:\home>D:\home\python361x64\python.exe --version
Python 3.6.1

D:\home>python --version
Python 2.7.8

The official Azure site extensions don't come with venv support: Azure/azure-python-siteextensions#6 (comment)

[benoit-pierre]

There's a test for the Python version in anyjson' setup, but it looks okay.

Setuptools however, will silently ignore an import error for its lib2to3 support module.

What are the outputs for:

• python -m lib2to3 --list-fixes
• and python -c "from setuptools.lib2to3_ex import Mixin2to3"
  ?

[twschiller]

Looks like importing Mixin2to3 works OK, but it's not finding any fixes:

D:\home>D:\home\python361x64\python.exe -m lib2to3 --list-fixes
Available transformations for the -f/--fix option:

D:\home>D:\home\python361x64\python.exe -c "from setuptools.lib2to3_ex import Mixin2to3"

D:\home> 

The lib2to3 fixes directory does contain fixes (I cut off the console output):

D:\home\python361x64\Lib\lib2to3\fixes>ls
__init__.pyc
fix_apply.pyc
fix_asserts.pyc
fix_basestring.pyc
fix_buffer.pyc
fix_dict.pyc

[benoit-pierre]

The problem is with get_all_fix_names:

def get_all_fix_names(fixer_pkg, remove_prefix=True):
    """Return a sorted list of all available fix names in the given package."""
    pkg = __import__(fixer_pkg, [], [], ["*"])
    fixer_dir = os.path.dirname(pkg.__file__)
    fix_names = []
    for name in sorted(os.listdir(fixer_dir)):
        if name.startswith("fix_") and name.endswith(".py"):
            if remove_prefix:
                name = name[4:]
            fix_names.append(name[:-3])
    return fix_names

As you can see, the list of available fixes is determined by looking for source files starting with fix_, and since you seem to be missing those...

[twschiller]

Thanks for investigating. I'll follow up with the Azure package maintainer and the 2to3 library maintainer on the expected behavior.

From a setuptools perspective, might be good idea to issue a warning if 2to3 is enabled but no fixes are found. A library maintainer that enables 2to3 probably expects at least 1 fix to be applied during the install

[benoit-pierre]

As a workaround, you could use the following Python code to create fake source files:

from pathlib import Path
import lib2to3.fixes

fixes_dir = Path(lib2to3.fixes.__file__).parent
for pyc in fixes_dir.glob('*.pyc'):
    py = pyc.parent / (pyc.name + '.py')
    if not py.exists():
        print('creating fake', py)
        py.touch()
        pyc.touch()

Just a create a backup of the lib2to3 directory before.

[twschiller]

Thanks, I think the script might need to strip the .pyc before appending the .py so the names match, e.g., fix_apply.pyc results in fix_apply.py:

py = pyc.parent / (pyc.name[:-4] + '.py')

After running the script, I then get the following error during the anyjson installation:

Fixing build\lib\anyjson\__init__.py
    Traceback (most recent call last):
      File "D:\Repos\azure-python-siteextensions\source_packages\python.3.6.1\tools\Lib\lib2to3\refactor.py", line 249, in get_fixers
    AttributeError: module 'lib2to3.fixes.fix_apply' has no attribute 'FixApply'

Relevant function is here: https://github.com/python/cpython/blob/master/Lib/lib2to3/refactor.py#L215

My guess is the import loads the blank .py file instead of the .pyc file. For now I'm just going to manually patch the anyjson library.

[benoit-pierre]

pyc.name is the basename without the extension. When creating a fake .py, the corresponding .pyc timestamp is updated to be more recent to make sure Python does not try to recompile it from the fake .py. I did the workaround locally without issues, but I'm not on Windows. Maybe a timestamp resolution issue?

Another solution is to simply retrieve the lib2to3 sources:

> wget https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz
> unzip python-3.6.2-embed-amd64.zip python36.zip
> tar xvf Python-3.6.1.tgz --strip-components=2 Python-3.6.1/Lib/lib2to3

[benoit-pierre]

Note you can use the tarfile module if you're on Windows and don't have tar/gunzip available: python -m tarfile -e Python-3.6.1.tgz

[twschiller]

Thanks for the workarounds. For reference, here's the discussion on the Python bug tracker: https://bugs.python.org/issue31143

[jaraco]

This issue is obsolete now that use_2to3 is removed and errors actively if the parameter is supplied (v58.0.2).

[chandchowdary]

I am still getting the syntax error from anyjson==0.3.3 when using Python 3.6 any workarounds?

[jaraco]

The proper fix will be for anyjson to provide a universal Python implementation (or Python 3 only, dropping support for Python 2). Another option could be for anyjson to release Python2/3 wheels for 0.3.3 (bypassing the need for the installer to build it).

I'd welcome a fix for Setuptools use_2to3 functionality, but that functionality would be targeted to Setuptools < 58 and still would not be available on Setuptools 58.

I'd be willing to advise on the anyjson project to help modernize that codebase.

I'd recommend to contact the author and find out if they have any intention of maintaining the project or if they wish to hand off maintenance.

I found I was able to create a wheel for anyjson thus:

~ $ docker run -it jaraco/multipy-tox python3.6 -m pip-run "setuptools<58" wheel -q -- -m pip wheel --no-build-isolation anyjson
WARNING: You are using pip version 20.3.4; however, version 21.2.4 is available.
You should consider upgrading via the '/usr/bin/python3.6 -m pip install --upgrade pip' command.
Collecting anyjson
  Downloading anyjson-0.3.3.tar.gz (8.3 kB)
    Preparing wheel metadata ... done
Building wheels for collected packages: anyjson
  Building wheel for anyjson (PEP 517) ... done
  Created wheel for anyjson: filename=anyjson-0.3.3-py3-none-any.whl size=4972 sha256=58cd970b29c4f74f17fda3725fca276ad05973ce660c844195018a4490d1d02c
  Stored in directory: /root/.cache/pip/wheels/b9/bd/95/00661ac6fe3bc60296ccb45e91518eb6b57c905fd077177aeb
Successfully built anyjson
WARNING: You are using pip version 20.3.4; however, version 21.2.4 is available.
You should consider upgrading via the '/usr/bin/python3.6 -m pip install --upgrade pip' command.

[chandchowdary]

I am not using docker, I was trying to do normal anyjson wheel file installation into my wheelhouse directory and getting the error while run "python3 setup.py sdist bdist_wheel" after anyjson.tar.gz extraction

[jaraco]

The main reason I used docker was because I needed something with Python 3.6 to demonstrate the usage.

I would recommend getting away from python setup.py and instead use pip wheel to produce the wheel. If you use --no-build-isolation, you'll retain control over which versions of setuptools are used, and by pinning to Setuptools<58 should allow you to continue to make the wheels.