Cannot `pip install` with Python 3.8

[reynoldsnlp]

$ python3.8 -m pip install --user hfst
Collecting hfst
  Using cached https://files.pythonhosted.org/packages/0e/33/4a584b5b79723469089be4b6322606e9e923febac54f8a7e3ede0f613a8a/hfst-3.12.2.4b0.tar.gz
Building wheels for collected packages: hfst
  Building wheel for hfst (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/local/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-install-mhd9_x6v/hfst/setup.py'"'"'; __file__='"'"'/private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-install-mhd9_x6v/hfst/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-wheel-fxmcodij --python-tag cp38
       cwd: /private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-install-mhd9_x6v/hfst/
  Complete output (72 lines):
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/dist.py:472: UserWarning: Normalizing '3.12.2.4_beta' to '3.12.2.4b0'
    warnings.warn(
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.14-x86_64-3.8
  copying libhfst.py -> build/lib.macosx-10.14-x86_64-3.8
  creating build/lib.macosx-10.14-x86_64-3.8/hfst
  copying hfst/__init__.py -> build/lib.macosx-10.14-x86_64-3.8/hfst
  creating build/lib.macosx-10.14-x86_64-3.8/hfst/exceptions
  copying hfst/exceptions/__init__.py -> build/lib.macosx-10.14-x86_64-3.8/hfst/exceptions
  creating build/lib.macosx-10.14-x86_64-3.8/hfst/sfst_rules
  copying hfst/sfst_rules/__init__.py -> build/lib.macosx-10.14-x86_64-3.8/hfst/sfst_rules
  creating build/lib.macosx-10.14-x86_64-3.8/hfst/xerox_rules
  copying hfst/xerox_rules/__init__.py -> build/lib.macosx-10.14-x86_64-3.8/hfst/xerox_rules
  running build_ext
  building '_libhfst' extension
  creating build/temp.macosx-10.14-x86_64-3.8
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/implementations
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/implementations/compose_intersect
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/implementations/optimized-lookup
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers/io_src
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers/string_src
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers/variable_src
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers/rule_src
  creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers/alphabet_src
  creating build/temp.macosx-10.14-x86_64-3.8/back-ends
  creating build/temp.macosx-10.14-x86_64-3.8/back-ends/openfst
  creating build/temp.macosx-10.14-x86_64-3.8/back-ends/openfst/src
  creating build/temp.macosx-10.14-x86_64-3.8/back-ends/openfst/src/lib
  creating build/temp.macosx-10.14-x86_64-3.8/back-ends/foma
  /usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHAVE_READLINE -DHAVE_GETLINE -DNO_CPLUSPLUS_11 -DUSE_TR1_UNORDERED_MAP_AND_SET -DINCLUDE_TR1_UNORDERED_MAP_AND_SET -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfst/src/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8 -c libhfst_wrap.cpp -o build/temp.macosx-10.14-x86_64-3.8/libhfst_wrap.o -Wno-sign-compare -Wno-strict-prototypes
  In file included from libhfst_wrap.cpp:171:
  In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/Python.h:85:
  In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/pytime.h:6:
  In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/object.h:746:
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:177:16: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
  typedef struct _typeobject {
                 ^
  libhfst_wrap.cpp:2009:23: note: in implicit copy assignment operator for '_typeobject' first required here
      swigpyobject_type = tmp;
                        ^
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
      Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
      ^
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
  #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                       ^
  In file included from libhfst_wrap.cpp:171:
  In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/Python.h:85:
  In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/pytime.h:6:
  In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/object.h:746:
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:177:16: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
  typedef struct _typeobject {
                 ^
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
      Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
      ^
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
  #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                       ^
  In file included from libhfst_wrap.cpp:3598:
  In file included from libhfst/src/parsers/SfstCompiler.h:17:
  libhfst/src/parsers/SfstAlphabet.h:9:10: fatal error: 'tr1/unordered_map' file not found
  #include <tr1/unordered_map>
           ^~~~~~~~~~~~~~~~~~~
  2 warnings and 1 error generated.
  error: command '/usr/bin/clang' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for hfst
  Running setup.py clean for hfst
Failed to build hfst
Installing collected packages: hfst
    Running setup.py install for hfst ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/local/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-install-mhd9_x6v/hfst/setup.py'"'"'; __file__='"'"'/private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-install-mhd9_x6v/hfst/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-record-o2o2_eff/install-record.txt --single-version-externally-managed --compile --user --prefix=
         cwd: /private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-install-mhd9_x6v/hfst/
    Complete output (72 lines):
    /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/dist.py:472: UserWarning: Normalizing '3.12.2.4_beta' to '3.12.2.4b0'
      warnings.warn(
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14-x86_64-3.8
    copying libhfst.py -> build/lib.macosx-10.14-x86_64-3.8
    creating build/lib.macosx-10.14-x86_64-3.8/hfst
    copying hfst/__init__.py -> build/lib.macosx-10.14-x86_64-3.8/hfst
    creating build/lib.macosx-10.14-x86_64-3.8/hfst/exceptions
    copying hfst/exceptions/__init__.py -> build/lib.macosx-10.14-x86_64-3.8/hfst/exceptions
    creating build/lib.macosx-10.14-x86_64-3.8/hfst/sfst_rules
    copying hfst/sfst_rules/__init__.py -> build/lib.macosx-10.14-x86_64-3.8/hfst/sfst_rules
    creating build/lib.macosx-10.14-x86_64-3.8/hfst/xerox_rules
    copying hfst/xerox_rules/__init__.py -> build/lib.macosx-10.14-x86_64-3.8/hfst/xerox_rules
    running build_ext
    building '_libhfst' extension
    creating build/temp.macosx-10.14-x86_64-3.8
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/implementations
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/implementations/compose_intersect
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/implementations/optimized-lookup
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers/io_src
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers/string_src
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers/variable_src
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers/rule_src
    creating build/temp.macosx-10.14-x86_64-3.8/libhfst/src/parsers/alphabet_src
    creating build/temp.macosx-10.14-x86_64-3.8/back-ends
    creating build/temp.macosx-10.14-x86_64-3.8/back-ends/openfst
    creating build/temp.macosx-10.14-x86_64-3.8/back-ends/openfst/src
    creating build/temp.macosx-10.14-x86_64-3.8/back-ends/openfst/src/lib
    creating build/temp.macosx-10.14-x86_64-3.8/back-ends/foma
    /usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -DHAVE_FOMA -DHAVE_OPENFST -DHAVE_OPENFST_LOG -DHAVE_READLINE -DHAVE_GETLINE -DNO_CPLUSPLUS_11 -DUSE_TR1_UNORDERED_MAP_AND_SET -DINCLUDE_TR1_UNORDERED_MAP_AND_SET -I. -Ilibhfst/src/ -Iback-ends/foma -Iback-ends -Iparsers -Ilibhfst/src/parsers -Iback-ends/openfst/src/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8 -c libhfst_wrap.cpp -o build/temp.macosx-10.14-x86_64-3.8/libhfst_wrap.o -Wno-sign-compare -Wno-strict-prototypes
    In file included from libhfst_wrap.cpp:171:
    In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/Python.h:85:
    In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/pytime.h:6:
    In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/object.h:746:
    /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:177:16: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
    typedef struct _typeobject {
                   ^
    libhfst_wrap.cpp:2009:23: note: in implicit copy assignment operator for '_typeobject' first required here
        swigpyobject_type = tmp;
                          ^
    /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
        Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
        ^
    /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
    #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                         ^
    In file included from libhfst_wrap.cpp:171:
    In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/Python.h:85:
    In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/pytime.h:6:
    In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/object.h:746:
    /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:177:16: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
    typedef struct _typeobject {
                   ^
    /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
        Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
        ^
    /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
    #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                         ^
    In file included from libhfst_wrap.cpp:3598:
    In file included from libhfst/src/parsers/SfstCompiler.h:17:
    libhfst/src/parsers/SfstAlphabet.h:9:10: fatal error: 'tr1/unordered_map' file not found
    #include <tr1/unordered_map>
             ^~~~~~~~~~~~~~~~~~~
    2 warnings and 1 error generated.
    error: command '/usr/bin/clang' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/local/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-install-mhd9_x6v/hfst/setup.py'"'"'; __file__='"'"'/private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-install-mhd9_x6v/hfst/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/m3/nbrhzgz54vx7vfjwzcbsvxyh0000gn/T/pip-record-o2o2_eff/install-record.txt --single-version-externally-managed --compile --user --prefix= Check the logs for full command output.

[tezer]

Hi, are there any plans to release the fix any time soon?

[santhoshtr]

Ubuntu LTS 20.04 comes with Python 3.8 by default. Installation of hfst fails in it. Please release a fix.

[eaxelson]

There isn't an hfst package for python 3.8 because of the unordered_map issue. The issue has been fixed in the development branch hfst-dev for which a python 3.8 package is available (python3.8 -m pip install hfst_dev). I'll have a look at the code and see if these changes could easily be applied also to the hfst branch.

I guess it should also be possible to use package hfst-dev instead of hfst. There are some changes in the API though.

[frankier]

The issue has also been fixed in master -- although in a different way to the dev-4.0 branch it looks like. Such is the nature of having long lived divergent branches.

The problem is that pip command is getting an old version. The reason is that the latest releases don't have either a Python 3.8 wheel or source code available and so an older version is fetched instead. Possibly most people aren't noticing because they are using Debian packages.

See: #462 #463

With some abuse of pip options, it's possible to build from git: python3.8 -m pip install --user --global-option="build_ext" 'git+https://github.com/hfst/hfst.git#egg=hfst&subdirectory=python'

[reynoldsnlp]

@eaxelson and @frankier, thanks for your help! :)

There are some changes in the API though.

Any specific changes to look out for?

[frankier]

From the perspective of what works for a fellow user, I would recommend sticking with HFST 3 over the untested HFST 4, which is also currently missing changes applied to HFST 3. You can still install the most up-to-date HFST 3 on Python 3.8, it's just a bit less convenient e.g. use the pip command I gave.

If you decide to instead use HFST 4 the docs are here https://github.com/hfst/python-hfst-4.0/wiki

[TinoDidriksen]

Ubuntu LTS 20.04 comes with Python 3.8 by default. Installation of hfst fails in it. Please release a fix.

Does the nightly build not work? The whole HFST is built for 20.04 with Python 3.8 for the provided python3-hfst

Same goes for macOS - the binary package is built against Python 3.8.

[frankier]

As far as I can see, the nightly builds do work and there is no issue with unordered_map in recent versions.

This issue is about pip install, which downloads packages from PyPI. This installation option allows for Python libraries to use HFST via a setup.py or anything else which works with those type of dependencies (Conda, Poetry, Pipenv, requirements.txt). The problem is that pip on Python 3.8 is forced to download an old version because latest releases don't have either a Python 3.8 wheel or source code available. The solution is to publish the source code and wheels for Python 3.8 on PyPI.

Using the Debian packages is another workaround, but it is applicable in a different (but overlapping) set of circumstances than pip.

[reynoldsnlp]

@eaxelson @frankier @TinoDidriksen do you know who has the permissions to upload a wheel for 3.8 to PyPI? Since this is building fine, hopefully getting a wheel uploaded should be fairly straightforward.

[santhoshtr]

Sorry for pinging again. since latest linux distributions comes with python 3.8 by default, installing and using hfst is becoming difficult. Either workarounds with pip or downgrading the python to older versions is required. May I request a new release with python 3.8 support? Thanks in advance!.

[reynoldsnlp]

I would be happy to do the work needed to upload wheels to PyPI for python 3.8 and python 3.9 (which was released earlier this month) and future versions of python, but I don't have the credentials to do it. @TinoDidriksen and @eaxelson, do you know who has the credentials to publish on PyPI?