Skip to content

Conversation

@pmiettinen
Copy link
Contributor

@pmiettinen pmiettinen commented May 14, 2019

@pmiettinen pmiettinen changed the title [WIP] Integrate CFFI into setuptools [ESD-1156] [WIP] Integrate CFFI and numba into setuptools [ESD-1156] May 14, 2019
@pmiettinen pmiettinen force-pushed the pmiettinen/esd-1156-numba-deployment branch 4 times, most recently from 9edaa98 to 0e3b31d Compare May 15, 2019 09:36
@pmiettinen
Copy link
Contributor Author

CFFI initial concept OK.

Numba CC compiler getting stuck in pytest context. Investigating.

@pmiettinen
Copy link
Contributor Author

pasi@pasi-linux:~/swiftnav/libsbp/python$ py.test -v tests/ -s
======================================================================================================== test session starts ========================================================================================================
platform linux2 -- Python 2.7.12, pytest-4.5.0, py-1.7.0, pluggy-0.9.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /mnt/users/pasi/swiftnav/libsbp/python, inifile: pytest.ini
plugins: xdist-1.26.1, forked-1.0.2, cov-2.5.1
collecting ... cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
generating LLVM code for 'parse_jit' into /tmp/pycc-build-parse_jit-_CyTzb/parse_jit.o
C compiler: x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC

creating /tmp/pycc-build-parse_jit-_CyTzb/mnt
creating /tmp/pycc-build-parse_jit-_CyTzb/mnt/users
creating /tmp/pycc-build-parse_jit-_CyTzb/mnt/users/pasi
creating /tmp/pycc-build-parse_jit-_CyTzb/mnt/users/pasi/.local
creating /tmp/pycc-build-parse_jit-_CyTzb/mnt/users/pasi/.local/lib
creating /tmp/pycc-build-parse_jit-_CyTzb/mnt/users/pasi/.local/lib/python2.7
creating /tmp/pycc-build-parse_jit-_CyTzb/mnt/users/pasi/.local/lib/python2.7/site-packages
creating /tmp/pycc-build-parse_jit-_CyTzb/mnt/users/pasi/.local/lib/python2.7/site-packages/numba
creating /tmp/pycc-build-parse_jit-_CyTzb/mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc
creating /tmp/pycc-build-parse_jit-_CyTzb/mnt/users/pasi/.local/lib/python2.7/site-packages/numba/runtime
compile options: '-DPYCC_MODULE_NAME=parse_jit -DPYCC_USE_NRT=1 -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -c'
x86_64-linux-gnu-gcc: /mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc/../runtime/nrt.c
x86_64-linux-gnu-gcc: /mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc/modulemixin.c
**HANG**

@pmiettinen
Copy link
Contributor Author

pmiettinen commented May 15, 2019

Not reproducible in standalone compilation, only one unrelated warning:

pasi@pasi-linux:~/tmp$ x86_64-linux-gnu-gcc -DPYCC_MODULE_NAME=parse_jit -DPYCC_USE_NRT=1 -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -c /mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc/modulemixin.c
In file included from /mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc/modulemixin.c:16:0:
/mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc/../_helperlib.c: In function ‘import_cython_function’:
/mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc/../_helperlib.c:578:42: warning: passing argument 2 of ‘PyMapping_GetItemString’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
     cobj = PyMapping_GetItemString(capi, function_name);
                                          ^
In file included from /usr/include/python2.7/Python.h:133:0,
                 from /mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc/../_pymodule.h:6,
                 from /mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc/modulemixin.c:8:
/usr/include/python2.7/abstract.h:1356:29: note: expected ‘char *’ but argument is of type ‘const char *’
      PyAPI_FUNC(PyObject *) PyMapping_GetItemString(PyObject *o, char *key);

@pmiettinen
Copy link
Contributor Author

Seems to be somehow related to Python version.

python -c "from sbp.jit import msg" reproduces the issue with 2.7 which uses x86_64-linux-gnu-gcc for compiling.

On 3.5.0 build does not hang and it uses gcc.

@pmiettinen
Copy link
Contributor Author

numba.pycc issue with py27 replicates with the example code from https://numba.pydata.org/numba-doc/dev/user/pycc.html

@pmiettinen
Copy link
Contributor Author

Replicates on OSX with clang compiler so it's not about the compiler.

@pmiettinen
Copy link
Contributor Author

pmiettinen commented May 16, 2019

To record the current status:

pycc.py standalone example

from numba.pycc import CC

cc = CC('my_module')
# Uncomment the following line to print out the compilation steps
cc.verbose = True

@cc.export('multf', 'f8(f8, f8)')
@cc.export('multi', 'i4(i4, i4)')
def mult(a, b):
    return a * b

@cc.export('square', 'f8(f8)')
def square(a):
    return a ** 2

cc.compile()

py35 success

pasi@pasi-linux:~/tmp$ python -c "import pycc"
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
generating LLVM code for 'my_module' into /tmp/pycc-build-my_module-nu_7foug/my_module.cpython-35m-x86_64-linux-gnu.o
C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

creating /tmp/pycc-build-my_module-nu_7foug/mnt
creating /tmp/pycc-build-my_module-nu_7foug/mnt/users
creating /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi
creating /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi/.local
creating /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi/.local/lib
creating /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi/.local/lib/python3.5
creating /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi/.local/lib/python3.5/site-packages
creating /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi/.local/lib/python3.5/site-packages/numba
creating /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi/.local/lib/python3.5/site-packages/numba/pycc
creating /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi/.local/lib/python3.5/site-packages/numba/runtime
compile options: '-DPYCC_MODULE_NAME=my_module -DPYCC_USE_NRT=1 -I/mnt/users/pasi/.pyenv/versions/3.5.0/include/python3.5m -I/mnt/users/pasi/.local/lib/python3.5/site-packages/numpy/core/include -c'
gcc: /mnt/users/pasi/.local/lib/python3.5/site-packages/numba/pycc/../_math_c99.c
gcc: /mnt/users/pasi/.local/lib/python3.5/site-packages/numba/pycc/modulemixin.c
gcc: /mnt/users/pasi/.local/lib/python3.5/site-packages/numba/pycc/../runtime/nrt.c
gcc -pthread -shared -L/mnt/users/pasi/.pyenv/versions/3.5.0/lib /tmp/pycc-build-my_module-nu_7foug/my_module.cpython-35m-x86_64-linux-gnu.o /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi/.local/lib/python3.5/site-packages/numba/pycc/modulemixin.o /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi/.local/lib/python3.5/site-packages/numba/_math_c99.o /tmp/pycc-build-my_module-nu_7foug/mnt/users/pasi/.local/lib/python3.5/site-packages/numba/runtime/nrt.o -L/mnt/users/pasi/.local/lib/python3.5/site-packages/numpy/core/lib -lnpymath -lm -o /mnt/users/pasi/tmp/my_module.cpython-35m-x86_64-linux-gnu.so

py27 hangs

pasi@pasi-linux:~/tmp$ python -c "import pycc"
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
generating LLVM code for 'my_module' into /tmp/pycc-build-my_module-Qq8jC8/my_module.o
C compiler: x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC

creating /tmp/pycc-build-my_module-Qq8jC8/mnt
creating /tmp/pycc-build-my_module-Qq8jC8/mnt/users
creating /tmp/pycc-build-my_module-Qq8jC8/mnt/users/pasi
creating /tmp/pycc-build-my_module-Qq8jC8/mnt/users/pasi/.local
creating /tmp/pycc-build-my_module-Qq8jC8/mnt/users/pasi/.local/lib
creating /tmp/pycc-build-my_module-Qq8jC8/mnt/users/pasi/.local/lib/python2.7
creating /tmp/pycc-build-my_module-Qq8jC8/mnt/users/pasi/.local/lib/python2.7/site-packages
creating /tmp/pycc-build-my_module-Qq8jC8/mnt/users/pasi/.local/lib/python2.7/site-packages/numba
creating /tmp/pycc-build-my_module-Qq8jC8/mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc
creating /tmp/pycc-build-my_module-Qq8jC8/mnt/users/pasi/.local/lib/python2.7/site-packages/numba/runtime
compile options: '-DPYCC_MODULE_NAME=my_module -DPYCC_USE_NRT=1 -I/usr/include/python2.7 -I/mnt/users/pasi/.local/lib/python2.7/site-packages/numpy/core/include -c'
x86_64-linux-gnu-gcc: /mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc/../_math_c99.c
x86_64-linux-gnu-gcc: /mnt/users/pasi/.local/lib/python2.7/site-packages/numba/pycc/../runtime/nrt.c

After narrowing down the scope with added debug prints in numba.pycc files, hanging seems to happen inside distutils.ccompiler.

@pmiettinen
Copy link
Contributor Author

pmiettinen commented May 16, 2019

Appears that build uses numpy.distutils.ccompiler and the part of the code that hangs:

    if len(build) > 1 and jobs > 1:
        # build parallel
        import multiprocessing.pool
        pool = multiprocessing.pool.ThreadPool(jobs)
        pool.map(single_compile, build_items)
        pool.close()

Code is identical in both py27 and py35 numpy installations.

After forcing serial build, build doesn't hang.

@pmiettinen
Copy link
Contributor Author

Narrowed down to subprocess.check_output(cmd) hanging in numpy.distutils.ccompiler when doing parallel build.

@pmiettinen
Copy link
Contributor Author

pmiettinen commented May 16, 2019

Monkeypatching to serial build only helps. Needs to be investigated how rugged solution this is.

from numba.pycc import CC

from distutils.ccompiler import CCompiler
from numpy.distutils.misc_util import get_num_build_jobs
from numpy.distutils.ccompiler import _global_lock, _job_semaphore, _processing_files
from numpy.distutils import log

import sys
import types

def replace_method(klass, method_name, func):
    if sys.version_info[0] < 3:
        m = types.MethodType(func, None, klass)
        setattr(klass, method_name, m)

def CCompiler_compile(self, sources, output_dir=None, macros=None,
                      include_dirs=None, debug=0, extra_preargs=None,
                      extra_postargs=None, depends=None):
    """
    Compile one or more source files.

    Please refer to the Python distutils API reference for more details.

    Parameters
    ----------
    sources : list of str
        A list of filenames
    output_dir : str, optional
        Path to the output directory.
    macros : list of tuples
        A list of macro definitions.
    include_dirs : list of str, optional
        The directories to add to the default include file search path for
        this compilation only.
    debug : bool, optional
        Whether or not to output debug symbols in or alongside the object
        file(s).
    extra_preargs, extra_postargs : ?
        Extra pre- and post-arguments.
    depends : list of str, optional
        A list of file names that all targets depend on.

    Returns
    -------
    objects : list of str
        A list of object file names, one per source file `sources`.

    Raises
    ------
    CompileError
        If compilation fails.

    """
    # This method is effective only with Python >=2.3 distutils.
    # Any changes here should be applied also to fcompiler.compile
    # method to support pre Python 2.3 distutils.
    global _job_semaphore

    jobs = get_num_build_jobs()

    # setup semaphore to not exceed number of compile jobs when parallelized at
    # extension level (python >= 3.5)
    with _global_lock:
        if _job_semaphore is None:
            _job_semaphore = threading.Semaphore(jobs)

    if not sources:
        return []

    ccomp = self.compiler_so
    display = "C compiler: %s\n" % (' '.join(ccomp),)
    log.info(display)
    macros, objects, extra_postargs, pp_opts, build = \
            self._setup_compile(output_dir, macros, include_dirs, sources,
                                depends, extra_postargs)
    cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
    display = "compile options: '%s'" % (' '.join(cc_args))
    if extra_postargs:
        display += "\nextra options: '%s'" % (' '.join(extra_postargs))
    log.info(display)

    def single_compile(args):
        obj, (src, ext) = args

        # check if we are currently already processing the same object
        # happens when using the same source in multiple extensions
        while True:
            # need explicit lock as there is no atomic check and add with GIL
            with _global_lock:
                # file not being worked on, start working
                if obj not in _processing_files:
                    _processing_files.add(obj)
                    break
            # wait for the processing to end
            time.sleep(0.1)

        try:
            # retrieve slot from our #job semaphore and build
            with _job_semaphore:
                self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
        finally:
            # register being done processing
            with _global_lock:
                _processing_files.remove(obj)

    for o in build.items():
        single_compile(o)

    # Return *all* object filenames, not just the ones we just built.
    return objects

replace_method(CCompiler, 'compile', CCompiler_compile)


cc = CC('my_module')
# Uncomment the following line to print out the compilation steps
cc.verbose = True

@cc.export('multf', 'f8(f8, f8)')
@cc.export('multi', 'i4(i4, i4)')
def mult(a, b):
    return a * b

@cc.export('square', 'f8(f8)')
def square(a):
    return a ** 2

cc.compile()

@pmiettinen pmiettinen force-pushed the pmiettinen/esd-1156-numba-deployment branch from 6738e80 to eeba8e3 Compare May 16, 2019 12:15
@pmiettinen
Copy link
Contributor Author

Next problem. tox/py35 environment doesn't recognize for which Python version the parse_jit module is built and tries to use parse_jit.so built for py27 when it should trigger a build for parse_jit.cpython-35m-x86_64-linux-gnu.so and use it as is the case on native py35 environment.

@pmiettinen
Copy link
Contributor Author

Actual integration to distutils in case of numba needs to be investigated. Main problem being that libsbp uses actually setuptools and not distutils so the numba example is not applicable as-is. Might need some monkeypatching also there.

@pmiettinen pmiettinen force-pushed the pmiettinen/esd-1156-numba-deployment branch from eeba8e3 to 4fe36a1 Compare May 17, 2019 06:20
@pmiettinen
Copy link
Contributor Author

Next some problem with the benchmark script:
ERROR: InvocationError for command /home/travis/build/swift-nav/libsbp/python/../test_data/benchmark.sh /home/travis/build/swift-nav/libsbp/haskell (exited with code 1)

@pmiettinen pmiettinen force-pushed the pmiettinen/esd-1156-numba-deployment branch 2 times, most recently from a8db61d to e15ec3f Compare May 17, 2019 13:23
@benjaminaltieri benjaminaltieri requested a review from silverjam May 17, 2019 18:01
@pmiettinen pmiettinen force-pushed the pmiettinen/esd-1156-numba-deployment branch from e15ec3f to e859e93 Compare May 22, 2019 08:42
@pmiettinen pmiettinen changed the title [WIP] Integrate CFFI and numba into setuptools [ESD-1156] Integrate numba into setuptools [ESD-1156] May 22, 2019
Copy link
Contributor

@silverjam silverjam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Just a few nit and questions.

INSTALL_REQUIRES += [i.strip() for i in f.readlines()]

with open(os.path.join(filedir, 'setup_requirements.txt')) as f:
INSTALL_REQUIRES += [i.strip() for i in f.readlines()]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want setup_requirements.txt in INSTALL_REQUIRES? Does this make numba and friends a library dependency? Is there an equivalent "build requirements"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to revisit this. I wasn't feeling I was quite done with the thought process with this. It's complicated to make a clean solution that caters for all the use cases.

Copy link
Contributor Author

@pmiettinen pmiettinen May 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the main points are close to:

  • with sdist package these requirements are needed as you are building sbp and the parse_jit module from the sources
  • with bdist_wheel package these requirements are not needed but the wheel needs to be built and deployed for each Python version separately

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sdist is not uploaded to PyPi so I'll just add the version specific wheels and remove the setup_requirements from the package.

# device.
SENDER_ID = 0x42

_crc16_tab = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_crc16_tab = [
crc16_tab = [


from sbp.constants import SENDER_ID as _SENDER_ID
from sbp.constants import SBP_PREAMBLE as _SBP_PREAMBLE
from sbp.constants import _crc16_tab
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from sbp.constants import _crc16_tab
crc16_tab = [

cc = CC(module_name)
cc.verbose = True

crc16_tab = np.array(_crc16_tab, dtype=np.uint16)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
crc16_tab = np.array(_crc16_tab, dtype=np.uint16)
np_crc16_tab = np.array(_crc16_tab, dtype=np.uint16)

"""CRC16 implementation acording to CCITT standards."""
for index in range(offset, offset + length):
data = buf[index]
lookup = crc16_tab[((nb.u2(crc) >> 8) & nb.u2(0xFF)) ^ (data & nb.u2(0xFF))]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
lookup = crc16_tab[((nb.u2(crc) >> 8) & nb.u2(0xFF)) ^ (data & nb.u2(0xFF))]
lookup = np_crc16_tab[((nb.u2(crc) >> 8) & nb.u2(0xFF)) ^ (data & nb.u2(0xFF))]


from sbp.constants import SENDER_ID as _SENDER_ID
from sbp.constants import SBP_PREAMBLE as _SBP_PREAMBLE
from sbp.constants import _crc16_tab
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from sbp.constants import _crc16_tab
from sbp.constants import crc16_tab

parse_jit = importlib.import_module('sbp.jit.' + parse_jit_name)


crc16_tab = np.array(_crc16_tab, dtype=np.uint16)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
crc16_tab = np.array(_crc16_tab, dtype=np.uint16)
np_crc16_tab = np.array(crc16_tab, dtype=np.uint16)

"""CRC16 implementation acording to CCITT standards."""
for ch in bytearray(s): # bytearray's elements are integers in both python 2 and 3
crc = ((crc << 8) & 0xFFFF) ^ _crc16_tab[((crc >> 8) & 0xFF) ^ (ch & 0xFF)]
crc = ((crc << 8) & 0xFFFF) ^ crc16_tab[((crc >> 8) & 0xFF) ^ (ch & 0xFF)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
crc = ((crc << 8) & 0xFFFF) ^ crc16_tab[((crc >> 8) & 0xFF) ^ (ch & 0xFF)]
crc = ((crc << 8) & 0xFFFF) ^ np_crc16_tab[((crc >> 8) & 0xFF) ^ (ch & 0xFF)]

include_dirs=None, debug=0, extra_preargs=None,
extra_postargs=None, depends=None):
"""
Compile one or more source files.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can delete their docs, and maybe just insert a link to the original source?

@pmiettinen pmiettinen force-pushed the pmiettinen/esd-1156-numba-deployment branch from c7628b9 to 1e09b98 Compare May 27, 2019 08:30
@pmiettinen
Copy link
Contributor Author

@silverjam to review the latest changes and decide on the merging.

@silverjam
Copy link
Contributor

Changes look good but I can't figure out how to build on Windows, will try to spend a few minutes working this out, otherwise we'll need to revisit later.

- sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce
- sudo pip install tox
- git clone -b master https://github.com/swift-nav/piksi_tools.git ../piksi_tools
- git clone -b pmiettinen/esd-1156-numba-deployment https://github.com/swift-nav/piksi_tools.git ../piksi_tools
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix after merge

@silverjam silverjam merged commit 0ef6a03 into master Jun 5, 2019
@silverjam silverjam deleted the pmiettinen/esd-1156-numba-deployment branch June 5, 2019 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants