AOT always installed to site-package folder not the module folder

[wzhtc]

I created a AOT module with the following folder structure:

-test_numba
  -test_numba
    -__init__.py
    -source_module.py
-setup.py

The source_module.py and setup.py has same functions as in the introduction
in the 'init.py' file, I simple import the functions from module.

from .my_module import *

When I run pip install, the my_module.cpython-37m-darwin.so is always installed to the site-package folder, instead of the site-packages/test_numba/ folder.

Here is the message on the install:

Created temporary directory: /private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-ephem-wheel-cache-hmnzdc3q
Created temporary directory: /private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-req-tracker-dtmbq7jy
Created requirements tracker '/private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-req-tracker-dtmbq7jy'
Created temporary directory: /private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-install-p3nwsdna
Processing /Users/xxx/test_numba
  Created temporary directory: /private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-req-build-4aq7o_p1
  Added file:///Users/xxx/test_numba to build tracker '/private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-req-tracker-dtmbq7jy'
    Running setup.py (path:/private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-req-build-4aq7o_p1/setup.py) egg_info for package from file:///Users/xxx/test_numba
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/test_numba.egg-info
    writing pip-egg-info/test_numba.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/test_numba.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/test_numba.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/test_numba.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/test_numba.egg-info/SOURCES.txt'
    writing manifest file 'pip-egg-info/test_numba.egg-info/SOURCES.txt'
  Source in /private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-req-build-4aq7o_p1 has version 0.0.1, which satisfies requirement test-numba==0.0.1 from file:///Users/xxx/test_numba
  Removed test-numba==0.0.1 from file:///Users/xxx/test_numba from build tracker '/private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-req-tracker-dtmbq7jy'
Building wheels for collected packages: test-numba
  Created temporary directory: /private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-wheel-w7xwl19y
  Building wheel for test-numba (setup.py) ...   Destination directory: /private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-wheel-w7xwl19y
  Running command /Users/xxx/miniconda3/envs/py/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-req-build-4aq7o_p1/setup.py'"'"'; __file__='"'"'/private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-req-build-4aq7o_p1/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-wheel-w7xwl19y --python-tag cp37
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.9-x86_64-3.7
  creating build/lib.macosx-10.9-x86_64-3.7/test_numba
  copying test_numba/source_module.py -> build/lib.macosx-10.9-x86_64-3.7/test_numba
  copying test_numba/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/test_numba
  running build_ext
  creating build/temp.macosx-10.9-x86_64-3.7
  generating LLVM code for 'my_module' into build/temp.macosx-10.9-x86_64-3.7/my_module.cpython-37m-darwin.o
  building 'my_module' extension
  C compiler: gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/xxx/miniconda3/envs/py/include -arch x86_64 -I/Users/xxx/miniconda3/envs/py/include -arch x86_64

  creating build/temp.macosx-10.9-x86_64-3.7/Users
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/pycc
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/cext
  creating build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/runtime
  compile options: '-DPYCC_MODULE_NAME=my_module -DPYCC_USE_NRT=1 -I/Users/xxx/miniconda3/envs/py/include/python3.7m -I/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numpy/core/include -I/Users/xxx/miniconda3/envs/py/include/python3.7m -c'
  gcc: /Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/pycc/modulemixin.cgcc: /Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/pycc/../_math_c99.c
  gcc: /Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/cext/listobject.c
  gcc: /Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/cext/utils.c
  gcc: /Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/cext/dictobject.c
  gcc: /Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/pycc/../runtime/nrt.c

  gcc -bundle -undefined dynamic_lookup -L/Users/xxx/miniconda3/envs/py/lib -arch x86_64 -L/Users/xxx/miniconda3/envs/py/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/pycc/modulemixin.o build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/_math_c99.o build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/cext/listobject.o build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/cext/utils.o build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/cext/dictobject.o build/temp.macosx-10.9-x86_64-3.7/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numba/runtime/nrt.o build/temp.macosx-10.9-x86_64-3.7/my_module.cpython-37m-darwin.o -L/Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/numpy/core/lib -lnpymath -o build/lib.macosx-10.9-x86_64-3.7/my_module.cpython-37m-darwin.so
  installing to build/bdist.macosx-10.9-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.macosx-10.9-x86_64
  creating build/bdist.macosx-10.9-x86_64/wheel
  copying build/lib.macosx-10.9-x86_64-3.7/my_module.cpython-37m-darwin.so -> build/bdist.macosx-10.9-x86_64/wheel
  creating build/bdist.macosx-10.9-x86_64/wheel/test_numba
  copying build/lib.macosx-10.9-x86_64-3.7/test_numba/source_module.py -> build/bdist.macosx-10.9-x86_64/wheel/test_numba
  copying build/lib.macosx-10.9-x86_64-3.7/test_numba/__init__.py -> build/bdist.macosx-10.9-x86_64/wheel/test_numba
  running install_egg_info
  running egg_info
  creating test_numba.egg-info
  writing test_numba.egg-info/PKG-INFO
  writing dependency_links to test_numba.egg-info/dependency_links.txt
  writing top-level names to test_numba.egg-info/top_level.txt
  writing manifest file 'test_numba.egg-info/SOURCES.txt'
  reading manifest file 'test_numba.egg-info/SOURCES.txt'
  writing manifest file 'test_numba.egg-info/SOURCES.txt'
  Copying test_numba.egg-info to build/bdist.macosx-10.9-x86_64/wheel/test_numba-0.0.1-py3.7.egg-info
  running install_scripts
  creating build/bdist.macosx-10.9-x86_64/wheel/test_numba-0.0.1.dist-info/WHEEL
  creating '/private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-wheel-w7xwl19y/test_numba-0.0.1-cp37-cp37m-macosx_10_9_x86_64.whl' and adding 'build/bdist.macosx-10.9-x86_64/wheel' to it
  adding 'my_module.cpython-37m-darwin.so'
  adding 'test_numba/__init__.py'
  adding 'test_numba/source_module.py'
  adding 'test_numba-0.0.1.dist-info/METADATA'
  adding 'test_numba-0.0.1.dist-info/WHEEL'
  adding 'test_numba-0.0.1.dist-info/top_level.txt'
  adding 'test_numba-0.0.1.dist-info/RECORD'
  removing build/bdist.macosx-10.9-x86_64/wheel
done
  Created wheel for test-numba: filename=test_numba-0.0.1-cp37-cp37m-macosx_10_9_x86_64.whl size=68204 sha256=43dc63501bbcd6c2e74561d80d455d1276d8f48a07b9b2f6c47154166cedc65b
  Stored in directory: /private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-ephem-wheel-cache-hmnzdc3q/wheels/11/07/1a/d515f6ebdf72b2a43b314b68c788d93fc6c0cf2168c8d2538a
  Removing source in /private/var/folders/2t/kh60843d47x17gkkqg0q3gc80000gn/T/pip-req-build-4aq7o_p1
Successfully built test-numba
Installing collected packages: test-numba

And it shows the packages are installed in the following location:

Uninstalling test-numba-0.0.1:
  Would remove:
    /Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/my_module.cpython-37m-darwin.so
    /Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/test_numba-0.0.1.dist-info/*
    /Users/xxx/miniconda3/envs/py/lib/python3.7/site-packages/test_numba/*

Then if I try to import test_numba, I will get an error saying test_numba.my_module doesn't exist.

Reporting a bug

• [ x ] I am using the latest released version of Numba (most recent is visible in
  the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
• [ x ] I have included below a minimal working reproducer (if you are unsure how
  to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

[esc]

@wzhtc thanks for reporting this and helping to improve Numba. Would it be possible to attach the source code for your project so that I can attempt to reproduce the issue you are having on my computer? For example your could try to 'zip' the directory and attach it to this issue.

[wzhtc]

@esc thanks for taking a look at this. Attached is the zip files.
test_numba.zip

[esc]

@wzhtc thank you for that, please could you also post the commands that you are using the build the extension? Thanks.

[wzhtc]

@esc I used “pip install . “ At the parent folder to install. The pip version I have is 19

[esc]

Interesting, when I try to run that, I get:

    ERROR: Command errored out with exit status 1:
     command: /Users/vhaenel/miniconda3/envs/new-numba/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/jk/s59wn3zx5g92kvk6t9k74fzr0000gn/T/pip-req-build-l8qhqd6c/setup.py'"'"'; __file__='"'"'/private/var/folders/jk/s59wn3zx5g92kvk6t9k74fzr0000gn/T/pip-req-build-l8qhqd6c/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /private/var/folders/jk/s59wn3zx5g92kvk6t9k74fzr0000gn/T/pip-req-build-l8qhqd6c/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/jk/s59wn3zx5g92kvk6t9k74fzr0000gn/T/pip-req-build-l8qhqd6c/setup.py", line 2, in <module>
        from test_numba.source_module import cc
      File "/private/var/folders/jk/s59wn3zx5g92kvk6t9k74fzr0000gn/T/pip-req-build-l8qhqd6c/test_numba/__init__.py", line 1, in <module>
        from .my_module import *
    ModuleNotFoundError: No module named 'test_numba.my_module'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[esc]

Commenting out the following however, allows me to compile:

diff --git i/test_numba/__init__.py w/test_numba/__init__.py
index c51799952d..bf9098714e 100644
--- i/test_numba/__init__.py
+++ w/test_numba/__init__.py
@@ -1,2 +1,2 @@
-from .my_module import *
+#from .my_module import *

[esc]

And in that case, I can observe the same behaviour you listed, so it is reproducible. I wonder if there is something you need to set to place the my_module in the test_numba package..

[wzhtc]

@esc I tried to set the output_dir of the cc extension. It is not working. Do you have any suggested method to install as desired structure?

[esc]

@wzhtc no I don't have any suggestions at this time, I have been digging but to no avail yet. My thinking however was also to use the kwargs of distutils_extension.

[wzhtc]

@esc I also looked at the disturbing.extendion.Extension class which is the return of the cc extension. unfortunately, it seems not accepting file location parameter.

[esc]

Yeah, this is probably a genuine bug: I patched Numba like so:

diff --git i/numba/pycc/cc.py w/numba/pycc/cc.py
index ee635eb5fc..80623e762d 100644
--- i/numba/pycc/cc.py
+++ w/numba/pycc/cc.py
@@ -235,6 +235,7 @@ class CC(object):
         Create a distutils extension object that can be used in your
         setup.py.
         """
+        python_package_path = kwargs.pop('python_package_path', "")
         macros = kwargs.pop('macros', []) + self._get_mixin_defines()
         depends = kwargs.pop('depends', []) + [self._source_path]
         extra_compile_args = (kwargs.pop('extra_compile_args', [])
@@ -248,7 +249,7 @@ class CC(object):
         library_dirs = (kwargs.pop('library_dirs', [])
                         + self._toolchain.get_python_library_dirs())

-        ext = _CCExtension(name=self._basename,
+        ext = _CCExtension(name=python_package_path + self._basename,
                            sources=self._get_mixin_sources(),
                            depends=depends,
                            define_macros=macros,

I.e. allowing to prepend the package name to the name of Extension. I could then do:

        ext_modules=[cc.distutils_extension(python_package_path="test_numba.")]

And the compiled Python module will be placed in the correct package. Probably, some code needs to get written for the CC class to get (maybe exists already) and use that name.

[esc]

Actually, the information is somewhat already gathered, using this different patch should also fix it:

diff --git i/numba/pycc/cc.py w/numba/pycc/cc.py
index ee635eb5fc..985e613a35 100644
--- i/numba/pycc/cc.py
+++ w/numba/pycc/cc.py
@@ -64,6 +64,7 @@ class CC(object):

         self._source_path = dct.get('__file__', '')
         self._source_module = source_module
+        self._package_path = source_module[:source_module.rfind(".") + 1]
         self._toolchain = Toolchain()
         self._verbose = False
         # By default, output in directory of caller module
@@ -248,7 +249,7 @@ class CC(object):
         library_dirs = (kwargs.pop('library_dirs', [])
                         + self._toolchain.get_python_library_dirs())

-        ext = _CCExtension(name=self._basename,
+        ext = _CCExtension(name=".".join((self._package_path, self._basename)),
                            sources=self._get_mixin_sources(),
                            depends=depends,
                            define_macros=macros,

@wzhtc can you confirm? In case it is good, @stuartarchibald should I make a PR then?

[esc]

The patch should fix the overall issue of using CC from within a (nested) module in a package.

[stuartarchibald]

@wzhtc can you confirm? In case it is good, @stuartarchibald should I make a PR then?

@esc yes please, with test to demonstrate if possible. Thanks.

[wzhtc]

@stuartarchibald The patch works. Thanks!

[tolysz]

Any chance to make this PR?

[esc]

@tolysz yes! feel free to submit one! It needs some sort of test-case though.

[tolysz]

As per commit description, the actual changes are slightly different. But the good news it works!

[esc]

🎉

[tolysz]

:) I tried to be smart, but at the end it is almost what you have here, the only change is that I had to mkdir some extra directories.