Unable to use numpy

[rjzak]

OS: Ubuntu 19.04 64-bit
Python: Python 3.7.3
Rust: 1.35.0

I'm unable to use numpy in my test project.

1. I tried to use numpy in my project, and I got an error saying there wasn't a module called numpy. I created a virtual environment with numpy, scipy, and matplotlib. I installed these modules with pip, and was able to confirm that they work with a simple test script which plots random numbers.
2. I ran into the __file__ issue in numpy.__config__, where it tries to look for an extra .libs directory. I replaced that line with a hardcoded string value.
3. I tried with and without sys_paths = ["$ORIGIN/lib"] in embedded_python_config, it didn't seem to make a difference.
4. Ultimately, I get this error: No module named 'numpy.core._multiarray_umath'.

I made sure that the gfortran, libopenblas, and other dependencies were installed, in case that was part of the issue, but it didn't help. Ultimately, I'd like to use this project to make a more portable version of a larger project which has a few C-based module dependencies.

[keir]

Consider posting your pyoxidizer.toml to help the maintainers help you.

[indygreg]

Packaging C extensions from pre-built virtualenvs won't work (this needs to be documented better). C extensions will only be packaged for packaging rules that invoke pip or setup.py.

But, uh, when I try this myself, the numpy package takes forever to build and I see it compiling object files, but PyOxidizer doesn't pick up the built C extensions.

I'll need to look into this in more detail.

Thanks for the bug report!

[indygreg]

TIL NumPy vendors its own complete copy of distutils, which it has modified for its own personal gain and uses exclusively as part of builds. So by providing its own version of distutils, Numpy bypasses the modifications PyOxidizer makes to distutils which enable PyOxidizer to intercept C extension building.

This is... extremely annoying. But I don't think it is the end of the world. There are likely ways we can still use NumPy's version of distutils and still capture enough data to facilitate building C extensions. It's a bit more work for PyOxidizer but I think it is still doable.

I wonder how many other Python packages go to the lengths that NumPy is. I'm willing to wager it is less than 10. I'm half tempted to submit patches against NumPy to do something infinitely more reasonable than vendoring a complete copy of distutils. That approach is kinda ridiculous IMO.

[rjzak]

I tried to use numpy without the virualenv, and I was getting an error saying that the module wasn't found, even though I could run python and import it. The virtualenv was the only way I found to make PyOxidizer see numpy. As for my toml file:

[[build]]
application_name = "pyapp_oxidizer"
[[embedded_python_config]]
raw_allocator = "jemalloc"
sys_paths = ["$ORIGIN/lib"] # Doesn't seem to make a difference
[[embedded_python_config]]
build_target = "x86_64-pc-windows-msvc"
raw_allocator = "system"
[[packaging_rule]]
type = "stdlib-extensions-policy"
policy = "all"
[[packaging_rule]]
type = "stdlib"
include_source = false
[[packaging_rule]]
type = "write-license-files"
path = "
[[packaging_rule]]
type = "virtualenv"
path = "/home/user/pyenv"
[[packaging_rule]]
type = "package-root"
path = "/home/user/Documents/pyapp_oxidizer/pysrc"
packages = ["testapp"]
[[embedded_python_run]]
mode = "eval"
code = "from testapp import dostuff; dostuff()"
[[python_distribution]]
# The normal stuff, didn't change anything...

Python code:

#!/usr/bin/python3
import numpy as np
from scipy.fftpack import fft
import matplotlib.pylab as plt
def dostuff(steps=100):
    np.random.seed()
    a = np.random.rand(steps)
    x = np.linspace(-np.pi, np.pi, steps)
    f = fft(a)
    print(len(f))
    plt.plot(x, np.sin(x))
    plt.plot(f)
    plt.savefig("out.png")

[nicola-zanardi]

@rjzak,

did you have any luck in actually running the script with numpy? I was also able to build with pyoxidizer build and a very similar toml file, but then numpy fails due to a file dependency

>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "numpy", line 126, in <module>
  File "numpy.__config__", line 9, in <module>
NameError: name '__file__' is not defined

As stated in #69 this should be reported to numpy, right?

[rjzak]

I was able to get numpy to work with Nuitka, but not PyOxidizer. I haven't tried since I opened the bug report, and it looks like there's been nine commits since. I should try again.

[danfulton]

I'm having issues installing pandas which I think is ultimately this problem traced back to numpy. Was there any work around to include numpy?

[leopd]

FWIW, I appear to be able to build numpy with pyox, but am running into the issue that it requires __file__ (issue #69). I'm building it on Mac, py3.7, pyox 0.6.0.

>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "numpy", line 126, in <module>
  File "numpy.__config__", line 9, in <module>
NameError: name '__file__' is not defined

[indygreg]

PyOxidizer 0.9 has a new files mode which can be leveraged to enable PyOxidizer to work with complex packages like NumPy. See https://pyoxidizer.readthedocs.io/en/v0.9.0/packaging_additional_files.html#installing-unclassified-files-on-the-filesystem for an example of how to get NumPy working with PyOxidizer.

The feature is still you and it doesn't provide the full PyOxidizer benefits. But it does enable NumPy to work with PyOxidizer.

[giiyms]

Hello, I am running into the issue with numpy.
I am using the files mode, it builds without errors but when I try to run I get this error:

Original error was: dlopen(/pyapp/build/aarch64-apple-darwin/debug/install/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_PyCapsule_GetContext'

Any ideas?

pyoxidizer.bzl:

def make_exe():
    dist = default_python_distribution(python_version="3.10")

    policy = dist.make_python_packaging_policy()
    policy.set_resource_handling_mode("files")
    policy.resources_location = "filesystem-relative:site-packages"

    python_config = dist.make_python_interpreter_config()
    python_config.module_search_paths = ["$ORIGIN/site-packages"]
    python_config.run_module = "tool"
    
    python_config.filesystem_importer = True
    python_config.oxidized_importer = False
    python_config.sys_frozen = True

    exe = dist.to_python_executable(
        name="tool",
        packaging_policy=policy,
        config=python_config,
    )

    exe.windows_runtime_dlls_mode = "always"
    exe.windows_subsystem = "console"

    exe.add_python_resources(exe.pip_install(["wheel"]))
    exe.add_python_resources(exe.pip_install(["-r", "requirements.txt"]))
    exe.add_python_resources(exe.pip_install(["."]))
    return exe

def make_embedded_resources(exe):
    return exe.to_embedded_resources()

def make_install(exe):
    files = FileManifest()
    files.add_python_resource(".", exe)
    files.install("dist", replace = True)
    return files

register_target("exe", make_exe)
register_target("resources", make_embedded_resources, depends=["exe"], default_build_script=True)
register_target("install", make_install, depends=["exe"], default=True)

resolve_targets()

pyproject.toml:

[tool.poetry]
name = "pyapp"
version = "0.1.0"
description = ""
authors = [""]
readme = "README.md"


packages = [{ include = "tool", from = "lib" }]


[tool.poetry.dependencies]
python = "^3.10"
arviz = "^0.14.0"
matplotlib = "^3.6.2"
pymc = "^5.0.0"
xarray = "^2022.12.0"
numpy = "^1.23.5"
scipy = "^1.9.3"
pyoxidizer = "^0.23.0"
cython = "^0.29.32"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"