Logistic regression example not working

[alanmarazzi]

Hi, I have an issue running the logistic regression example, when running it after installing h5py as recommended on documentation I get this import error:

Traceback (most recent call last):
  File "generate_data/gen_logistic_regression.py", line 7, in <module>
    @hpat.jit
  File "/home/alanm/miniconda3/envs/HPAT/lib/python3.6/site-packages/hpat-0.1.0-py3.6-linux-x86_64.egg/hpat/__init__.py", line 12, in jit
    from .compiler import add_hpat_stages
  File "/home/alanm/miniconda3/envs/HPAT/lib/python3.6/site-packages/hpat-0.1.0-py3.6-linux-x86_64.egg/hpat/compiler.py", line 5, in <module>
    from .hiframes import HiFrames
  File "/home/alanm/miniconda3/envs/HPAT/lib/python3.6/site-packages/hpat-0.1.0-py3.6-linux-x86_64.egg/hpat/hiframes.py", line 23, in <module>
    from hpat import pio
  File "/home/alanm/miniconda3/envs/HPAT/lib/python3.6/site-packages/hpat-0.1.0-py3.6-linux-x86_64.egg/hpat/pio.py", line 13, in <module>
    from hpat import pio_api, pio_lower, utils
  File "/home/alanm/miniconda3/envs/HPAT/lib/python3.6/site-packages/hpat-0.1.0-py3.6-linux-x86_64.egg/hpat/pio_lower.py", line 10, in <module>
    import hio
ModuleNotFoundError: No module named 'hio'

[ehsantn]

HPAT needs to be rebuilt after installing h5py/HDF5:

LDSHARED="mpicxx -shared" CXX=mpicxx LD=mpicxx \
    CC="mpicxx -std=c++11" python setup.py install

[alanmarazzi]

Ok, thanks, at least now I get a building error 😄

mpicxx -shared build/temp.linux-x86_64-3.6/hpat/_io.o -L/home/alanm/miniconda3/envs/HPAT/lib -lpython3.6m -o build/lib.linux-x86_64-3.6/hio.cpython-36m-x86_64-linux-gnu.so -lmpi -lhdf5

/usr/bin/ld: cannot find -lmpi
collect2: error: ld returned 1 exit status
error: command 'mpicxx' failed with exit status 1

Anyway I installed PyArrow as well, shall I get rid of it or keep it since I don't see it mentioned in the documentation?

[ehsantn]

You can remove -lmpi from setup.py and try again. PyArrow should be ok.

[alanmarazzi]

I tried on another machine was able to rebuild HPAT correctly by removing -lmpi (on the other I still couldn't, I will retry from scratch), but when I launch mpirun -n 4 python examples/logistic_regression.py I get

OSError: Failed at nopython (convert DataFrames)
Unable to open file (file signature not found)

And it seems an HDF5 issue because when I check the file with h5debug lr.hdf5 I get

HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 140576896845568:
  #000: H5F.c line 604 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1087 in H5F_open(): unable to read superblock
    major: File accessibilty
    minor: Read failed
  #002: H5Fsuper.c line 277 in H5F_super_read(): file signature not found
    major: File accessibilty
    minor: Not an HDF5 file
cannot open file

By the way, while the linear regression example generates data by launching python generate_data/gen_linear_regression.py, to generate data for logistic regression I had to do it from an interactive python session, and I get the same HDF5 error with the linear regression example.

[ehsantn]

Seems like an issue with your HDF5 setup. Maybe the HDF5 library used in h5py doesn't match the HDF5 library in the system path. There might be file corruption or file system access issues also. Can you read and write HDF5 files using C code?

[alanmarazzi]

That's likely one issue. I'm currently trying installing HPAT without HDF5 support and to avoid problems with previous installs I'm doing it in a Docker container.

I'll let you know how it goes

[ehsantn]

The new installation provides the HDF5 package which avoids these issues. Closing. Feel free to reopen if you still have issues.