Skip to content
This repository has been archived by the owner on Jul 22, 2024. It is now read-only.

torchmeta support for powerai ppc64le #269

Open
brando90 opened this issue Sep 24, 2020 · 1 comment
Open

torchmeta support for powerai ppc64le #269

brando90 opened this issue Sep 24, 2020 · 1 comment

Comments

@brando90
Copy link

brando90 commented Sep 24, 2020 •
edited
Loading

Hi,

I wanted to use torchmeta in a ppc64le architecture. I was having some issues. The first few I addressed when installing powerai 1.7.0 thanks to the suggestions in this issues #268.

But now when I tried to install it I had an error with h5py:

(my_new_env) [miranda9@hal-login ~]$ pip install torchmeta==1.3.1
Collecting torchmeta==1.3.1
  Using cached torchmeta-1.3.1-py3-none-any.whl (144 kB)
Requirement already satisfied: requests in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (2.22.0)
Requirement already satisfied: torchvision<0.6.0,>=0.4.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (0.4.2)
Requirement already satisfied: torch<1.5.0,>=1.3.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (1.3.1)
Processing ./.cache/pip/wheels/87/f5/ad/9f04a48453875e8054c19f9fe3f50cbbe0c09b956835555019/Pillow-6.2.2-cp37-cp37m-linux_ppc64le.whl
Requirement already satisfied: numpy>=1.14.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (1.17.4)
Requirement already satisfied: tqdm>=4.0.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (4.36.1)
Collecting h5py~=2.9.0
  Using cached h5py-2.9.0.tar.gz (287 kB)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (2020.6.20)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (1.25.10)
Requirement already satisfied: idna<2.9,>=2.5 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (2.8)
Requirement already satisfied: six in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchvision<0.6.0,>=0.4.0->torchmeta==1.3.1) (1.13.0)
Building wheels for collected packages: h5py
  Building wheel for h5py (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ccg1oj0n
       cwd: /tmp/pip-install-bpmeop26/h5py/
  Complete output (1321 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-ppc64le-3.7
  creating build/lib.linux-ppc64le-3.7/h5py
  copying h5py/__init__.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/h5py_warnings.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/highlevel.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/ipy_completer.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/version.py -> build/lib.linux-ppc64le-3.7/h5py
  creating build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/attrs.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/base.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/compat.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/dataset.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/datatype.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/dims.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/files.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/filters.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/group.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/selections.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/selections2.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/vds.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  creating build/lib.linux-ppc64le-3.7/h5py/tests
  copying h5py/tests/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests
  copying h5py/tests/common.py -> build/lib.linux-ppc64le-3.7/h5py/tests
  creating build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_attrs.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_attrs_data.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_base.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_dataset.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_datatype.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_dimension_scales.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_file.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_file_image.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_group.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5d_direct_chunk_write.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5f.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5p.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5t.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_objects.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_selections.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_slicing.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  creating build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_attribute_create.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_dataset_getitem.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_dataset_swmr.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_datatype.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_deprecation.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_dims_dimensionproxy.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_file.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_filters.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_threads.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  creating build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/test_highlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  running build_ext
  Autodetected HDF5 1.10.2
  ********************************************************************************
                         Summary of the h5py configuration
  
      Path to HDF5: None
      HDF5 Version: '1.10.2'
       MPI Enabled: False
  Rebuild Required: True
  
  ********************************************************************************
  Executing api_gen rebuild of defs
  Executing cythonize()
  [ 1/22] Cythonizing /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pyx
  /tmp/pip-install-bpmeop26/h5py/.eggs/Cython-0.29.21-py3.7.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pxd
    tree = Parsing.p_module(s, pxd, full_module_name)

...

  /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   #warning "Using deprecated NumPy API, disable it with " \
    ^
  In file included from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:0:
  /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
   #include "hdf5.h"
                    ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for h5py
  Running setup.py clean for h5py
Failed to build h5py
DEPRECATION: Could not build wheels for h5py which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
Installing collected packages: Pillow, h5py, torchmeta
  Attempting uninstall: Pillow
    Found existing installation: Pillow 7.1.2
    Uninstalling Pillow-7.1.2:
      Successfully uninstalled Pillow-7.1.2
  Attempting uninstall: h5py
    Found existing installation: h5py 2.8.0
    Uninstalling h5py-2.8.0:
      Successfully uninstalled h5py-2.8.0
    Running setup.py install for h5py ... error
    ERROR: Command errored out with exit status 1:
     command: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hlwpfooj/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py


... 

    copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
    copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
    running build_ext
    Autodetected HDF5 1.10.2
    ********************************************************************************
                           Summary of the h5py configuration
    
        Path to HDF5: None
        HDF5 Version: '1.10.2'
         MPI Enabled: False
    Rebuild Required: True
    
    ********************************************************************************
    Executing cythonize()
    [ 1/22] Cythonizing /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pyx
    /tmp/pip-install-bpmeop26/h5py/.eggs/Cython-0.29.21-py3.7.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pxd


...

    warning: h5py/api_types_hdf5.pxd:730:6: 'H5Z_ERROR_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:731:6: 'H5Z_DISABLE_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:732:6: 'H5Z_ENABLE_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:733:6: 'H5Z_NO_EDC' redeclared
    building 'h5py.defs' extension
    creating build/temp.linux-ppc64le-3.7
    creating build/temp.linux-ppc64le-3.7/tmp
    creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26
    creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py
    creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py/h5py
    gcc -pthread -B /home/miranda9/.conda/envs/my_new_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DH5_USE_16_API -I./h5py -I/tmp/pip-install-bpmeop26/h5py/lzf -I/opt/local/include -I/usr/local/include -I/home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include -I/home/miranda9/.conda/envs/my_new_env/include/python3.7m -c /tmp/pip-install-bpmeop26/h5py/h5py/defs.c -o build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py/h5py/defs.o
    In file included from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830:0,
                     from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                     from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:26,
                     from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:
    /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
     #warning "Using deprecated NumPy API, disable it with " \
      ^
    In file included from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:0:
    /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
     #include "hdf5.h"
                      ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
  Rolling back uninstall of h5py
  Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py
   from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py
  Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py-2.8.0-py3.7.egg-info
   from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py-2.8.0-py3.7.egg-info
ERROR: Command errored out with exit status 1: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hlwpfooj/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py Check the logs for full command output.

unfortunately I wasn't able to install that dependency and I am unsure what I should do. Any help?

similar post in SO: https://stackoverflow.com/questions/64049603/how-does-one-install-torchmeta-for-a-ppc64le-architecture-in-pytorch
@brando90 brando90 changed the title torchmeta support torchmeta support for powerai ppc64le Sep 24, 2020
This was referenced Sep 24, 2020
Open
Closed
Closed
@brando90
Copy link
Author

SO: https://stackoverflow.com/questions/64049603/how-does-one-install-torchmeta-for-a-ppc64le-architecture-in-pytorch

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@brando90