Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARM Installation Issues #285

Closed
Rhys-L opened this issue Sep 17, 2023 · 9 comments
Closed

ARM Installation Issues #285

Rhys-L opened this issue Sep 17, 2023 · 9 comments

Comments

@Rhys-L
Copy link

Rhys-L commented Sep 17, 2023

Hi @jgrss! I'm a student of Dr. Mann's @mmann1123 and Dr. Engstrom's, hoping to work with and build on Geowombat in some projects this semester. I've been having issues installing and running the package on an ARM-based mac. I have tried 3 different approaches, returning different errors. Do you have any recommendations or guidance on the best way to install? If you have any ideas on the docker image issue described below, I'm happy to push an ARM-version to a registry, or help debug more with any other approaches.

  1. The mamba and conda installations fail with different versions of the following error message from Mamba:
Could not solve for environment specs
The following package could not be installed
└─ geowombat   does not exist (perhaps a typo or a missing channel).
  1. I have attempted to build from source locally using the following in a python 3.9.6 venv using the following commands python setup.py build && python setup.py install, but I run into some clang issues with some of the cython generated code, returning the following error message: src/geowombat/moving/_moving.c:721:10: fatal error: 'omp.h' file not found #include <omp.h> ^~~~~~~ 1 warning and 1 error generated. error: command '/usr/bin/clang' failed with exit code 1

  2. I also cloned the code and have been working on natively building a docker image on mac. I modified the Docker file to use the following base image and env, but keep running into gdal version issues inside the docker image:
    FROM arm64v8/ubuntu:20.04 WORKDIR /source

I also modified env.yml and setup.cfg to contain gdal>=3.1.0. I also tried further modifying the docker file with variants of this, but haven't been able to successfully build the image: RUN pip install GDAL>=3.1.* --no-binary=gdal

Full error:
`12.61 Installing build dependencies: started

15.60 Installing build dependencies: finished with status 'done'

15.60 Getting requirements to build wheel: started

15.73 Getting requirements to build wheel: finished with status 'error'

15.73 error: subprocess-exited-with-error

15.73

15.73 × Getting requirements to build wheel did not run successfully.

15.73 │ exit code: 1

15.73 ╰─> [3 lines of output]

15.73 :22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg\_resources.html

15.73 INFO:root:GDAL API version obtained from gdal-config: 3.0.4

15.73 ERROR: GDAL >= 3.1 is required for rasterio. Please upgrade GDAL.

15.73 [end of output]`

ENV PLATFORM linux/arm64`

@mmann1123
Copy link
Collaborator

Hey Rhys, can you try the following for a dockerfile?

FROM jupyter/scipy-notebook:2021-11-20

USER root

RUN apt update -y && apt upgrade -y && \
    apt install -y software-properties-common && \
    add-apt-repository ppa:ubuntugis/ppa && \
    apt update -y && apt install -y \
    gdal-bin \
    geotiff-bin \
    git \
    libgdal-dev \
    libgl1 \
    libspatialindex-dev \ 
    wget \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN pip uninstall numpy -y 

# Install Conda packages
RUN conda install -c conda-forge -y \
    #python=3.7 # had to be excluded because newer version of python req by Jupyterhub \
    cython \
    gdal \
    libspatialindex \
    nbgrader \
    #nbtutor \
    numpy \
    opencv \
    pip \
    pygeos \
    requests \
    rtree \
    scipy \
    wheel \
    zarr \ 
    && conda install -c phausamann sklearn-xarray -y \
    && conda clean -a -y

RUN pip3 install --upgrade pip --no-cache-dir && pip3 install --no-cache-dir \
    pip-tools \
    git+https://github.com/jgrss/geowombat.git

@mmann1123
Copy link
Collaborator

I'm going to try to add ARM to my cond-forge build - I think the issue is it has specific C compiler needs.

@Rhys-L
Copy link
Author

Rhys-L commented Sep 18, 2023

It looks like the build got close but failed to from issues with rasterio and fiona. Going to keep googling through the error message, and wanted to share below in case anything jumps out.

67.28       INFO:root:gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /opt/conda/include -fPIC -O3 -isystem /opt/conda/include -fPIC -I/tmp/pip-build-env-tkwl4h61/overlay/lib/python3.9/site-packages/numpy/core/include -I/opt/conda/include -I/opt/conda/include/python3.9 -c rasterio/_warp.cpp -o build/temp.linux-aarch64-cpython-39/rasterio/_warp.o -Wno-unused-parameter -Wno-unused-function -std=c++11
67.28       gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
67.28       compilation terminated.
67.28       error: command '/usr/bin/gcc' failed with exit code 1
67.28       [end of output]
67.28   
67.28   note: This error originates from a subprocess, and is likely not a problem with pip.
67.28   ERROR: Failed building wheel for rasterio
67.28   Building wheel for fiona (pyproject.toml): started
69.50   Building wheel for fiona (pyproject.toml): finished with status 'error'
69.50   error: subprocess-exited-with-error
69.50   
69.50   × Building wheel for fiona (pyproject.toml) did not run successfully.
69.50   │ exit code: 1
69.50   ╰─> [76 lines of output]
69.50       /tmp/pip-build-env-e4ryhszq/overlay/lib/python3.9/site-packages/setuptools/_distutils/extension.py:134: UserWarning: Unknown Extension options: 'cython_compile_time_env'
69.50         warnings.warn(msg)
69.50       INFO:root:running bdist_wheel
69.50       INFO:root:running build
69.50       INFO:root:running build_py
69.50       INFO:root:creating build
69.50       INFO:root:creating build/lib.linux-aarch64-cpython-39
69.50       INFO:root:creating build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/enums.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/vfs.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/path.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/crs.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/model.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/io.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/meta.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/drvsupport.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/session.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/collection.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/logutils.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/transform.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/compat.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/__init__.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/_show_versions.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/errors.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/env.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/inspector.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/rfc3339.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:creating build/lib.linux-aarch64-cpython-39/fiona/_vendor
69.50       INFO:root:copying fiona/_vendor/__init__.py -> build/lib.linux-aarch64-cpython-39/fiona/_vendor
69.50       INFO:root:creating build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/filter.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/options.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/bounds.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/ls.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/cat.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/collect.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/dump.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/info.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/insp.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/main.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/distrib.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/__init__.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/rm.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/calc.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/helpers.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/load.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:copying fiona/fio/env.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50       INFO:root:creating build/lib.linux-aarch64-cpython-39/fiona/_vendor/munch
69.50       INFO:root:copying fiona/_vendor/munch/__init__.py -> build/lib.linux-aarch64-cpython-39/fiona/_vendor/munch
69.50       INFO:root:copying fiona/_vendor/munch/python3_compat.py -> build/lib.linux-aarch64-cpython-39/fiona/_vendor/munch
69.50       INFO:root:copying fiona/gdal.pxi -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/_err.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/crs.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/gdal.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/ogrext2.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/_geometry.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/_cpl.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/_csl.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/ogrext1.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/_shim.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/ogrext3.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:copying fiona/_env.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50       INFO:root:running build_ext
69.50       INFO:root:building 'fiona._geometry' extension
69.50       INFO:root:creating build/temp.linux-aarch64-cpython-39
69.50       INFO:root:creating build/temp.linux-aarch64-cpython-39/fiona
69.50       INFO:root:gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /opt/conda/include -fPIC -O3 -isystem /opt/conda/include -fPIC -I/opt/conda/include -I/opt/conda/include/python3.9 -c fiona/_geometry.c -o build/temp.linux-aarch64-cpython-39/fiona/_geometry.o
69.50       INFO:root:gcc -pthread -B /opt/conda/compiler_compat -shared -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib build/temp.linux-aarch64-cpython-39/fiona/_geometry.o -L/opt/conda/lib -lgdal -o build/lib.linux-aarch64-cpython-39/fiona/_geometry.cpython-39-aarch64-linux-gnu.so
69.50       INFO:root:building 'fiona.schema' extension
69.50       INFO:root:gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /opt/conda/include -fPIC -O3 -isystem /opt/conda/include -fPIC -I/opt/conda/include -I/opt/conda/include/python3.9 -c fiona/schema.c -o build/temp.linux-aarch64-cpython-39/fiona/schema.o
69.50       INFO:root:gcc -pthread -B /opt/conda/compiler_compat -shared -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib build/temp.linux-aarch64-cpython-39/fiona/schema.o -L/opt/conda/lib -lgdal -o build/lib.linux-aarch64-cpython-39/fiona/schema.cpython-39-aarch64-linux-gnu.so
69.50       INFO:root:building 'fiona._transform' extension
69.50       INFO:root:gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /opt/conda/include -fPIC -O3 -isystem /opt/conda/include -fPIC -I/opt/conda/include -I/opt/conda/include/python3.9 -c fiona/_transform.cpp -o build/temp.linux-aarch64-cpython-39/fiona/_transform.o -std=c++11
69.50       gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
69.50       compilation terminated.
69.50       error: command '/usr/bin/gcc' failed with exit code 1
69.50       [end of output]
69.50   
69.50   note: This error originates from a subprocess, and is likely not a problem with pip.
69.50   ERROR: Failed building wheel for fiona
69.50 Successfully built geowombat
69.50 Failed to build rasterio fiona
69.50 ERROR: Could not build wheels for rasterio, fiona, which is required to install pyproject.toml-based projects
------
Dockerfile:41
--------------------
  40 |     
  41 | >>> RUN pip3 install --upgrade pip --no-cache-dir && pip3 install --no-cache-dir \
  42 | >>>     pip-tools \
  43 | >>>     git+https://github.com/jgrss/geowombat.git
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -c pip3 install --upgrade pip --no-cache-dir && pip3 install --no-cache-dir     pip-tools     git+https://github.com/jgrss/geowombat.git" did not complete successfully: exit code: 1

@mmann1123
Copy link
Collaborator

For dockerfile try the simple version:

FROM jupyter/scipy-notebook 
USER root

RUN mamba install -c conda-forge -y geowombat

CMD ["/bin/bash"]

@Rhys-L
Copy link
Author

Rhys-L commented Sep 19, 2023

Got another error with that, I think because it is failing to find a version of the package. I'm currently looking into how we can use Docker compose and base image of linux with an intermediate image of a C compiler. Also reading up on this: https://learning.oreilly.com/library/view/learn-docker-in/9781617297052/OEBPS/Text/16.xhtml#heading_id_8, and open to any other docker reading/watching recommendations.

Error message:

> [2/2] RUN mamba install -c conda-forge -y geowombat:                                                                               
8.332                                                                                                                                 
8.332                   __    __    __    __                                                                                          
8.332                  /  \  /  \  /  \  /  \                                                                                         
8.332                 /    \/    \/    \/    \                                                                                        
8.332 ███████████████/  /██/  /██/  /██/  /████████████████████████
8.332               /  / \   / \   / \   / \  \____
8.332              /  /   \_/   \_/   \_/   \    o \__,
8.332             / _/                       \_____/  `
8.332             |/
8.332         ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
8.332         ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
8.332         ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
8.332         ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
8.332         ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
8.332         ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝
8.332 
8.332         mamba (0.18.1) supported by @QuantStack
8.332 
8.332         GitHub:  https://github.com/mamba-org/mamba
8.332         Twitter: https://twitter.com/QuantStack
8.332 
8.332 █████████████████████████████████████████████████████████████
8.332 
8.332 
8.332 Looking for: ['geowombat']
8.332 
8.332 
8.332 Pinned packages:
8.332   - python 3.9.*
8.332   - python 3.9.7
8.332 
8.332 
8.332 Encountered problems while solving:
8.332   - nothing provides requested geowombat
8.332 
8.332 conda-forge/linux-aarch6  
8.332 conda-forge/noarch        
------
Dockerfile:5
--------------------
   3 |     USER root
   4 |     
   5 | >>> RUN mamba install -c conda-forge -y geowombat
   6 |     
   7 |     CMD ["/bin/bash"]
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -c mamba install -c conda-forge -y geowombat" did not complete successfully: exit code: 1

@mmann1123
Copy link
Collaborator

That is very strange. Geowombat is definitely available through conda-forge, and I was able to run that docker file. How are you building it? Could u be limiting Internet access somehow???

@Rhys-L
Copy link
Author

Rhys-L commented Sep 19, 2023

I cloned the repo and I'm inside the top level directory and running the following command:

docker image build -t geowombat:arm .

Are there any different flags I should use here? I don't think it's a connectivity issue from inside the image or from my home internet.

@mmann1123
Copy link
Collaborator

I'll let you know how the conda-forge build goes. Till then use apporto.

@mmann1123
Copy link
Collaborator

@Rhys-L @jgrss I just pushed the osx arm build to the conda feedstock. Hopefully within a few hours it should be good to go!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants