Add podman-hpc container for Isambard AI

CHANGELOG.md

@@ -25,6 +25,7 @@ _Add new changes here_
 - Added ability to save and reload text predictions ([#536](https://github.com/maps-as-data/MapReader/pull/536)
 - Added minimal dataclasses for text predictions ([#536](https://github.com/maps-as-data/MapReader/pull/536)
 - Added `skip_blank_patches` argument to `MapImages.patchify_all()` ([#540](https://github.com/maps-as-data/MapReader/pull/540))
+- Added `container` directory with everything needed for running MapReader with podman-hpc ([#552](https://github.com/maps-as-data/MapReader/pull/552))
 
 ## [v1.6.1](https://github.com/Living-with-machines/MapReader/releases/tag/v1.6.1) (2024-11-18)
 

container/Dockerfile

@@ -0,0 +1,82 @@
+# Use the NVIDIA PyTorch base image with CUDA support
+FROM nvcr.io/nvidia/pytorch:24.06-py3
+
+# Set environment variables to prevent Python from writing .pyc files and buffering stdout/stderr
+ENV PYTHONDONTWRITEBYTECODE=1
+ENV PYTHONUNBUFFERED=1
+
+# Install system-level dependencies and build tools
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    cmake \
+    wget \
+    software-properties-common \
+    libgeos-dev \
+    libproj-dev \
+    proj-data \
+    proj-bin \
+    libffi-dev \
+    libssl-dev \
+    libcurl4-openssl-dev \
+    libexpat1-dev \
+    libxerces-c-dev \
+    zlib1g-dev \
+    libpng-dev \
+    libjpeg-dev \
+    libtiff-dev \
+    libspatialite-dev \
+    libgl1 \
+    python3-dev \
+    python3-pip \
+    && rm -rf /var/lib/apt/lists/*
+
+# Upgrade pip to the latest version
+RUN pip3 install --upgrade pip
+
+# Install GDAL dependencies
+RUN apt-get update && apt-get install -y \
+    libcurl4-openssl-dev \
+    libexpat1-dev \
+    libxerces-c-dev \
+    zlib1g-dev \
+    libpng-dev \
+    libjpeg-dev \
+    libtiff-dev \
+    libspatialite-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Download and build GDAL 3.5.3 from source
+RUN wget https://download.osgeo.org/gdal/3.5.3/gdal-3.5.3.tar.gz -O /tmp/gdal-3.5.3.tar.gz && \
+    tar -xzf /tmp/gdal-3.5.3.tar.gz -C /tmp && \
+    cd /tmp/gdal-3.5.3 && \
+    ./configure --prefix=/usr/local && \
+    make -j$(nproc) && \
+    make install && \
+    ldconfig && \
+    cd / && \
+    rm -rf /tmp/gdal-3.5.3 /tmp/gdal-3.5.3.tar.gz
+
+# Verify GDAL installation
+RUN gdalinfo --version
+
+# Set environment variables to help rasterio find GDAL
+ENV CPLUS_INCLUDE_PATH=/usr/local/include
+ENV C_INCLUDE_PATH=/usr/local/include
+ENV LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
+
+# Copy requirements into the container
+COPY requirements.txt /tmp/
+COPY text-requirements.txt /tmp/
+
+# Install Python dependencies from requirements.txt
+RUN pip3 install --no-cache-dir -r /tmp/requirements.txt
+RUN pip3 install --no-cache-dir -r /tmp/text-requirements.txt
+
+# Install MapReader from GitHub without dependencies to prevent altering existing packages
+RUN pip3 install git+https://github.com/Living-with-machines/MapReader.git --no-deps
+
+# Set the working directory
+WORKDIR /workspace
+
+# Define the default command to run when starting the container
+CMD ["/bin/bash"]

container/README.md

@@ -0,0 +1,83 @@
+# Run MapReader on Isambard-AI
+
+This document helps you to run MapReader on [Isambard-AI](https://docs.isambard.ac.uk/specs/#system-specifications-isambard-ai-phase-1).
+
+## Set-up
+
+1. Get familar with [Podman-HPC at Isambard](https://docs.isambard.ac.uk/user-documentation/guides/containers/podman-hpc/).
+
+2. Clone MapReader and navigate to the container directory.
+
+```bash
+git clone https://github.com/maps-as-data/MapReader.git
+cd MapReader/container
+```
+## Build and run the image
+
+### Build the image
+
+After navigating the `container` directory in the MapReader repo, run the following command to build the image:
+
+```bash
+podman-hpc build -t mapreader .
+```
+
+If you run into any issues with this, see the ```Pull the image``` section at the bottom of this document.
+
+### Migrate the image
+
+Migration is the process of moving the image to the shared filesystem.
+This is needed to run the MapReader image on the compute nodes.
+
+To migrate the image, run the following command:
+
+```bash
+podman-hpc migrate mapreader:latest
+```
+
+### Run as a batch job
+
+To run the MapReader image as part of a batch job, you will need a slurm script.
+Ours looks like this:
+
+
+To run the script, run the following command:
+
+```bash
+sbatch pytorch_run_podmanhpc.sh
+```
+
+### Run in an interactive shell
+
+If instead you'd like to run in an interactive shell, you can use:
+
+```bash
+podman-hpc run -it --gpu --rm --name mapreader \
+  localhost/mapreader:latest /bin/bash
+```
+
+If you'd like to do this using GPU, you will need to launch an interactive job first using srun:
+
+```bash
+srun --gres=gpu:1 -A <project> --time 1:00:00 --pty /bin/bash
+```
+
+Then you can run the podman-hpc command above, the `--gpu` flag will ensure that your requested GPUs are available to the container.
+
+### Run the dependency test
+
+To check everything is working as expected, run the dependency test using the following command:
+
+```bash
+python test.py
+```
+
+## Pull the image
+
+If you have issues building the image, please to pull it from the [docker hub](https://hub.docker.com/repository/docker/oxfordfun/mapreader/tags)
+
+```bash
+podman-hpc pull oxfordfun/mapreader:3.5.3-f
+```
+
+Then go on to migrate and run the image as described above.

container/requirements.txt

@@ -0,0 +1,42 @@
+aiohttp>=3.8.1,<4.0.0
+azure-storage-blob>=12.9.0,<13.0.0
+black>=23.7.0,<25.0.0
+cartopy>=0.22.0
+Cython>=0.29.24,<0.30.0
+fiona>=1.10.1
+flake8>=6.0.0,<8.0.0
+geopandas<1.0.0
+geopy==2.1.0
+ipyannotate==0.1.0-beta.0
+ipykernel>=6.5.1,<7.0.0
+ipywidgets>=8.0.0,<9.0.0
+isodate>=0.7.2
+joblib>=1.4.0
+jupyter>=1.0.0,<2.0.0
+jupyter-console>=6.6.3
+jupyterlab-widgets>=3.0.13
+mapclassify>=2.0.0,<3.0.0
+matplotlib>=3.5.0,<4.0.0
+nest-asyncio>=1.5.1,<2.0.0
+numpy>=1.21.5,<2.0.0
+opencv-python>=4.8.1.78,<5.0.0
+pandas>=2.0.0
+parhugin>=0.0.3,<0.0.4
+pyogrio>=0.7.2
+pyproj>=3.2.0,<4.0.0
+pytest<9.0.0
+pytest-cov>=4.1.0,<6.0.0
+PyYAML>=6.0,<7.0
+rasterio>=1.2.10,<2.0.0
+scikit-image>=0.18.3
+scikit-learn>=1.0.1,<2.0.0
+Shapely>=2.0.0,<3.0.0
+simplekml>=1.3.6,<2.0.0
+tensorboard>=2.7.0,<3.0.0
+timm<1.0.0
+torchinfo<2.0.0
+tqdm<5.0.0
+transformers<5.0.0
+versioneer>=0.28
+widgetsnbextension>=4.0.13
+xyzservices==2024.9.0

container/run.sh

@@ -0,0 +1,7 @@
+
+#! /bin/bash
+
+# podman-hpc build -t mapreader .
+
+podman-hpc run -it --gpu --rm --name mapreader \
+  localhost/mapreader:latest /bin/bash

container/test.output

@@ -0,0 +1,140 @@
+=== Running Container: mapreader ===
+✅ Container 'mapreader' is running with ID: 85c4da8d8d7b131499ec37d83822efa0cb8f21f8a5450add23ab0e9a4529d0f3
+
+=== Executing Dependency Tests Inside Container 'mapreader' ===
+
+🔍 Running Test: PyTorch
+📦 Library: PyTorch
+✅ PyTorch Output: 2.4.0a0+f70bd71a48.nv24.06
+
+🔍 Running Test: PyTorch GPU
+📦 Library: PyTorch GPU
+✅ PyTorch GPU Output: True
+
+🔍 Running Test: Torchvision
+📦 Library: Torchvision
+✅ Torchvision Output: 0.19.0a0
+
+🔍 Running Test: Geopandas
+📦 Library: Geopandas
+✅ Geopandas Output: 0.14.4
+
+🔍 Running Test: Geopy
+📦 Library: Geopy
+✅ Geopy Output: 2.1.0
+
+🔍 Running Test: Cython Python
+📦 Library: Cython Python
+✅ Cython Python Output: 0.29.37
+
+🔍 Running Test: Torchinfo
+📦 Library: Torchinfo
+✅ Torchinfo Output: 1.8.0
+
+🔍 Running Test: Parhugin MultiFun
+📦 Library: Parhugin MultiFun
+✅ Parhugin MultiFun Output: [INFO] #requested processes: 10
+
+🔍 Running Test: MapReader
+📦 Library: MapReader
+✅ MapReader Output: 1.6.2.post0.dev4
+
+🔍 Running Test: GDAL
+📦 Library: GDAL
+✅ GDAL Output: GDAL 3.5.3, released 2022/10/21
+
+🔍 Running Test: Fiona
+📦 Library: Fiona
+✅ Fiona Output: 1.10.1
+
+🔍 Running Test: Shapely
+📦 Library: Shapely
+✅ Shapely Output: 2.0.6
+
+🔍 Running Test: Scikit-learn
+📦 Library: Scikit-learn
+✅ Scikit-learn Output: 1.5.0
+
+🔍 Running Test: Scikit-image
+📦 Library: Scikit-image
+✅ Scikit-image Output: 0.25.0
+
+🔍 Running Test: Tensorboard
+📦 Library: Tensorboard
+✅ Tensorboard Output: 2.9.0
+
+🔍 Running Test: Jupyter
+📦 Library: Jupyter
+✅ Jupyter Output: Selected Jupyter core packages...
+IPython          : 8.21.0
+ipykernel        : 6.29.4
+ipywidgets       : 8.1.5
+jupyter_client   : 8.6.2
+jupyter_core     : 5.7.2
+jupyter_server   : not installed
+jupyterlab       : 2.3.2
+nbclient         : 0.10.0
+nbconvert        : 7.16.4
+nbformat         : 5.10.4
+notebook         : 6.4.10
+qtconsole        : not installed
+traitlets        : 5.9.0
+
+🔍 Running Test: IPython Kernel
+📦 Library: IPython Kernel
+✅ IPython Kernel Output: 6.29.4
+
+🔍 Running Test: IPyWidgets
+📦 Library: IPyWidgets
+✅ IPyWidgets Output: 8.1.5
+
+🔍 Running Test: OpenCV
+📦 Library: OpenCV
+✅ OpenCV Output: 4.5.5
+
+🔍 Running Test: Rasterio
+📦 Library: Rasterio
+✅ Rasterio Output: 1.4.3
+
+=== Analyzing Test Results ===
+
+✅ PyTorch: 2.4.0a0+f70bd71a48.nv24.06
+✅ PyTorch GPU: True
+✅ Torchvision: 0.19.0a0
+✅ Geopandas: 0.14.4
+✅ Geopy: 2.1.0
+✅ Cython Python: 0.29.37
+✅ Torchinfo: 1.8.0
+✅ Parhugin MultiFun: [INFO] #requested processes: 10
+✅ MapReader: 1.6.2.post0.dev4
+✅ GDAL: GDAL 3.5.3, released 2022/10/21
+✅ Fiona: 1.10.1
+✅ Shapely: 2.0.6
+✅ Scikit-learn: 1.5.0
+✅ Scikit-image: 0.25.0
+✅ Tensorboard: 2.9.0
+✅ Jupyter: Selected Jupyter core packages...
+IPython          : 8.21.0
+ipykernel        : 6.29.4
+ipywidgets       : 8.1.5
+jupyter_client   : 8.6.2
+jupyter_core     : 5.7.2
+jupyter_server   : not installed
+jupyterlab       : 2.3.2
+nbclient         : 0.10.0
+nbconvert        : 7.16.4
+nbformat         : 5.10.4
+notebook         : 6.4.10
+qtconsole        : not installed
+traitlets        : 5.9.0
+✅ IPython Kernel: 6.29.4
+✅ IPyWidgets: 8.1.5
+✅ OpenCV: 4.5.5
+✅ Rasterio: 1.4.3
+
+=== Test Analysis Complete ===
+
+🎉 All dependency tests passed successfully!
+=== Cleaning Up: Removing Container 'mapreader' ===
+✅ Container 'mapreader' has been removed.
+