Skip to content

Conversation

@TomAugspurger
Copy link
Contributor

@TomAugspurger TomAugspurger commented Jul 17, 2023

  • moved pctasks-frontend to an aux server
  • shrink container images a bit

Trying to fix the CI failures at https://github.com/microsoft/planetary-computer-tasks/actions/runs/5579847626/jobs/10195946932#step:11:419, where we seem to be running out of disk space.

--- ErrImagePull: rpc error: code = Unknown desc = failed to pull and unpack image "localhost:5001/pctasks-ingest:latest": failed to extract layer sha256:d32bc1b36f4802dd6c638c4501c6eb836f140b71383c0d5e87db50115060811a: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/285/fs/opt/az/lib/python3.10/site-packages/azure/synapse/artifacts/models/__pycache__/_models_py3.cpython-310.pyc: no space left on device: unknown

[419](https://github.com/microsoft/planetary-computer-tasks/actions/runs/5579847626/jobs/10195946932#step:11:420)

Tom Augspurger added 3 commits July 17, 2023 15:54
* moved pctasks-frontend to an aux server
* shrink container images a bit
@TomAugspurger
Copy link
Contributor Author

3403573 fixes an issue where local dev files like .mypy_cache might have been copied in to the container image. I don't know if this affected the images on CI (probably not) but it made my local image ~200 MB smaller)

e9ef5f8 disables the pip cache, which saves ~20MB.

@TomAugspurger
Copy link
Contributor Author

TomAugspurger commented Jul 17, 2023

The next big target is the task-* images. Currently, pctasks-task-base is >2.5GB.

See the diff below for a way to get it down to ~370MB.

diff --git a/Dockerfile.task_base b/Dockerfile.task_base
index c60bbeb..2cffc3b 100644
--- a/Dockerfile.task_base
+++ b/Dockerfile.task_base
@@ -1,4 +1,4 @@
-FROM ubuntu:20.04
+FROM python:3.9-slim
 
 # Setup timezone info
 ENV TZ=UTC
@@ -9,31 +9,6 @@ ENV PIP_NO_CACHE_DIR=1
 
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 
-RUN apt-get update && apt-get install -y software-properties-common
-
-RUN add-apt-repository ppa:ubuntugis/ppa && \
-     apt-get update && \
-     apt-get install -y build-essential python3-dev python3-pip \
-     jq unzip ca-certificates wget curl git && \
-     apt-get autoremove && apt-get autoclean && apt-get clean
-
-RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
-
-# See https://github.com/mapbox/rasterio/issues/1289
-ENV CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
-
-# Install Python 3.8
-RUN curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh" \
-   && bash "Mambaforge-$(uname)-$(uname -m).sh" -b -p /opt/conda \
-   && rm -rf "Mambaforge-$(uname)-$(uname -m).sh"
-
-ENV PATH /opt/conda/bin:$PATH
-ENV LD_LIBRARY_PATH /opt/conda/lib/:$LD_LIBRARY_PATH
-
-RUN mamba install -y -c conda-forge python=3.8 gdal pip setuptools cython numpy
-
-RUN python -m pip install --upgrade pip
-
 # Install common packages
 COPY requirements-task-base.txt /tmp/requirements.txt
 RUN python -m pip install --no-build-isolation -r /tmp/requirements.txt
diff --git a/requirements-task-base.txt b/requirements-task-base.txt
index 3de345b..39fe26a 100644
--- a/requirements-task-base.txt
+++ b/requirements-task-base.txt
@@ -1,2 +1,2 @@
 pystac[validation]==1.*
-rasterio==1.3.0 --no-binary=rasterio
\ No newline at end of file
+rasterio==1.3.0
\ No newline at end of file

This removes a bunch of "cruft" from the task-base Dockerfile. But we need to confirm whether any of that cruft is actually load-bearing.

  • Which base image do we need? Do we need ubuntu or can we use some other image?
  • Do we need conda / mamba?
  • Do we need to build rasterio from source?

@TomAugspurger
Copy link
Contributor Author

CI passed, so I'll punt on the questions from the previous comment.

@TomAugspurger TomAugspurger merged commit 5227f1d into main Jul 17, 2023
@TomAugspurger TomAugspurger deleted the tom/fix/aux-frontend branch July 17, 2023 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants