Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate docker image off geobase to osgeo/gdal #1233

Merged
merged 5 commits into from Feb 21, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
96 changes: 33 additions & 63 deletions docker/Dockerfile
Expand Up @@ -4,67 +4,55 @@
## Copyright (c) 2015-2020 ODC Contributors
## SPDX-License-Identifier: Apache-2.0
##
ARG ORG=opendatacube
ARG V_BASE=3.3.0
ARG V_PG=12
FROM ${ORG}/geobase-builder:${V_BASE} as env_builder
FROM osgeo/gdal:ubuntu-small-3.4.1

# Update and install Ubuntu packages

USER root
RUN apt-get update -y \
&& DEBIAN_FRONTEND=noninteractive apt-get upgrade -y \
&& DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-change-held-packages --fix-missing --no-install-recommends \
git \
libpq-dev libudunits2-dev \
python3-dev python3-distutils python3-pip \
build-essential \
postgresql \
postgresql-client-${V_PG} \
postgresql-${V_PG} \
sudo make graphviz \
tini \
SpacemanPaul marked this conversation as resolved.
Show resolved Hide resolved
&& apt-get clean

# Build constrained python environment

# Set the locale, this is required for some of the Python packages
ENV LC_ALL C.UTF-8

COPY docker/constraints.in /conf/requirements.txt
COPY docker/constraints.txt docker/nobinary.txt /conf/

# build wheels in cached directory to allow re-use even when requirements.txt is changed
# then copy to final location
# /root/.cache/pip <-- pip cache
# /var/cache/wheels <-- downloaded sources and compiled wheels
# /wheels <-- final destination copied from /var/cache/wheels cache
RUN --mount=type=cache,target=/root/.cache/pip \
--mount=type=cache,target=/var/cache/wheels \
mkdir /wheels \
&& env-build-tool wheels /conf/requirements.txt /conf/constraints.txt /var/cache/wheels \
&& python3 -m pip wheel \

RUN python3 -m pip install \
-r /conf/requirements.txt \
--find-links /var/cache/wheels \
--no-index \
--only-binary=:all: \
--wheel-dir /wheels
-c /conf/constraints.txt

# Copy datacube-core source code into container and install from source (with addons for tests).
COPY . /code

#################################################################################
# Runner stage
#################################################################################
FROM ${ORG}/geobase-runner:${V_BASE}
ARG V_PG
RUN python3 -m pip install '/code/[all]' \
&& python3 -m pip install /code/examples/io_plugin \
&& python3 -m pip install /code/tests/drivers/fail_drivers

# Set the locale, this is required for some of the Python packages
ENV LC_ALL C.UTF-8
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --fix-missing --no-install-recommends --allow-change-held-packages \
# datacube tests need redis
redis-server \
# include db for running tests
postgresql \
postgresql-client-${V_PG} \
postgresql-${V_PG} \
# to become test user
sudo \
# git is needed for sdist|bdist_wheel
git \
# for docs
make \
graphviz \
&& rm -rf /var/lib/apt/lists/*

# if using binary rasterio package, to support https
RUN mkdir -p /etc/pki/tls/certs \
&& ln -s /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt;
# Copy bootstrap script into image.
COPY docker/assets/with_bootstrap /usr/local/bin/

# prep db
RUN install --owner postgres --group postgres -D -d /var/run/postgresql /srv/postgresql \
&& sudo -u postgres "$(find /usr/lib/postgresql/ -type f -name initdb)" -D "/srv/postgresql" --auth-host=md5 --encoding=UTF8

# users and groups.
RUN groupadd --gid 1000 odc \
&& useradd --gid 1000 \
--uid 1000 \
Expand All @@ -77,26 +65,8 @@ RUN groupadd --gid 1000 odc \
&& install -d -o odc -g odc /code \
&& true

COPY --from=env_builder /bin/tini /bin/tini
USER odc
RUN --mount=type=bind,from=env_builder,target=/b \
--mount=type=cache,target=/home/odc/.cache/pip,uid=1000,gid=1000 \
env-build-tool new_no_index /b/conf/requirements.txt /b/conf/constraints.txt /env /b/wheels

# All steps from above should cache well
# from here down things depend on entire content of code
RUN --mount=type=bind,from=env_builder,target=/b \
--mount=type=bind,target=/src \
--mount=type=cache,target=/home/odc/.cache/pip,uid=1000,gid=1000 \
(cd /src && tar c . ) | (cd /code && tar x) \
&& /env/bin/python -m pip install -e '/code/[all]' --no-index --find-links=/b/wheels \
&& /env/bin/python -m pip install -e /code/examples/io_plugin --no-index --find-links=/b/wheels \
&& /env/bin/python -m pip install -e /code/tests/drivers/fail_drivers --no-index --find-links=/b/wheels

USER root
COPY docker/assets/with_bootstrap /usr/local/bin/

VOLUME /code
WORKDIR /code

ENTRYPOINT ["/bin/tini", "-s", "--", "/usr/local/bin/with_bootstrap"]
ENTRYPOINT ["/bin/tini", "-s", "--", "/usr/local/bin/with_bootstrap"]
SpacemanPaul marked this conversation as resolved.
Show resolved Hide resolved
1 change: 1 addition & 0 deletions docs/about/whats_new.rst
Expand Up @@ -8,6 +8,7 @@ What's New
v1.8.next
=========

- Migrate test docker image from `datacube/geobase` to `osgeo/gdal`. (:pull:`1233`)
- Separate index driver interface definition from default index driver implementation. (:pull:`1226`)
- Prefer WKT over EPSG when guessing CRS strings. (:pull:`1223`)
- Updates install docs. (:pull:`1208`, :pull:`1212`, :pull:`1215`)
Expand Down