Skip to content

Improve layer reuse of python images #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Nov 18, 2024
Merged

Improve layer reuse of python images #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
259c852
Use matrix parameters to simplify job definition
hc2p Nov 5, 2024
dcd4861
Remove not needed Dockerfile
hc2p Nov 6, 2024
cf6c086
Make more layers of python image reusable
hc2p Nov 6, 2024
bb91be7
Ensure images are for amd64 only
hc2p Nov 8, 2024
0d17795
Add missing ca-certificates
hc2p Nov 11, 2024
12228e4
Centralize python-version list
hc2p Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 87 additions & 38 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,74 @@
version: 2.1

orbs:
docker: circleci/docker@2.2.0
docker: circleci/docker@2.8


commands:
split_python_version:
steps:
- run:
name: Split python version
command: |
echo "export PYTHON_VERSION=$(echo $PYTHON_VERSION_PATCH | cut -d'.' -f1,2)" >> $BASH_ENV
source $BASH_ENV
setup_buildkit_builder:
steps:
- run:
name: Setup BuildKit builder
command: |
docker buildx create \
--name custom-builder \
--use \
--driver docker-container \
jobs:
build-and-push-base:
executor: docker/docker
steps:
- checkout
- setup_remote_docker
- setup_buildkit_builder
- docker/check:
use-docker-credentials-store: true
- docker/build:
step-name: Base python image (debian + apt-get deps)
path: ./python/base
dockerfile: Dockerfile.base
extra_build_args: |
--progress plain
--platform linux/amd64
--cache-from type=registry,ref=deepnote/python:base-buildcache,mode=max
--cache-to type=registry,ref=deepnote/python:base-buildcache,mode=max
--output type=registry,push=true
image: deepnote/python
tag: base${CIRCLE_PULL_REQUEST:+-ra-${CIRCLE_PULL_REQUEST##*/}}
build-and-push-python:
executor: docker/docker
parameters:
python-version:
type: string
environment:
PYTHON_VERSION: << parameters.python-version >>
PYTHON_VERSION_PATCH: << parameters.python-version >>
steps:
- checkout
- setup_remote_docker
- docker/check:
use-docker-credentials-store: true
- split_python_version
- setup_buildkit_builder
- docker/build:
step-name: Base Python image
step-name: Python image
path: ./python/base
dockerfile: Dockerfile
extra_build_args: "--build-arg FROM_PYTHON_VERSION=${PYTHON_VERSION}"
image: deepnote/python
tag: ${PYTHON_VERSION}${CIRCLE_PULL_REQUEST:+-ra-${CIRCLE_PULL_REQUEST##*/}}
- docker/push:
dockerfile: Dockerfile.python
extra_build_args: |
--build-arg CIRCLE_PULL_REQUEST=${CIRCLE_PULL_REQUEST}
--build-arg PYTHON_VERSION=${PYTHON_VERSION}
--build-arg PYTHON_VERSION_PATCH=${PYTHON_VERSION_PATCH}
--progress plain
--platform linux/amd64
--cache-from type=registry,ref=deepnote/python:${PYTHON_VERSION}-buildcache,mode=max
--cache-to type=registry,ref=deepnote/python:${PYTHON_VERSION}-buildcache,mode=max
--output type=registry,push=true
image: deepnote/python
tag: ${PYTHON_VERSION}${CIRCLE_PULL_REQUEST:+-ra-${CIRCLE_PULL_REQUEST##*/}}
build-and-push-python-datascience:
Expand All @@ -32,24 +77,30 @@ jobs:
python-version:
type: string
environment:
PYTHON_VERSION: << parameters.python-version >>
PYTHON_VERSION_PATCH: << parameters.python-version >>
steps:
- checkout
- setup_remote_docker
- docker/check:
use-docker-credentials-store: true
- split_python_version
- setup_buildkit_builder
- docker/build:
step-name: Datascience Python image
path: ./python/datascience
docker-context: ./python/datascience
dockerfile: Dockerfile.datascience
extra_build_args: "--build-arg FROM_PYTHON_VERSION=${PYTHON_VERSION}"
extra_build_args: |
--build-arg PYTHON_VERSION=${PYTHON_VERSION}
--build-arg CIRCLE_PULL_REQUEST=${CIRCLE_PULL_REQUEST}
--progress plain
--platform linux/amd64
--cache-from type=registry,ref=deepnote/python:${PYTHON_VERSION}-buildcache,mode=max
--cache-from type=registry,ref=deepnote/python-datascience:${PYTHON_VERSION}-buildcache,mode=max
--cache-to type=registry,ref=deepnote/python-datascience:${PYTHON_VERSION}-buildcache,mode=max
--output type=registry,push=true
image: deepnote/python-datascience
tag: ${PYTHON_VERSION}${CIRCLE_PULL_REQUEST:+-ra-${CIRCLE_PULL_REQUEST##*/}}
- docker/push:
image: deepnote/python-datascience
tag: ${PYTHON_VERSION}${CIRCLE_PULL_REQUEST:+-ra-${CIRCLE_PULL_REQUEST##*/}}

build-and-push-gpu:
executor: docker/docker
parameters:
Expand All @@ -68,44 +119,42 @@ jobs:
- setup_remote_docker
- docker/check:
use-docker-credentials-store: true
- split_python_version
- docker/build:
step-name: Tensorflow GPU image
path: ./gpu
extra_build_args: "--build-arg CUDA_IMAGE_TAG=${CUDA_TAG} --build-arg TF_VERSION=${TF_VERSION} --build-arg PYTHON_VER=${PY_VERSION}"
extra_build_args: "--build-arg CUDA_IMAGE_TAG=${CUDA_TAG} --build-arg TF_VERSION=${TF_VERSION} --build-arg PYTHON_VER=${PY_VERSION} --progress plain"
image: deepnote/tensorflow
tag: ${TF_VERSION}${CIRCLE_PULL_REQUEST:+-ra-${CIRCLE_PULL_REQUEST##*/}}-gpu
- docker/push:
image: deepnote/tensorflow
tag: ${TF_VERSION}${CIRCLE_PULL_REQUEST:+-ra-${CIRCLE_PULL_REQUEST##*/}}-gpu

python-versions: &python-versions
- "3.8.19"
- "3.9.19"
- "3.10.15"
- "3.11.10"

workflows:
build:
jobs:
- build-and-push-base:
name: Building the base image
- build-and-push-python:
name: Python 3.8
python-version: "3.8"
- build-and-push-python:
name: Python 3.9
python-version: "3.9"
- build-and-push-python:
name: Python 3.10
python-version: "3.10"
- build-and-push-python:
name: Python 3.11
python-version: "3.11"

- build-and-push-python-datascience:
name: Python Datascience 3.8
python-version: "3.8"
name: Python << matrix.python-version >>
matrix:
parameters:
python-version: *python-versions
requires:
- Building the base image
- build-and-push-python-datascience:
name: Python Datascience 3.9
python-version: "3.9"
- build-and-push-python-datascience:
name: Python Datascience 3.10
python-version: "3.10"
- build-and-push-python-datascience:
name: Python Datascience 3.11
python-version: "3.11"
name: Python Datascience << matrix.python-version >>
matrix:
parameters:
python-version: *python-versions
requires:
- Python << matrix.python-version >>

- build-and-push-gpu:
name: Tensorflow 2.9
Expand Down
36 changes: 0 additions & 36 deletions python/Dockerfile
View file
Open in desktop

This file was deleted.

36 changes: 0 additions & 36 deletions python/base/Dockerfile
View file
Open in desktop

This file was deleted.

32 changes: 32 additions & 0 deletions python/base/Dockerfile.base
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
FROM debian:bullseye-slim
ENV DEBIAN_FRONTEND noninteractive

# Install OS dependencies
RUN apt-get update && apt-get -yq dist-upgrade \
&& apt-get install -yq --no-install-recommends \
build-essential \
bzip2 \
cmake \
curl \
git \
graphviz \
libgtk2.0-dev \
locales \
sudo \
unzip \
vim \
wget \
ssh \
gnupg2 \
ca-certificates \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

ENV SHELL=/bin/bash \
LC_ALL=en_US.UTF-8 \
LANG=en_US.UTF-8 \
LANGUAGE=en_US.UTF-8 \
DEEPNOTE_PYTHON_KERNEL_ONLY=true

RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
locale-gen
51 changes: 51 additions & 0 deletions python/base/Dockerfile.python
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
ARG CIRCLE_PULL_REQUEST
FROM debian:bullseye-slim AS builder

# Install dependencies for building Python
RUN apt-get update && apt-get install -y --no-install-recommends \
wget \
cmake \
build-essential \
libssl-dev \
zlib1g-dev \
libncurses5-dev \
libncursesw5-dev \
libreadline-dev \
libsqlite3-dev \
libgdbm-dev \
libdb5.3-dev \
libbz2-dev \
libexpat1-dev \
liblzma-dev \
tk-dev \
libffi-dev \
uuid-dev \
ca-certificates \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

# Download and extract the Python source code
WORKDIR /tmp
ARG PYTHON_VERSION_PATCH
RUN wget -qO- "https://www.python.org/ftp/python/${PYTHON_VERSION_PATCH}/Python-${PYTHON_VERSION_PATCH}.tgz" | tar xvz \
&& cd "/tmp/Python-${PYTHON_VERSION_PATCH}" \
&& ./configure --enable-optimizations --with-ensurepip=install \
&& make -j "$(nproc)" \
&& make altinstall


FROM deepnote/python:base${CIRCLE_PULL_REQUEST:+-ra-${CIRCLE_PULL_REQUEST##*/}}
# Copy Python from the builder stage
ARG PYTHON_VERSION
# Layers will be different between python versions from here onwards because of the build-arg

COPY --from=builder "/usr/local/bin/python${PYTHON_VERSION}" "/usr/local/bin/python${PYTHON_VERSION}"
COPY --from=builder "/usr/local/bin/pip${PYTHON_VERSION}" "/usr/local/bin/pip${PYTHON_VERSION}"
COPY --from=builder "/usr/local/lib/python${PYTHON_VERSION}" "/usr/local/lib/python${PYTHON_VERSION}"

RUN update-alternatives --install /usr/bin/python python "/usr/local/bin/python${PYTHON_VERSION}" 1
RUN update-alternatives --install /usr/bin/pip pip "/usr/local/bin/pip${PYTHON_VERSION}" 1

Comment on lines +39 to +48
Copy link
Contributor

@chudyandrej chudyandrej Nov 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic is somehow copied from official Python because I'm not sure if it is the recommended way or not.

Copy link
Contributor

@chudyandrej chudyandrej Nov 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am just afraid about some side effects that could not be transferred from the first stage.

Copy link
Contributor

@chudyandrej chudyandrej Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But if you are sure that it is fine, I'm okay with that. It's working, so I think we are good.

Copy link
Contributor Author

@hc2p hc2p Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's basically just for symlinking python and pip binaries.
Official python is doing it differently, which we can of course adapt to.

# make some useful symlinks that are expected to exist ("/usr/local/bin/python" and friends)
RUN set -eux; \
	for src in idle3 pip3 pydoc3 python3 python3-config; do \
		dst="$(echo "$src" | tr -d 3)"; \
		[ -s "/usr/local/bin/$src" ]; \
		[ ! -e "/usr/local/bin/$dst" ]; \
		ln -svT "$src" "/usr/local/bin/$dst"; \
	done

Copy link
Contributor

@chudyandrej chudyandrej Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, let's discuss this. I think I don't understand how this can work.

# We create the virtual environment in the home directory in the Dockerfile
# for performance improvement.
RUN python -m venv --system-site-packages ~/venv
11 changes: 6 additions & 5 deletions python/datascience/Dockerfile.datascience
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
ARG FROM_PYTHON_VERSION=3.8
FROM deepnote/python:${FROM_PYTHON_VERSION}
ARG PYTHON_VERSION=3.8
ARG CIRCLE_PULL_REQUEST
FROM deepnote/python:${PYTHON_VERSION}${CIRCLE_PULL_REQUEST:+-ra-${CIRCLE_PULL_REQUEST##*/}}

RUN pip install --upgrade pip

Expand All @@ -10,11 +11,11 @@ ADD requirements-below-3.11.txt /requirements-below-3.11.txt
ADD constraints.txt /constraints.txt

#Determine the Python version and set the version-specifications file
ARG FROM_PYTHON_VERSION
ARG PYTHON_VERSION

RUN python -m pip install --upgrade setuptools pip

RUN if [ "$(printf '%s\n' "$FROM_PYTHON_VERSION" "3.11" | sort -V | head -n1)" = "3.11" ]; then \
RUN if [ "$(printf '%s\n' "$PYTHON_VERSION" "3.11" | sort -V | head -n1)" = "3.11" ]; then \
mv "requirements-3.11+.txt" "requirements.txt" \
; else \
mv "requirements-below-3.11.txt" "requirements.txt" \
Expand All @@ -28,4 +29,4 @@ RUN apt-get update \
&& apt-get purge -y gcc \
&& apt-get autoremove -y \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /root/.cache
&& rm -rf /var/lib/apt/lists/* /root/.cache