How to multi-stage Docker build to reduce image size

[scheung38]

Description

How can I reduce Docker image size via multi stage build?

FROM tiangolo...

COPY .   /app

this produces over 1GB?

[dmontagu]

You might find this useful:

FROM python:3.7 as build
# Could have used FROM tiangolo... above instead

ENV POETRY_VIRTUALENVS_CREATE=false \
    PYTHONFAULTHANDLER=1 \
    PYTHONUNBUFFERED=1 \
    PYTHONHASHSEED=random \
    PIP_NO_CACHE_DIR=off \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100
RUN pip install --upgrade pip

# Install Poetry
# See https://github.com/sdispater/poetry/issues/1301 for pros/cons of this approach
RUN curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | POETRY_PREVIEW=1 python

WORKDIR /app
COPY /app/requirements/requirements-poetry.txt /app/requirements/
RUN pip install \
    --user \
    --no-warn-script-location \
    -r requirements/requirements-poetry.txt
COPY ./app/requirements /app/requirements
RUN pip install \
    --user \
    --no-warn-script-location \
    --find-links=requirements/wheels \
    -r requirements/requirements.txt
COPY ./app /app
RUN /root/.poetry/bin/poetry install --no-dev

FROM python:3.7-slim as output

RUN useradd -m worker \
 && mkdir /app \
 && chown -R worker:worker /app
USER worker
WORKDIR /app

ENV PYTHONPATH=/app \
    HOME=/home/worker \
    PATH="/home/worker/.local/bin:${PATH}"

COPY --from=build --chown=worker:worker /usr/local/lib/python3.7/site-packages /usr/local/lib/python3.7/site-packages
COPY --from=build --chown=worker:worker /root/.local /home/worker/.local
COPY --from=build --chown=worker:worker /app /app

You'll obviously need to adapt it to your own use case, but might find it to be a useful starting point.

You'd also need to copy over the various scripts used in the tiangolo:X docker images, like /start.sh or /start-reload.sh. (I use the above as a base image; I build a few other images on top of it by adding minor tweaks.)

The above results in a container size that is ~300MB for me, rather than ~1GB.

Multi stage builds do have some downsides for day-to-day development though -- build caching generally doesn't work as nicely.

[trim21]

I'm using docher image python:3.7

FROM python:3.7 as generator
WORKDIR /src/app
COPY .circleci/export_requirements.py poetry.lock /src/app/
RUN pip install toml==0.10.0 && python export_requirements.py

#######################################

FROM python:3.7
LABEL MAINTAINER="Trim21 <Trim21me@gmail.com>"
ENV PYTHONPATH=/

COPY --from=generator /src/app/requirements.txt /

RUN pip install --require-hashes --no-cache-dir --no-deps -r /requirements.txt

COPY / /

export_requirements.py

import os
import argparse

import toml


def _poetry_lock_to_requirements_txt(start_dir, options):
    with open(f'{start_dir}/poetry.lock') as f:
        lock = toml.load(f)
    hashes = lock['metadata']['hashes']
    s = ''
    for package in lock['package']:
        if not options.all and (
            package.get('category') != 'main' or package.get('optional')
        ):
            continue

        name = package['name']
        version = package['version']
        marker = f'; {package["marker"]}' if 'marker' in package else ''

        hash_args = ' '.join([f'--hash=sha256:{x}' for x in hashes.get(name, [])])

        s += f'{name}=={version} {marker} {hash_args}\n'

    with open(f'{start_dir}/requirements.txt', 'w', encoding='utf-8') as f:
        f.write(s)


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--all', default=False, action='store_true')
    opt = parser.parse_args()
    workdir = os.getcwd()
    _poetry_lock_to_requirements_txt(workdir, opt)

[pmav99]

The fastapi docker images are based on the the base python image which is 900MB+ so you can't get slimmer than that... You can either use the alpine based fastapi images or create your own images (e.g. based on python:##-slim like @dmontagu's example).

That being said, I don't find multi-stage builds particularly useful in pure python applications. Just remember to disable the pip cache and remove the *.pyc files each time you run pip install (e.g. with find / -xdev -name *.pyc -delete;).

[tiangolo]

Thanks everyone for the discussion and suggestions!

And yeah, agreed with them.

I would also recommend to start simple, and fine-tune performance or size later when you need it, or when you're confident about it. But avoid premature optimizations.

[github-actions]

Assuming the original issue was solved, it will be automatically closed now. But feel free to add more comments or create new issues.