emailservice Image Optimization #51

orthros · 2018-09-25T17:36:16Z

Reduce docker image for emailservice to ~240 MB (down from ~ 1.31 GB)

Main application (email_server.py) now runs as python 2.7. Before we had both Python 2.7 and Python 3 installed in the image.

Switched to using python:2.7-alpine3.8 as the base image, and used multi-stage dockerfiles to keep dependencies minimal.

Fixes #49

From my shell:

$ docker build -t emailservice:dev . && docker run -it emailservice:dev
Sending build context to Docker daemon  97.28kB
Step 1/17 : FROM python:2.7-alpine3.8 as base
 ---> b2bc7255b42c
Step 2/17 : FROM base as builder
 ---> b2bc7255b42c
Step 3/17 : RUN apk add --update --no-cache     gcc     linux-headers     make     musl-dev     python-dev     g++     cairo-dev     cairo     openssl-dev     gobject-introspection-dev
 ---> Using cache
 ---> 6daf3d9fe49a
Step 4/17 : ENV GRPC_PYTHON_VERSION 1.15.0
 ---> Using cache
 ---> 3e33d97d9580
Step 5/17 : RUN python -m pip install --upgrade pip
 ---> Using cache
 ---> e8fa3879c282
Step 6/17 : RUN pip install grpcio==${GRPC_PYTHON_VERSION} grpcio-tools==${GRPC_PYTHON_VERSION}
 ---> Using cache
 ---> c6fba7743eed
Step 7/17 : COPY requirements.txt .
 ---> Using cache
 ---> 1f6b0a444980
Step 8/17 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> 8cc0a7af6aa8
Step 9/17 : FROM base as final
 ---> b2bc7255b42c
Step 10/17 : RUN GRPC_HEALTH_PROBE_VERSION=v0.1.0-alpha.1 &&     wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64 &&     chmod +x /bin/grpc_health_probe
 ---> Using cache
 ---> e954a0384081
Step 11/17 : ENV PYTHONUNBUFFERED=0
 ---> Using cache
 ---> 64ece3d72a66
Step 12/17 : WORKDIR /email_server
 ---> Using cache
 ---> 27b34dc14492
Step 13/17 : COPY --from=builder /usr/local/lib/python2.7/ /usr/local/lib/python2.7/
 ---> Using cache
 ---> 60035ec8dfd4
Step 14/17 : RUN apk add --no-cache libstdc++
 ---> Using cache
 ---> 920be90c126e
Step 15/17 : COPY . .
 ---> Using cache
 ---> 9541bed2d7a0
Step 16/17 : EXPOSE 8080
 ---> Using cache
 ---> 48fbeaa852b9
Step 17/17 : ENTRYPOINT [ "python", "email_server.py" ]
 ---> Using cache
 ---> ff317770992d
Successfully built ff317770992d
Successfully tagged emailservice:dev
starting the email service in dummy mode.
listening on port: 8080

Merge from Upstream

Merge from master

…builds and alpine image

ahmetb · 2018-09-25T18:07:12Z

src/emailservice/Dockerfile

+
+# Install grpcio
+RUN python -m pip install --upgrade pip
+RUN pip install grpcio==${GRPC_PYTHON_VERSION} grpcio-tools==${GRPC_PYTHON_VERSION}


I recommend we just put these two inside requirements.txt. Not sure why we need to treat them any differently.

Can do. I originally left them in there to mimic the gRPC docker image

I think we should just move it to requirements.txt.

Also I recommend keeping the original grpc version. The 1.15 has introduced new methods in the health.proto that we haven't implemented (not sure if it's a problem in Python, but it broke the app in Go).

Moved to requirements.txt & rolled versions back to 1.12.1

src/emailservice/Dockerfile

ahmetb · 2018-09-25T18:07:51Z

src/emailservice/Dockerfile

-RUN pip3 install -r requirements.txt
+
+# Grab packages from builder
+COPY --from=builder /usr/local/lib/python2.7/ /usr/local/lib/python2.7/


Is this a legit thing to do? Seemed a bit scary to me :P

I've seen some articles that recommend doing either that or copying the cache over. The goal was to get rid of the dependencies gRPC needs at compile time but not at run time (linux-headers etc.)

ahmetb · 2018-09-25T18:08:10Z

src/emailservice/Dockerfile

@@ -1,27 +1,52 @@
+FROM python:2.7-alpine3.8 as base


It would be great if we could keep it python3.

👍 Will move back to Python 3

Have moved everything to python 3

ahmetb · 2018-09-25T18:08:53Z

~240 MB (down from ~ 1.31 GB)

…e is now ~271MB

…s to their older versions (1.12.1, 1.14.1, and 1.12.1 respectively)

orthros · 2018-09-25T19:21:32Z

Latest image size is now ~270MB, which is to be expected as the python 3 docker images are ~20MB larger than the 2.7 images (but both are still smaller than the grpc/python image)

From my terminal:

$ docker images | grep "python"
python                                                2.7-alpine3.8           b2bc7255b42c        13 days ago          58.8MB
python                                                3.7.0-alpine3.8         d30308ec4dc1        13 days ago          78.1MB
grpc/python                                           1.0                     248bf048ecd6        8 weeks ago          1.04GB

ahmetb · 2018-09-25T19:46:47Z

src/emailservice/requirements.txt

-grpcio-health-checking==1.15.0
-grpcio-tools==1.15.0
+grpcio==1.12.1
+grpcio-health-checking==1.14.1


ideally this also should be 1.12.1 but it's probably identical to 1.14.1.

Agreed. The previous requirements.txt had it at 1.14.1 so I kept it at that. I'll downgrade it and test

ahmetb · 2018-09-25T19:47:54Z

src/emailservice/Dockerfile

@@ -1,32 +1,25 @@
-FROM python:2.7-alpine3.8 as base
+FROM python:3.7.0-alpine3.8 as base


I recommend doing python:3-alpine (or :3.7-alpine) here so that:

we get python 3.x patches

we get stable alpine patches (not edge)

…logs

ahmetb · 2018-09-25T21:46:51Z

src/emailservice/Dockerfile

I recommend keeping this PYTHONUNBUFFERED=0 so that all processes (like pip) benefit from it.

…' argument

Reduce docker image for emailservice to ~240 MB (down from ~ 1.31 GB) Main application (`email_server.py`) now runs as python 2.7. Before we had both Python 2.7 and Python 3 installed in the image. Switched to using `python:2.7-alpine3.8` as the base image, and used multi-stage dockerfiles to keep dependencies minimal. Fixes GoogleCloudPlatform#49 From my shell: ``` $ docker build -t emailservice:dev . && docker run -it emailservice:dev Sending build context to Docker daemon 97.28kB Step 1/17 : FROM python:2.7-alpine3.8 as base ---> b2bc7255b42c Step 2/17 : FROM base as builder ---> b2bc7255b42c Step 3/17 : RUN apk add --update --no-cache gcc linux-headers make musl-dev python-dev g++ cairo-dev cairo openssl-dev gobject-introspection-dev ---> Using cache ---> 6daf3d9fe49a Step 4/17 : ENV GRPC_PYTHON_VERSION 1.15.0 ---> Using cache ---> 3e33d97d9580 Step 5/17 : RUN python -m pip install --upgrade pip ---> Using cache ---> e8fa3879c282 Step 6/17 : RUN pip install grpcio==${GRPC_PYTHON_VERSION} grpcio-tools==${GRPC_PYTHON_VERSION} ---> Using cache ---> c6fba7743eed Step 7/17 : COPY requirements.txt . ---> Using cache ---> 1f6b0a444980 Step 8/17 : RUN pip install -r requirements.txt ---> Using cache ---> 8cc0a7af6aa8 Step 9/17 : FROM base as final ---> b2bc7255b42c Step 10/17 : RUN GRPC_HEALTH_PROBE_VERSION=v0.1.0-alpha.1 && wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64 && chmod +x /bin/grpc_health_probe ---> Using cache ---> e954a0384081 Step 11/17 : ENV PYTHONUNBUFFERED=0 ---> Using cache ---> 64ece3d72a66 Step 12/17 : WORKDIR /email_server ---> Using cache ---> 27b34dc14492 Step 13/17 : COPY --from=builder /usr/local/lib/python2.7/ /usr/local/lib/python2.7/ ---> Using cache ---> 60035ec8dfd4 Step 14/17 : RUN apk add --no-cache libstdc++ ---> Using cache ---> 920be90c126e Step 15/17 : COPY . . ---> Using cache ---> 9541bed2d7a0 Step 16/17 : EXPOSE 8080 ---> Using cache ---> 48fbeaa852b9 Step 17/17 : ENTRYPOINT [ "python", "email_server.py" ] ---> Using cache ---> ff317770992d Successfully built ff317770992d Successfully tagged emailservice:dev starting the email service in dummy mode. listening on port: 8080 ```

orthros and others added 3 commits September 21, 2018 08:09

Merge pull request #1 from GoogleCloudPlatform/master

e56748f

Merge from Upstream

Merge pull request #2 from GoogleCloudPlatform/master

6a72e16

Merge from master

emailservice: Reduced Docker image size to ~240 MB. Used multi stage …

ce40475

…builds and alpine image

googlebot added cla: yes This human has signed the Contributor License Agreement. labels Sep 25, 2018

Fixed typeo in requirements.txt

982f47e

ahmetb reviewed Sep 25, 2018

View reviewed changes

src/emailservice/Dockerfile Show resolved Hide resolved

ahmetb reviewed Sep 25, 2018

View reviewed changes

orthros added 2 commits September 25, 2018 12:04

emailservice: Updated to python 3, streamlined pip install. Image siz…

ad529a6

…e is now ~271MB

emailservice: Reverted grpcio, grpcio-health-checking and grpcio-tool…

d3c546a

…s to their older versions (1.12.1, 1.14.1, and 1.12.1 respectively)

ahmetb reviewed Sep 25, 2018

View reviewed changes

orthros added 3 commits September 25, 2018 12:53

emailservice: Updated base image for dockerfile to python:3-alpine

2373440

emailservice: downgraded grpcio-health-checking to 1.12.1 from 1.14.1

dce3c26

emailservice: Updated ENTRYPOINT args to include '-u' for unbuffered …

65192b3

…logs

ahmetb reviewed Sep 25, 2018

View reviewed changes

emailservice: Added PYTHONUNBUFFERED=1 to environment and removed '-u…

e10efd0

…' argument

ahmetb approved these changes Sep 25, 2018

View reviewed changes

ahmetb merged commit 2924250 into GoogleCloudPlatform:master Sep 25, 2018

orthros deleted the emailservice-img-opt branch September 26, 2018 15:33

ahmetb mentioned this pull request Sep 27, 2018

image build time doubled after image size optimizations #58

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

emailservice Image Optimization #51

emailservice Image Optimization #51

orthros commented Sep 25, 2018

ahmetb Sep 25, 2018

orthros Sep 25, 2018

ahmetb Sep 25, 2018

orthros Sep 25, 2018

ahmetb Sep 25, 2018

orthros Sep 25, 2018

ahmetb Sep 25, 2018

orthros Sep 25, 2018

orthros Sep 25, 2018

ahmetb commented Sep 25, 2018 •

edited

Loading

orthros commented Sep 25, 2018

ahmetb Sep 25, 2018

orthros Sep 25, 2018

ahmetb Sep 25, 2018 •

edited

Loading

orthros Sep 25, 2018

ahmetb Sep 25, 2018

orthros Sep 25, 2018

		@@ -1,32 +1,25 @@
		FROM python:2.7-alpine3.8 as base
		FROM python:3.7.0-alpine3.8 as base

emailservice Image Optimization #51

emailservice Image Optimization #51

Conversation

orthros commented Sep 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmetb commented Sep 25, 2018 • edited Loading

orthros commented Sep 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmetb Sep 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmetb commented Sep 25, 2018 •

edited

Loading

ahmetb Sep 25, 2018 •

edited

Loading