Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add black Dockerfile #1916

Merged
merged 4 commits into from Apr 8, 2021
Merged

Add black Dockerfile #1916

merged 4 commits into from Apr 8, 2021

Conversation

cooperlees
Copy link
Collaborator

  • Build a black container based on python3-slim
  • Test: docker build --network-host --tag black .
cooper-mbp1:black cooper$ docker image ls
REPOSITORY                                         TAG            IMAGE ID       CREATED              SIZE
black                                              latest         0c7636402ac2   4 seconds ago        332MB

Addresses part of #1914

- Build a `black` container based on python3-slim
- Test: `docker build --network-host --tag black .`
```console
cooper-mbp1:black cooper$ docker image ls
REPOSITORY                                         TAG            IMAGE ID       CREATED              SIZE
black                                              latest         0c7636402ac2   4 seconds ago        332MB
```

Addresses part of #1914
@cooperlees cooperlees added the skip news Pull requests that don't need a changelog entry. label Apr 3, 2021
@cooperlees
Copy link
Collaborator Author

Retested and rebased.

crl-m1:black cooper$ docker image ls
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
black        latest    2e7b11beab8a   19 seconds ago   237MB
  • Uploaded it's only 94mb (larger than I hoped)

https://hub.docker.com/repository/docker/cooperlees/black/tags?page=1&ordering=last_updated

@cooperlees
Copy link
Collaborator Author

When we land this I will look to automate a docker push + tagging for people to use (similar to bandersnatch)

@ichard26 ichard26 removed their request for review April 3, 2021 16:44
@ichard26
Copy link
Collaborator

ichard26 commented Apr 3, 2021

I'm sorry but I won't be able to review this since I have never used docker ... tip-toes away awkwardly

Copy link

@gliptak gliptak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to decrease layer size consider multi-stage setup or setup like

FROM python:3-slim

RUN mkdir /src
COPY . /src/
RUN cd /src && pip install --upgrade pip setuptools wheel && apt update && apt install -y git && pip install --no-cache-dir .[colorama,d] && rm -rf /src && apt remove -y git && apt autoremove -y && rm -rf /var/lib/apt/lists/*

CMD ["black"]

Copy link
Collaborator

@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks reasonable to me but I have no real expertise on Docker.

Dockerfile Outdated
@@ -0,0 +1,10 @@
FROM python:3-slim

RUN pip install --upgrade pip setuptools wheel
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps also use --no-cache-dir here?

- Combine the commands all into 1 RUN to down the size
- Get to 65.98 MB image size now: https://hub.docker.com/repository/docker/cooperlees/black/tags?page=1&ordering=last_updated
Dockerfile Outdated
&& pip install --no-cache-dir .[colorama,d] \
&& rm -rf /src \
&& apt remove -y git \
&& apt autoremove -y
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add && rm -rf /var/lib/apt/lists/*

@cooperlees cooperlees mentioned this pull request Apr 4, 2021
@TBBle
Copy link
Contributor

TBBle commented Apr 5, 2021

I had a quick look at the current state, and apart from /usr/local/lib/python3.9, here's the other files in the resulting image.

-rw-------  0 0      0           0 Mar 29 11:00 etc/.pwd.lock
lrwxrwxrwx  0 0      0           0 Apr 05 11:58 etc/alternatives/pager -> /bin/more
-rw-------  0 0      0           0 Apr 05 11:58 etc/alternatives/.wh.pager.1.gz
-rwxr-xr-x  0 0      0           0 Jan 01  1970 etc/bash_completion.d/.wh..wh..opq
-rw-r--r--  0 0      0         439 Apr 20  2020 etc/bash_completion.d/git-prompt
-rw-r--r--  0 0      0         457 Apr 05 11:58 etc/group
-rw-r--r--  0 0      0         446 Mar 29 11:00 etc/group-
-rw-r-----  0 0      42        382 Apr 05 11:58 etc/gshadow
-rw-r-----  0 0      42        374 Mar 29 11:00 etc/gshadow-
-rw-r--r--  0 0      0        7727 Apr 05 11:58 etc/ld.so.cache
-rwxr-xr-x  0 0      0           0 Jan 01  1970 etc/ldap/.wh..wh..opq
-rw-r--r--  0 0      0         332 Feb 15 05:32 etc/ldap/ldap.conf
-rwxr-xr-x  0 0      0           0 Jan 01  1970 etc/logcheck/.wh..wh..opq
-rw-r--r--  0 0      0          79 Dec 20  2019 etc/logcheck/ignore.d.server/libsasl2-modules
-rwxr-xr-x  0 0      0           0 Jan 01  1970 etc/perl/.wh..wh..opq
-rw-r--r--  0 0      0         611 Jul 22  2020 etc/perl/Net/libnet.cfg
-rwxr-xr-x  0 0      0           0 Jan 01  1970 etc/ssh/.wh..wh..opq
-rw-r--r--  0 0      0        1580 Feb 01  2020 etc/ssh/ssh_config
-rwxr-xr-x  0 0      0           0 Jan 01  1970 root/.cache/.wh..wh..opq
-rw-------  0 0      0       28690 Apr 05 11:58 root/.cache/pip/http/0/2/c/5/c/02c5c8626047699b8cd071cba977743b1b46f86acc2ac49acacd1fa0
-rw-------  0 0      0      100405 Apr 05 11:58 root/.cache/pip/http/0/4/1/8/c/0418c83b80f7f7bfaec2738bfbbee53d2c1562196c0781702f6eddc8
-rw-------  0 0      0       29376 Apr 05 11:58 root/.cache/pip/http/1/9/3/d/2/193d2b14cf4cd704ac983a718ec8edac2afad38c2e26f3f739d3d911
-rw-------  0 0      0       11097 Apr 05 11:58 root/.cache/pip/http/4/d/2/7/2/4d272e6453941ce8b0a37a02cdb1685fc612c33441fa74691fb40656
-rw-------  0 0      0      785896 Apr 05 11:58 root/.cache/pip/http/6/6/e/d/5/66ed5a0988296cfe4c54eb3e06e283b4ad3514d56b4c082f7bdc4722
-rw-------  0 0      0       35863 Apr 05 11:58 root/.cache/pip/http/8/f/2/3/3/8f233a7ef98244b58a65bda7b4835617661195d1b46924f356c68d3c
-rw-------  0 0      0       16848 Apr 05 11:58 root/.cache/pip/http/a/1/9/5/3/a19537d3cf37c122db841d6fe4cd322bc10d1a558bb00d146b85cb9a
-rw-------  0 0      0          80 Apr 05 11:58 root/.cache/pip/selfcheck/fe300af6f7d708c14827daac3afc81fbb8306b73de8dd6e3f1f8ea3b
-rw-------  0 0      0           0 Apr 05 11:58 .wh.src
-rwxr-xr-x  0 0      0         225 Apr 05 11:58 usr/local/bin/black
-rwxr-xr-x  0 0      0         220 Apr 05 11:58 usr/local/bin/black-primer
-rwxr-xr-x  0 0      0         226 Apr 05 11:58 usr/local/bin/blackd
-rwxr-xr-x  0 0      0         226 Apr 05 11:58 usr/local/bin/chardetect
-rw-r-----  0 0      0           0 Mar 31 22:54 var/cache/apt/archives/lock
-rw-r--r--  0 0      0       16421 Apr 03 08:13 var/cache/debconf/config.dat
-rw-------  0 0      0           0 Mar 29 11:00 var/cache/debconf/passwords.dat
-rw-r--r--  0 0      0      794492 Apr 03 08:13 var/cache/debconf/templates.dat
-rw-------  0 0      0        5385 Apr 05 11:58 var/cache/ldconfig/aux-cache
-rw-r--r--  0 0      0        4502 Apr 05 11:58 var/lib/apt/extended_states
-rw-r--r--  0 0      0         107 Apr 05 11:58 var/lib/dpkg/alternatives/pager
-rw-r--r--  0 0      0          62 Apr 05 11:58 var/lib/dpkg/info/git.list
-rwxr-xr-x  0 0      0         756 Apr 20  2020 var/lib/dpkg/info/git.postrm
-rw-r--r--  0 0      0           0 Apr 05 11:58 var/lib/dpkg/info/less.list
-rwxr-xr-x  0 0      0         183 May 07  2018 var/lib/dpkg/info/less.postrm
-rw-r--r--  0 0      0           0 Apr 05 11:58 var/lib/dpkg/info/libgssapi-krb5-2:amd64.list
-rwxr-xr-x  0 0      0          96 Nov 20 01:50 var/lib/dpkg/info/libgssapi-krb5-2:amd64.postrm
-rw-r--r--  0 0      0          35 Apr 05 11:58 var/lib/dpkg/info/libldap-common.list
-rw-r--r--  0 0      0          96 Apr 05 11:58 var/lib/dpkg/info/libsasl2-modules:amd64.list
-rw-r--r--  0 0      0          34 Apr 05 11:58 var/lib/dpkg/info/openssh-client.list
-rwxr-xr-x  0 0      0         685 Feb 01  2020 var/lib/dpkg/info/openssh-client.postrm
-rw-r--r--  0 0      0          54 Apr 05 11:58 var/lib/dpkg/info/perl.list
-rwxr-xr-x  0 0      0         115 Jul 22  2020 var/lib/dpkg/info/perl.postrm
-rw-r-----  0 0      0           0 Apr 05 11:58 var/lib/dpkg/lock
-rw-r-----  0 0      0           0 Mar 29 11:00 var/lib/dpkg/lock-frontend
-rw-r--r--  0 0      0       93757 Apr 05 11:58 var/lib/dpkg/status
-rw-r--r--  0 0      0       93791 Apr 05 11:58 var/lib/dpkg/status-old
-rw-------  0 0      0           0 Apr 05 11:58 var/lib/dpkg/triggers/Lock
-rw-r--r--  0 0      0        5719 Apr 05 11:58 var/log/alternatives.log
-rw-r--r--  0 0      0        7844 Apr 05 11:58 var/log/apt/eipp.log.xz
-rw-r--r--  0 0      0       14133 Apr 05 11:58 var/log/apt/history.log
-rw-r-----  0 0      4       58633 Apr 05 11:58 var/log/apt/term.log
-rw-r--r--  0 0      0      155570 Apr 05 11:58 var/log/dpkg.log

I shortened this list by adding --purge to both of apt remove and apt autoremove, and adding --no-install-recommends to the apt install call, adding --disable-pip-version-check to pip install and deleting /root/.cache afterwards, resulting in a 1MB-smaller image tarball.

That also cut 10s off the build by reducing the amount of stuff apt had to download, but the vast majority of time spent was in installing git, and it's not obvious why we need git installed here. Is it just a pip issue?

Deleting `/root/.cache`?

For some reason, pip is ignoring --no-cache-dir at some points in its lifecycle, leaving the following files lying around:

-rwxr-xr-x  0 0      0           0 Jan 01  1970 root/.cache/.wh..wh..opq
-rw-------  0 0      0       28691 Apr 05 12:44 root/.cache/pip/http/0/2/c/5/c/02c5c8626047699b8cd071cba977743b1b46f86acc2ac49acacd1fa0
-rw-------  0 0      0      100405 Apr 05 12:44 root/.cache/pip/http/0/4/1/8/c/0418c83b80f7f7bfaec2738bfbbee53d2c1562196c0781702f6eddc8
-rw-------  0 0      0       29376 Apr 05 12:44 root/.cache/pip/http/1/9/3/d/2/193d2b14cf4cd704ac983a718ec8edac2afad38c2e26f3f739d3d911
-rw-------  0 0      0       11097 Apr 05 12:44 root/.cache/pip/http/4/d/2/7/2/4d272e6453941ce8b0a37a02cdb1685fc612c33441fa74691fb40656
-rw-------  0 0      0      785896 Apr 05 12:44 root/.cache/pip/http/6/6/e/d/5/66ed5a0988296cfe4c54eb3e06e283b4ad3514d56b4c082f7bdc4722
-rw-------  0 0      0       35863 Apr 05 12:44 root/.cache/pip/http/8/f/2/3/3/8f233a7ef98244b58a65bda7b4835617661195d1b46924f356c68d3c
-rw-------  0 0      0       16848 Apr 05 12:44 root/.cache/pip/http/a/1/9/5/3/a19537d3cf37c122db841d6fe4cd322bc10d1a558bb00d146b85cb9a
-rw-------  0 0      0          80 Apr 05 12:44 root/.cache/pip/selfcheck/fe300af6f7d708c14827daac3afc81fbb8306b73de8dd6e3f1f8ea3b

root/.cache/.wh..wh..opq is marking that anything else in that directory should be removed, i.e. it was created in this layer.

I initially thought this would be fixed by --disable-pip-version-check, but it wasn't, so either those files are being fetched by something else that ignores --no-cache-dir, or --disable-pip-version-check skips the check, but doesn't prevent the requirements for the self-check being fetched first, and is ignoring --no-cache-dir there.


Resulting layer tarstream non-directory entries:

-rw-r--r--  0 0      0        7727 Apr 05 12:55 etc/ld.so.cache
-rw-------  0 0      0           0 Apr 05 12:55 .wh.src
-rwxr-xr-x  0 0      0         225 Apr 05 12:55 usr/local/bin/black
-rwxr-xr-x  0 0      0         220 Apr 05 12:55 usr/local/bin/black-primer
-rwxr-xr-x  0 0      0         226 Apr 05 12:55 usr/local/bin/blackd
-rwxr-xr-x  0 0      0         226 Apr 05 12:55 usr/local/bin/chardetect
-rw-r-----  0 0      0           0 Mar 31 22:54 var/cache/apt/archives/lock
-rw-r--r--  0 0      0       16421 Apr 03 08:13 var/cache/debconf/config.dat
-rw-------  0 0      0           0 Mar 29 11:00 var/cache/debconf/passwords.dat
-rw-r--r--  0 0      0      794492 Apr 03 08:13 var/cache/debconf/templates.dat
-rw-------  0 0      0        5385 Apr 05 12:55 var/cache/ldconfig/aux-cache
-rw-r--r--  0 0      0        4502 Apr 05 12:55 var/lib/apt/extended_states
-rw-r-----  0 0      0           0 Apr 05 12:55 var/lib/dpkg/lock
-rw-r-----  0 0      0           0 Mar 29 11:00 var/lib/dpkg/lock-frontend
-rw-r--r--  0 0      0       82835 Apr 05 12:55 var/lib/dpkg/status
-rw-r--r--  0 0      0       82835 Apr 05 12:55 var/lib/dpkg/status-old
-rw-------  0 0      0           0 Apr 05 12:55 var/lib/dpkg/triggers/Lock
-rw-r--r--  0 0      0        7360 Apr 05 12:55 var/log/apt/eipp.log.xz
-rw-r--r--  0 0      0       12923 Apr 05 12:55 var/log/apt/history.log
-rw-r-----  0 0      4       55867 Apr 05 12:55 var/log/apt/term.log
-rw-r--r--  0 0      0      142577 Apr 05 12:55 var/log/dpkg.log

I reckon that's as-minimal a layer as one could get from the python:3-slim base image. At this point, all the changes in /var/ and /etc are updating existing files.

Resulting Dockerfile:

FROM python:3-slim

RUN mkdir /src
COPY . /src/
RUN pip install --no-cache-dir --disable-pip-version-check --upgrade pip setuptools wheel \
    && apt update && apt install --no-install-recommends -y git \
    && cd /src \
    && pip install --no-cache-dir --disable-pip-version-check .[colorama,d] \
    && rm -rf /src \
    && apt remove --purge -y git \
    && apt autoremove --purge -y \
    && rm -rf /var/lib/apt/lists/* \
    && rm -rf /root/.cache

CMD ["black"]

@cooperlees
Copy link
Collaborator Author

Ok, I feel I've waited long enough and had enough help from people. I'm going to merge this.

@cooperlees cooperlees merged commit 8e0803e into master Apr 8, 2021
@cooperlees cooperlees deleted the docker branch April 8, 2021 04:13
@ichard26 ichard26 removed the request for review from ambv April 11, 2021 23:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skip news Pull requests that don't need a changelog entry.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants