Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support running all docker images locally #2698

Closed
wants to merge 4 commits into from
Closed

Support running all docker images locally #2698

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d134fa9
Support locally running Ubuntu 23.10 docker container
dennisschagt Mar 10, 2024
ebb21cd
Change mounting location to be outside of home
dennisschagt Mar 10, 2024
3659766
Change docker path to be in line with default on CI
dennisschagt Mar 10, 2024
4c81d14
Set default user for CI, specify `--user root` for local docker runs
dennisschagt Mar 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker/code-formatting-tools.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
# docker build \
# --tag=newsboat-code-formatting-tools \
# --file=docker/code-formatting-tools.dockerfile \
# docker
# .
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed? It doesn't look like Dockerfiles use anything outside the docker directory.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not strictly necessary, however, I prefer to keep these consistent between all of our docker files.
In the buildtools docker files, I've used a COPY command to pull in the new entrypoint.sh so there it is was necessary to update the path (cirrus seems to always specify the root directory of the git checkout, without option to override it so I did have to include the docker/ part in the COPY command)

#
# Run on your local files with:
#
Expand Down
10 changes: 10 additions & 0 deletions docker/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/bin/bash

USER_ID=${HOST_UID:-1000}
GROUP_ID=${HOST_GID:-1000}

echo "Starting with UID: $USER_ID, GID: $GROUP_ID"
usermod -u $USER_ID builder
groupmod -g $GROUP_ID builder

exec /usr/sbin/gosu builder "$@"
2 changes: 1 addition & 1 deletion docker/i18nspector.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# docker build \
# --tag=newsboat-i18nspector \
# --file=docker/i18nspector.dockerfile \
# docker
# .
#
# Run on your local files:
#
Expand Down
24 changes: 17 additions & 7 deletions docker/ubuntu_18.04-build-tools.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# docker build \
# --tag=newsboat-build-tools \
# --file=docker/ubuntu_18.04-build-tools.dockerfile \
# docker
# .
#
# Build with non-default compiler and Rust version:
#
Expand All @@ -26,7 +26,7 @@
# --build-arg cc=clang-7 \
# --build-arg cxx=clang++-7 \
# --build-arg rust_version=1.26.1 \
# docker
# .
#
# Before building in a container, run this to remove any binaries that you
# might've compiled on your host system (or in another container):
Expand All @@ -37,8 +37,10 @@
#
# docker run \
# --rm \
# --mount type=bind,source=$(pwd),target=/home/builder/src \
# --user $(id -u):$(id -g) \
# --mount type=bind,source=$(pwd),target=/src \
# --user root
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have to specify the user here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The approach taken in this PR requires the entrypoint script to update the user/group id of the builder user.
We need to be root to do that.
However, in the end, we want to run the build commands as the builder user.

The main issue is that cirrus does not expose a way to specify a user, so it will use the user as specified by the last USER statement in the docker file.
Cirrus also does not run the command specified by ENTRYPOINT.
That makes it necessary to specify the builder user at the end of the docker file.
However, locally, we do want to run the entrypoint script as root to fixup the user/group id in the container to match the user/group id of the host user.
That is why the --user root option is necessary at runtime.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, this solution feels far too convoluted, so I plan to try a different approach
(specifying the uid/gid at buildtime. If it works I will create a PR in a bit)

# -e HOST_UID=$(id -u) \
# -e HOST_GID=$(id -g) \
# newsboat-build-tools \
# make
#
Expand Down Expand Up @@ -69,14 +71,14 @@ RUN apt-get update \
&& apt-get install --assume-yes --no-install-recommends \
build-essential $cxx_package libsqlite3-dev libcurl4-openssl-dev libssl-dev \
libxml2-dev libstfl-dev libjson-c-dev libncursesw5-dev gettext git \
asciidoctor wget \
asciidoctor wget gosu \
&& apt-get autoremove \
&& apt-get clean

RUN addgroup --gid 1000 builder \
&& adduser --home /home/builder --uid 1000 --ingroup builder \
--disabled-password --shell /bin/bash builder \
&& mkdir -p /home/builder/src \
&& mkdir -p /src \
&& chown -R builder:builder /home/builder

RUN apt-get install locales \
Expand All @@ -89,7 +91,7 @@ ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

USER builder
WORKDIR /home/builder/src
WORKDIR /src

ARG rust_version=1.76.0

Expand All @@ -107,3 +109,11 @@ ARG cxx=g++-8

ENV CC=$cc
ENV CXX=$cxx

USER root
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

# Cirrus doesn't run the entrypoint script so we switch directly to the target user.
USER builder
22 changes: 16 additions & 6 deletions docker/ubuntu_18.04-i686.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# docker build \
# --tag=newsboat-i686-build-tools \
# --file=docker/ubuntu_18.04-i686.dockerfile \
# docker
# .
#
# Before building in a container, run this to remove any binaries that you
# might've compiled on your host system (or in another container):
Expand All @@ -16,8 +16,10 @@
#
# docker run \
# --rm \
# --mount type=bind,source=$(pwd),target=/home/builder/src \
# --user $(id -u):$(id -g) \
# --mount type=bind,source=$(pwd),target=/src \
# --user root
# -e HOST_UID=$(id -u) \
# -e HOST_GID=$(id -g) \
# newsboat-i686-build-tools \
# make
#
Expand Down Expand Up @@ -54,15 +56,15 @@ RUN apt-get update \
# `curl` would be enough for our needs, but it pulls in amd64 versions
# of libraries we use, interfering with the build environment. So
# `wget` it is.
wget \
wget gosu \
&& apt-get install --assume-yes --no-install-recommends asciidoctor \
&& apt-get autoremove \
&& apt-get clean

RUN addgroup --gid 1000 builder \
&& adduser --home /home/builder --uid 1000 --ingroup builder \
--disabled-password --shell /bin/bash builder \
&& mkdir -p /home/builder/src \
&& mkdir -p /src \
&& chown -R builder:builder /home/builder

RUN apt-get update \
Expand All @@ -76,7 +78,7 @@ ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

USER builder
WORKDIR /home/builder/src
WORKDIR /src

RUN wget -O $HOME/rustup.sh --secure-protocol=TLSv1_2 https://sh.rustup.rs \
&& chmod +x $HOME/rustup.sh \
Expand All @@ -86,3 +88,11 @@ RUN wget -O $HOME/rustup.sh --secure-protocol=TLSv1_2 https://sh.rustup.rs \
&& chmod a+w $HOME/.cargo

ENV HOME /home/builder

USER root
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

# Cirrus doesn't run the entrypoint script so we switch directly to the target user.
USER builder
24 changes: 17 additions & 7 deletions docker/ubuntu_20.04-build-tools.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# docker build \
# --tag=newsboat-build-tools \
# --file=docker/ubuntu_20.04-build-tools.dockerfile \
# docker
# .
#
# Build with non-default compiler and Rust version:
#
Expand All @@ -26,7 +26,7 @@
# --build-arg cc=clang-10 \
# --build-arg cxx=clang++-10 \
# --build-arg rust_version=1.40.0 \
# docker
# .
#
# Before building in a container, run this to remove any binaries that you
# might've compiled on your host system (or in another container):
Expand All @@ -37,8 +37,10 @@
#
# docker run \
# --rm \
# --mount type=bind,source=$(pwd),target=/home/builder/src \
# --user $(id -u):$(id -g) \
# --mount type=bind,source=$(pwd),target=/src \
# --user root
# -e HOST_UID=$(id -u) \
# -e HOST_GID=$(id -g) \
# newsboat-build-tools \
# make
#
Expand Down Expand Up @@ -69,14 +71,14 @@ RUN apt-get update \
&& apt-get install --assume-yes --no-install-recommends \
build-essential $cxx_package libsqlite3-dev libcurl4-openssl-dev libssl-dev \
libxml2-dev libstfl-dev libjson-c-dev libncursesw5-dev gettext git \
pkg-config zlib1g-dev asciidoctor wget \
pkg-config zlib1g-dev asciidoctor wget gosu \
&& apt-get autoremove \
&& apt-get clean

RUN addgroup --gid 1000 builder \
&& adduser --home /home/builder --uid 1000 --ingroup builder \
--disabled-password --shell /bin/bash builder \
&& mkdir -p /home/builder/src \
&& mkdir -p /src \
&& chown -R builder:builder /home/builder

RUN apt-get install locales \
Expand All @@ -89,7 +91,7 @@ ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

USER builder
WORKDIR /home/builder/src
WORKDIR /src

ARG rust_version=1.76.0

Expand All @@ -107,3 +109,11 @@ ARG cxx=g++-9

ENV CC=$cc
ENV CXX=$cxx

USER root
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

# Cirrus doesn't run the entrypoint script so we switch directly to the target user.
USER builder
24 changes: 17 additions & 7 deletions docker/ubuntu_22.04-build-tools.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# docker build \
# --tag=newsboat-build-tools \
# --file=docker/ubuntu_22.04-build-tools.dockerfile \
# docker
# .
#
# Build with non-default compiler and Rust version:
#
Expand All @@ -26,7 +26,7 @@
# --build-arg cc=clang-13 \
# --build-arg cxx=clang++-13 \
# --build-arg rust_version=1.40.0 \
# docker
# .
#
# Before building in a container, run this to remove any binaries that you
# might've compiled on your host system (or in another container):
Expand All @@ -37,8 +37,10 @@
#
# docker run \
# --rm \
# --mount type=bind,source=$(pwd),target=/home/builder/src \
# --user $(id -u):$(id -g) \
# --mount type=bind,source=$(pwd),target=/src \
# --user root
# -e HOST_UID=$(id -u) \
# -e HOST_GID=$(id -g) \
# newsboat-build-tools \
# make
#
Expand Down Expand Up @@ -70,14 +72,14 @@ RUN apt-get update \
&& apt-get install --assume-yes --no-install-recommends \
build-essential $cxx_package libsqlite3-dev libcurl4-openssl-dev libssl-dev \
libxml2-dev libstfl-dev libjson-c-dev libncursesw5-dev gettext git \
pkg-config zlib1g-dev asciidoctor \
pkg-config zlib1g-dev asciidoctor gosu \
&& apt-get autoremove \
&& apt-get clean

RUN addgroup --gid 1000 builder \
&& adduser --home /home/builder --uid 1000 --ingroup builder \
--disabled-password --shell /bin/bash builder \
&& mkdir -p /home/builder/src \
&& mkdir -p /src \
&& chown -R builder:builder /home/builder

RUN apt-get install locales \
Expand All @@ -90,7 +92,7 @@ ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

USER builder
WORKDIR /home/builder/src
WORKDIR /src

ARG rust_version=1.76.0

Expand All @@ -108,3 +110,11 @@ ARG cxx=g++-12

ENV CC=$cc
ENV CXX=$cxx

USER root
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh

# Cirrus doesn't run the entrypoint script so we switch directly to the target user.
USER builder
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
64 changes: 59 additions & 5 deletions docker/ubuntu_23.10-build-tools.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,52 @@
# - cxx -- C++ compiler to use. This gets copied into CXX environment variable.
# Default: g++-13
#
# For now, this Dockerfile can only be used in our CI.
# Build with defaults:
#
# docker build \
# --tag=newsboat-build-tools \
# --file=docker/ubuntu_23.10-build-tools.dockerfile \
# .
#
# Build with non-default compiler and Rust version:
#
# docker build \
# --tag=newsboat-build-tools \
# --file=docker/ubuntu_23.10-build-tools.dockerfile \
# --build-arg cxx_package=clang-16 \
# --build-arg cc=clang-16 \
# --build-arg cxx=clang++-16 \
# --build-arg rust_version=1.76.0 \
# .
#
# Before building in a container, run this to remove any binaries that you
# might've compiled on your host system (or in another container):
#
# make distclean
#
# Run on your local files:
#
# docker run \
# --rm \
# --mount type=bind,source=$(pwd),target=/src \
# --user root
# -e HOST_UID=$(id -u) \
# -e HOST_GID=$(id -g) \
# newsboat-build-tools \
# make
#
# To save bandwidth, and speed up the build slightly, share the host's Cargo
# cache with the container:
#
# mkdir -p ~/.cargo/registry
# docker run \
# --mount type=bind,source=$HOME/.cargo/registry,target=/home/builder/.cargo/registry \
# ... # the rest of the options
#
# If you want to build on the host again, run this to remove binary files
# compiled in the container:
#
# make distclean

FROM ubuntu:23.10

Expand All @@ -27,15 +72,16 @@ RUN apt-get update \
&& apt-get install --assume-yes --no-install-recommends \
build-essential $cxx_package libsqlite3-dev libcurl4-openssl-dev libssl-dev \
libxml2-dev libstfl-dev libjson-c-dev libncursesw5-dev gettext git \
pkg-config zlib1g-dev asciidoctor \
pkg-config zlib1g-dev asciidoctor gosu \
&& apt-get autoremove \
&& apt-get clean

RUN addgroup builder \
&& deluser ubuntu \
&& adduser --home /home/builder --ingroup builder \
--disabled-password --shell /bin/bash builder \
&& mkdir -p /home/builder/src \
&& chown -R builder:builder /home/builder
&& mkdir -p /src \
&& chown -R builder:builder /src

RUN apt-get install locales \
&& echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen \
Expand All @@ -47,7 +93,7 @@ ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

USER builder
WORKDIR /home/builder/src
WORKDIR /src

ARG rust_version=1.76.0

Expand All @@ -65,3 +111,11 @@ ARG cxx=g++-13

ENV CC=$cc
ENV CXX=$cxx

USER root
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

# Cirrus doesn't run the entrypoint script so we switch directly to the target user.
USER builder