Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(postgres): modify the docker base image as postgres:11-alpine #5

Merged
merged 8 commits into from Nov 28, 2018
Copy path View file
@@ -1,47 +1,40 @@
FROM postgres:11

ARG DEBIAN_FRONTEND=noninteractive
ARG BUILD_DEPS='gcc git libffi-dev libssl-dev python3-dev python3-pip python3-wheel'

RUN apt-get update && \
apt-get install -y --no-install-recommends \
$BUILD_DEPS \
gosu \
lzop \
libpq-dev \
pv \
python3 \
util-linux \
# swift package needs pkg_resources and setuptools
python3-pkg-resources \
python3-setuptools \
python3-pip && \
ln -sf /usr/bin/python3 /usr/bin/python && \
ln -sf /usr/bin/pip3 /usr/bin/pip

# setuptools from ubuntu archives is too old for googleapis-common-protos
RUN pip install --upgrade setuptools && \
pip install --disable-pip-version-check --no-cache-dir \
envdir==1.0.1 \
wal-e[aws,azure,google,swift]==1.1.0 \
gcloud==0.18.3 \
oauth2client==4.1.3 \
azure-storage==0.20.0

# cleanup
RUN apt-get purge -y --auto-remove $BUILD_DEPS && \
This conversation was marked as resolved by Cryptophobia

This comment has been minimized.

@till

till Nov 19, 2018

Member

No clean-up necessary on alpine?

Or maybe we should multi-stage this?

This comment has been minimized.

@duanhongyi

duanhongyi Nov 19, 2018

Author Member

see this command: apk del .build-deps

-_-

This comment has been minimized.

@Cryptophobia

Cryptophobia Nov 20, 2018

Member

Yes, this is 💯 🥇 !

apt-get autoremove -y && \
apt-get clean -y
FROM postgres:11-alpine
This conversation was marked as resolved by Cryptophobia

This comment has been minimized.

@till

till Nov 19, 2018

Member

Should we use a more specific tag? Like 11.0-alpine or 11.1-alpine? (No idea if Postgres knows semver.)

This comment has been minimized.

@duanhongyi

duanhongyi Nov 19, 2018

Author Member

The version after postgres10 changed the naming policy, so postgres11.x is backward compatible.

This comment has been minimized.

@Cryptophobia

Cryptophobia Nov 20, 2018

Member

Sounds like postgres:11-alpine will be alright then.

This comment has been minimized.

@Cryptophobia

Cryptophobia Nov 20, 2018

Member

Should we use a 11.x version that is higher than 11.0 then or does 11 have the fixes in the later versions of 11.x?

This comment has been minimized.

@duanhongyi

duanhongyi Nov 20, 2018

Author Member

The exact point postgres:11-alpine can be, it is just a link, pointing to the latest postgres:11.x-alpine. In this way, we will use the latest image every time.

see:
https://hub.docker.com/r/library/postgres/tags/

This comment has been minimized.

@Cryptophobia

Cryptophobia Nov 25, 2018

Member

Good 👍


ENV WALE_LOG_DESTINATION stderr
ENV WALE_ENVDIR /etc/wal-e.d/env

RUN mkdir -p $WALE_ENVDIR \
&& echo 'http://dl-cdn.alpinelinux.org/alpine/v3.5/main' >> /etc/apk/repositories \
&& apk add --update --virtual .build-deps \
git \
build-base \
libffi-dev \
openssl-dev \
python3-dev=3.5.6-r0 \
This conversation was marked as resolved by duanhongyi

This comment has been minimized.

@till

till Nov 19, 2018

Member

Seems oddly specific, any reason? I know this is likely to break in a few weeks when this package version is gone. ;)

This comment has been minimized.

@duanhongyi

duanhongyi Nov 19, 2018

Author Member

Wal-e does not support python3.6, so I used an old version of python3-dev=3.5.6-r0.

echo 'http://dl-cdn.alpinelinux.org/alpine/v3.5/main' >> /etc/apk/repositories

see:
wal-e/wal-e#322

This comment has been minimized.

@duanhongyi

duanhongyi Nov 19, 2018

Author Member

Fixing the backbone that has been merged into wal-e, I think the next official version should fix this problem.

This comment has been minimized.

@Cryptophobia

Cryptophobia Nov 20, 2018

Member

@duanhongyi @till When is the fix for wal-e released and what is the plan, should we update to python3-dev 3.6 at that time?

This comment has been minimized.

@duanhongyi

duanhongyi Nov 20, 2018

Author Member

Yes, once wal-e releases the latest version, I will update this component. but it seems that wal-e has not released a new version for a long time.

This comment has been minimized.

@duanhongyi

duanhongyi Nov 20, 2018

Author Member

Wal-g looks good, but now it only supports S3.

This comment has been minimized.

@till

till Nov 20, 2018

Member

I am lol'ing a little with these projects.

For reference:
https://github.com/wal-g/wal-g

I think in comparison to wal-e, this is also actively maintained. Not sure if we need to worry much about supporting custom APIs when everyone aims to be S3-compatible right now?

This comment has been minimized.

@till

till Nov 20, 2018

Member

Btw, KubeDB also supports wal-g, maybe we should move towards that. ;)

This comment has been minimized.

@duanhongyi

duanhongyi Nov 20, 2018

Author Member

Btw, KubeDB also supports wal-g, maybe we should move towards that. ;)

Can we discuss this? If our team confirms using wal-g, it's one of my favorite results.

This comment has been minimized.

@duanhongyi

duanhongyi Nov 20, 2018

Author Member

But I'm not sure if anyone used openstack-swift, azure or GCS before.

linux-headers \
&& apk add \
lzo \
pv \
util-linux \
ca-certificates \
python3=3.5.6-r0 \
&& pip3 install --upgrade pip setuptools \
&& pip install --disable-pip-version-check --no-cache-dir \
psycopg2-binary==2.7.6.1 \
envdir==1.0.1 \
wal-e[aws,azure,google,swift]==1.1.0 \
gcloud==0.18.3 \
oauth2client==4.1.3 \
azure-storage==0.20.0 \
&& apk del .build-deps \
This conversation was marked as resolved by Cryptophobia

This comment has been minimized.

@till

till Nov 19, 2018

Member

👍

&& rm -rf /var/cache/apk/*

COPY rootfs /
ENV WALE_ENVDIR=/etc/wal-e.d/env
RUN mkdir -p $WALE_ENVDIR

ARG PATCH_CMD="python3 /patcher-script.py"
RUN $PATCH_CMD file /bin/create_bucket /patcher-script.d/patch_boto_s3.py
RUN $PATCH_CMD file /usr/local/bin/wal-e /patcher-script.d/patch_boto_s3.py
RUN $PATCH_CMD module wal_e.cmd /patcher-script.d/patch_boto_s3.py
RUN $PATCH_CMD module wal_e.worker.worker_util /patcher-script.d/patch_wal_e_s3.py


CMD ["/docker-entrypoint.sh", "postgres"]
EXPOSE 5432
@@ -41,11 +41,15 @@ spec:
value: "{{.Values.global.storage}}"
- name: PGCTLTIMEOUT
value: "{{.Values.postgres.timeout}}"
{{- if eq .Values.global.storage "s3" }}
- name: S3_SSE
value: "{{.Values.s3.use_sse}}"
{{- end}}
lifecycle:
preStop:
exec:
command:
- gosu
- su-exec
This conversation was marked as resolved by Cryptophobia

This comment has been minimized.

@till

till Nov 19, 2018

Member

This is funny, it's a 10kb version of gosu? :D

This comment has been minimized.

@duanhongyi

duanhongyi Nov 19, 2018

Author Member

Yes, su-exec can replace Gosu, which is the tool Alpine brings.

This comment has been minimized.

@Cryptophobia

Cryptophobia Nov 20, 2018

Member

Very nice!

- postgres
- do_backup
readinessProbe:
Copy path View file
@@ -1,4 +1,4 @@
#!/usr/bin/env python
#!/usr/bin/env python3

import os

Copy path View file
@@ -5,6 +5,7 @@ export BACKUPS_TO_RETAIN=${BACKUPS_TO_RETAIN:-5}
echo "Performing a base backup..."
if [[ -f "$PGDATA/recovery.conf" ]] ; then
echo "Database is currently recovering from a backup. Aborting"
sleep 9
else
# perform a backup
envdir "$WALE_ENVDIR" wal-e backup-push "$PGDATA"
Copy path View file
@@ -8,4 +8,4 @@ if [[ -f "$PGDATA/recovery.conf" ]]; then
exit 1
fi

gosu postgres pg_ctl status
su-exec postgres pg_ctl status
@@ -17,6 +17,11 @@ if [[ "$DATABASE_STORAGE" == "s3" || "$DATABASE_STORAGE" == "minio" ]]; then
else
echo "https+path://s3-${AWS_REGION}.amazonaws.com:443" > WALE_S3_ENDPOINT
fi
if [[ $S3_SSE ]]; then
echo $S3_SSE > WALE_S3_SSE
else
echo "false" > WALE_S3_SSE
fi
else
AWS_REGION="us-east-1"
BUCKET_NAME="dbwal"
@@ -13,13 +13,13 @@ chmod 0700 "$PGDATA"

# reboot the server for wal_level to be set before backing up
echo "Rebooting postgres to enable archive mode"
gosu postgres pg_ctl -D "$PGDATA" -w restart
su-exec postgres pg_ctl -D "$PGDATA" -w restart

# check if there are any backups -- if so, let's restore
# we could probably do better than just testing number of lines -- one line is just a heading, meaning no backups
if [[ $(envdir "$WALE_ENVDIR" wal-e --terse backup-list | wc -l) -gt "1" ]]; then
echo "Found backups. Restoring from backup..."
gosu postgres pg_ctl -D "$PGDATA" -w stop
su-exec postgres pg_ctl -D "$PGDATA" -w stop
rm -rf "$PGDATA/*"
envdir "$WALE_ENVDIR" wal-e backup-fetch "$PGDATA" LATEST
cat << EOF > "$PGDATA/postgresql.conf"
@@ -50,20 +50,11 @@ EOF
echo "restore_command = 'envdir /etc/wal-e.d/env wal-e wal-fetch \"%f\" \"%p\"'" >> "$PGDATA/recovery.conf"
chown -R postgres:postgres "$PGDATA"
chmod 0700 "$PGDATA"
gosu postgres pg_ctl -D "$PGDATA" \
su-exec postgres pg_ctl -D "$PGDATA" \
-o "-c listen_addresses=''" \
-w start

echo "Waiting for recovery completion..."
This conversation was marked as resolved by Cryptophobia

This comment has been minimized.

@till

till Nov 19, 2018

Member

Can you add a comment or two how this changed? It seems like maybe postgres got smarter and maybe provides a lock file for the backup process, or ... — I am not an expert.

This comment has been minimized.

@duanhongyi

duanhongyi Nov 19, 2018

Author Member

When the original script was resumed, the push was unified once. In fact, this operation is unnecessary. Because there is another lock in do_backup.

See:
https://github.com/duanhongyi/postgres/blob/master/rootfs/bin/do_backup

This comment has been minimized.

@till

till Nov 19, 2018

Member

Now that makes sense! Thank you!

This comment has been minimized.

@Cryptophobia

Cryptophobia Nov 20, 2018

Member

Sounds good, thanks for the explanation and information.

while [ ! -f "$PGDATA/recovery.done" ]
do
sleep 2
done
fi

echo "Performing an initial backup..."
gosu postgres envdir "$WALE_ENVDIR" wal-e backup-push "$PGDATA"

# ensure $PGDATA has the right permissions
chown -R postgres:postgres "$PGDATA"
chmod 0700 "$PGDATA"
@@ -1,4 +1,4 @@
#!/usr/bin/env bash

# Run periodic backups in the background
gosu postgres backup &
su-exec postgres backup &
Copy path View file
@@ -23,7 +23,7 @@ if [ "$1" = 'postgres' ]; then

# look specifically for PG_VERSION, as it is expected in the DB dir
if [ ! -s "$PGDATA/PG_VERSION" ]; then
gosu postgres initdb
su-exec postgres initdb

# check password first so we can output the warning before postgres
# messes it up
@@ -54,7 +54,7 @@ if [ "$1" = 'postgres' ]; then

# internal start of server in order to allow set-up using psql-client
# does not listen on TCP/IP and waits until start finishes
gosu postgres pg_ctl -D "$PGDATA" \
su-exec postgres pg_ctl -D "$PGDATA" \
-o "-c listen_addresses=''" \
-w start

@@ -94,15 +94,15 @@ if [ "$1" = 'postgres' ]; then
echo
done

gosu postgres pg_ctl -D "$PGDATA" -m fast -w stop
su-exec postgres pg_ctl -D "$PGDATA" -m fast -w stop
set_listen_addresses '*'

echo
echo 'PostgreSQL init process complete; ready for start up.'
echo
fi

exec gosu postgres "$@"
exec su-exec postgres "$@"
fi

exec "$@"
@@ -7,10 +7,10 @@ def wrap_uri_put_file(creds, uri, fp, content_type=None, conn=None):
k = s3_util._uri_to_key(creds, uri, conn=conn)
if content_type is not None:
k.content_type = content_type
encrypt_key = False
if os.getenv('DATABASE_STORAGE') == 's3':
encrypt_key=True
else:
encrypt_key=False
if os.getenv('WALE_S3_SSE', 'false') == 'true':
encrypt_key = True
k.set_contents_from_file(fp, encrypt_key=encrypt_key)
return k
s3.uri_put_file = wrap_uri_put_file
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.