incompatible change when switching from bitnami to OSI postgresql

[phil9909]

Summary

This commit 44e0688 switched the cf-db postgresql image from bitnami/postgresql to relintdockerhubpushbot/cf-for-k8s-postgres. This corrupts the data on the volume when upgrading.

Reproduction Steps

Can be reproduced in isolation

OLD_IMAGE=bitnami/postgresql@sha256:9762d9a80b90a5efe299d4848057ac5c45fb384570b36f60aad38fe2b1704bd6
NEW_IMAGE=relintdockerhubpushbot/cf-for-k8s-postgres@sha256:e7dd977c4de19a3848c16843f7cd724ad44141433c2460fc1297bab8b70e414c

# Start the old image
docker run --rm -v pgtest:/bitnami/postgresql -e POSTGRES_PASSWORD=test -e PGDATA=/bitnami/postgresql/data -e POSTGRESQL_VOLUME_DIR=/bitnami/postgresql $OLD_IMAGE
# press [CTRL] - [C] after "database system is ready to accept connections" is printed

# Start the new image
docker run --rm -v pgtest:/bitnami/postgresql -e POSTGRES_PASSWORD=test -e PGDATA=/bitnami/postgresql/data -e POSTGRESQL_VOLUME_DIR=/bitnami/postgresql $NEW_IMAGE

# cleanup
docker volume rm pgtest

Starting the new image will print

PostgreSQL Database directory appears to contain a database; Skipping initialization

postgres: could not access the server configuration file "/bitnami/postgresql/data/postgresql.conf": No such file or directory

After the new image has been started (and crashed) once you can also no longer start the old image:

postgresql 08:23:11.27 INFO  ==> Initializing PostgreSQL database...
chmod: changing permissions of '/bitnami/postgresql/data': Operation not permitted
postgresql 08:23:11.30 WARN  ==> Lack of permissions on data directory!
chmod: changing permissions of '/bitnami/postgresql/data': Operation not permitted
postgresql 08:23:11.31 WARN  ==> Lack of permissions on data directory!
postgresql 08:23:11.31 INFO  ==> postgresql.conf file not detected. Generating it...
postgresql 08:23:11.33 INFO  ==> pg_hba.conf file not detected. Generating it...
postgresql 08:23:11.34 INFO  ==> Generating local authentication configuration
ls: cannot open directory '/bitnami/postgresql/data': Permission denied
postgresql 08:23:11.39 INFO  ==> Stopping PostgreSQL...

starting and stopping the old image or new image multiple times is no problem.

See also

https://cloudfoundry.slack.com/archives/CH9LF6V1P/p1590670391438200

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/173076090

The labels on this github issue will be updated when the story is started.

[acosta11]

For visibility, @davewalter and I did a quick investigation and found:

1. We're using the official docker image inside the osl compliant image
2. According to the bitnami postgresql readme when using the bitnami helm chart with the official docker image instead of the bitnami-provided image, we need to update the mounting configuration

From chart version 4.0.0, it is possible to use this chart with the Docker Official PostgreSQL image. Besides specifying the new Docker repository and tag, it is important to modify the PostgreSQL data directory and volume mount point. Basically, the PostgreSQL data dir cannot be the mount point directly, it has to be a subdirectory.

[davewalter]

On further investigation, we found that there appears to be an incompatibility between the bitnami image and the "official" docker image. Specifically, the bitnami image uses separate directories for the Postgres configuration (stored in /opt/bitnami/postgresql/conf) and data (stored in /bitnami/postgresql/data). By comparison, the "official" docker image stores all of it's configuration and data in a single directory (stored in /bitnami/postgresql/data when injected into the same K8s templates). This leads to the problem described above, where Postgres cannot find its configuration file on startup.

As far as we can tell, the decision to use a different directory for the configuration file is baked into the bitnami image startup script, and is therefore not something we can override via the chart.

[lisamburns]

We have reverted this commit for now. That should resolve this issue
a26b241

We have this story to investigate how to handle the postgres issue longer term.
https://www.pivotaltracker.com/story/show/173114137

CC @davewalter

[cppforlife]

if this is something team is interested in fixing, one solution might be to add an initContainer that moves data into expected locations (though be careful about making it idempotent).

[Syerram]

Closing this issue since we fixed the issue.