Improve handling of errors in init scripts docker-entrypoint-initdb.d

[olivierdalang]

Currently, scripts under docker-entrypoint-initdb.d are run with -v ON_ERROR_STOP=1.

This seems sensible, as we by default don't want a the DB to initialize to an indeterminate state with silently failing init scripts. However, even if the container stops, the volume is still initialized, so that on the next restart (which will happen silently in most use cases due to restart policies), the database will start without complaints, with a partially initialized database.

Suggested fixes :

• wipe the database if init scripts fail, so that a running container guarantees the database was successfully initialized
• add an option (env var) to run scripts with -v ON_ERROR_STOP=0 for users who still want to continue even if scripts errors (see Stop the container if running docker-entrypoint-initdb.d scripts error out #432 (comment)), in which case failing fast on errors is damaging (e.g. when including a partially compatible database dump)

[tianon]

A simple way to accomplish this to day is to use .sh drop-in files instead of .sql (so you can control the psql implementation yourself).

We've talked before about how we really ought to wipe out the entire data directory if the initialization doesn't succeed, but it feels really dangerous to do so. It's kind of a shame that containers never gained a way to be able to say "I'm failing and exiting, and should not be restarted because no amount of automatic restarting is going to fix what's wrong with me" 😞

[olivierdalang]

Thanks for the answer. Any hint or pointers to old discussions about why it would be dangerous ? The init script anyways only runs on a uninitialized db, doesn't it ? What is the use case for keeping a failed/partially initialized data directory ?

And yes, you're right about the custom .sh, but that's not as nice as just mounting a single sql using the base image.

[tianon]

Mostly because it's a user-supplied directory, and we don't necessarily know for certain that there won't be negative side-effects to doing so. It's a bit of a wild/extreme example, but it's "supported" for someone to have a .sh script which interacts with a remote service, and thus there's a side effect to completely deleting the half-initialized directory (imagine it generated some sort of ID, stored it in the database, and sent that to a separate service such that the user might prefer to try to salvage the half-initialized database rather than just discard it).

There are also users who are configuring PostgreSQL to explicitly ignore specific sub-directories of the volume they provide such that they can use it for other things (however misguided that may be), and wiping it out might wipe out more data than we intended.

Another example would be perhaps a slow, encrypted, or even remote filesystem such that deleting the half-initialized content is expensive in some way (computation, bandwidth, etc).

[olivierdalang]

What about the following strategy : the init script creates a flag (file) in the datadir to indicate that initialization is in progress. On successful completion, it deletes the flag. The entrypoint exits with error incomplete initialization if the flag exists on container startup.

Would that work ?

And what about running scripts -v ON_ERROR_STOP=0 ? It's indeed a bit a separate issue, should I open another one to discuss that ?

[tianon]

Sorry for the delay -- I would still suggest that -v ON_ERROR_STOP=0 should be implemented via a drop-in .sh file in the initialization directory.

For help implementing that or further discussion around customization, I would suggest trying a dedicated support forum, such as the Docker Community Forums, the Docker Community Slack, or Stack Overflow. 👍

[garymoon]

I can understand the reluctance to wipe the DB if an init script fails, but nevertheless the problem described in this issue still remains. I was briefly excited by this comment which implies the existence /always-initdb.d/, but alas.

---

If there was an additional init directory that ran on each startup instead of only post-init, the problem described in this issue could be worked around in a lot of cases with IF NOT EXISTS or similar.

In my case I'm creating an unprivileged user via bash script which aborts on undefined variables, but should that condition occur Postgres itself will simply continue on and the user is not created. When Postgres is restarted the situation will be no different, and the user will have quite a bit of debugging on their hands.

If there were an always-initdb.d I could use IF NOT EXISTS to create the missing user on subsequent runs after the user has corrected the environment.