[Bug]: Postgres fails to be preloaded

[michalbiesek]

Steps To Reproduce

Minimal Dockerfile which can be used to repro the issue

FROM postgres:14

COPY --from=cribl/scope:dev /usr/local/bin/scope /usr/local/bin/scope
RUN /usr/local/bin/scope extract /usr/local/lib/

# ENV LD_PRELOAD="/usr/local/lib/libscope.so"

USER postgres

ENV POSTGRES_USER LoremIpsum
ENV POSTGRES_PASSWORD LoremIpsum

CMD ["postgres"]

If we uncomment the LD_PRELOAD see that the container will not be able to start
To start the test:

docker build -t postgr:latest .
docker run -it postgr

1. With disable LD_PRELOAD log from docker run

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....2023-06-21 10:15:03.540 UTC [35] LOG:  starting PostgreSQL 14.8 (Debian 14.8-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2023-06-21 10:15:03.541 UTC [35] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-06-21 10:15:03.543 UTC [36] LOG:  database system was shut down at 2023-06-21 10:15:03 UTC
2023-06-21 10:15:03.546 UTC [35] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down....2023-06-21 10:15:03.742 UTC [35] LOG:  received fast shutdown request
2023-06-21 10:15:03.742 UTC [35] LOG:  aborting any active transactions
2023-06-21 10:15:03.744 UTC [35] LOG:  background worker "logical replication launcher" (PID 42) exited with exit code 1
2023-06-21 10:15:03.744 UTC [37] LOG:  shutting down
2023-06-21 10:15:03.751 UTC [35] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2023-06-21 10:15:03.879 UTC [1] LOG:  starting PostgreSQL 14.8 (Debian 14.8-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2023-06-21 10:15:03.880 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-06-21 10:15:03.880 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-06-21 10:15:03.881 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-06-21 10:15:03.884 UTC [50] LOG:  database system was shut down at 2023-06-21 10:15:03 UTC
2023-06-21 10:15:03.887 UTC [1] LOG:  database system is ready to accept connections

2. With enable LD_PRELOAD log from docker run

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....2023-06-21 11:42:00.675 UTC [35] LOG:  starting PostgreSQL 14.8 (Debian 14.8-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2023-06-21 11:42:00.676 UTC [35] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-06-21 11:42:00.680 UTC [38] LOG:  database system was shut down at 2023-06-21 11:42:00 UTC
2023-06-21 11:42:00.685 UTC [35] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down....2023-06-21 11:42:00.998 UTC [35] LOG:  received fast shutdown request
2023-06-21 11:42:00.999 UTC [35] LOG:  aborting any active transactions
2023-06-21 11:42:01.001 UTC [35] LOG:  background worker "logical replication launcher" (PID 47) exited with exit code 1
........................................................... failed
pg_ctl: server does not shut down

[michalbiesek]

Potential root cause:
SIGUSR2 seems to be used for communication between postmaster and backend https://www.postgresql.org/docs/7.0/signals.htm
With SCOPE_NO_SIGNAL=true in Dockerfile we can use LD_PRELOAD and postgres can start normally

[michalbiesek]

The work is done in #1538 .

QA instructions:
Two integration test were added to cover the feature:

• postgresql integration test
• extending musl and glibc integration test with scoping the example C application which employs SIGUSR2 signal withing an application handler.

If the reviewer wants to test it manually I would recommend to see the integration tests in the previously mentioned Pull Request