Bus error during database system initialization

[qingguee]

Hi,

As there is is a K8s issue about huge pages, we turned off huge page in our installation. But we still meet bus error. Could you help?
I don't understand why huge_pages=off is not working.

2020-02-17 06:33:21,146 INFO: --------------------------------------------------------------------------------
2020-02-17 06:33:21,146 INFO: Patroni logging start...
2020-02-17 06:33:21,146 INFO: --------------------------------------------------------------------------------
2020-02-17 06:33:21,362 INFO: Reaped pid=32, exit status=0
2020-02-17 06:33:21,579 INFO: postgres connection_string is postgres://10.233.90.72:5432/postgres
2020-02-17 06:33:21,601 INFO: Postgres stop: success: True, signaled: False, block_callbacks: False
2020-02-17 06:33:21,601 INFO: Lock owner: None; I am database-pg-0
2020-02-17 06:33:21,606 INFO: trying to bootstrap a new cluster
2020-02-17 06:33:21,610 INFO: pg_ctl args: ('-o', '--auth-host=md5 --auth-local=trust --encoding=UTF8 --locale=en_US.UTF-8 --data-checksums --username=postgres --pwfile=/tmp/tmpcdHEH3'), {}
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
creating subdirectories ... ok
sh: line 1:   100 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=100 -c shared_buffers=1000 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   102 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=50 -c shared_buffers=500 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   104 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=40 -c shared_buffers=400 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   106 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=30 -c shared_buffers=300 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   108 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=200 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
selecting default max_connections ... 20
sh: line 1:   110 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=16384 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   112 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=8192 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   114 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=4096 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   116 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=3584 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   118 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=3072 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   120 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=2560 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   122 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=2048 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   124 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=1536 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   126 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=1000 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   128 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=900 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   130 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=800 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   132 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=700 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   134 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=600 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   136 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=500 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   138 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=400 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   140 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=300 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   142 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=200 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   144 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=100 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
sh: line 1:   146 Bus error               (core dumped) "/usr/lib/postgresql10/bin/postgres" --boot -x0 -F -c max_connections=20 -c shared_buffers=50 -c dynamic_shared_memory_type=none < "/dev/null" > "/dev/null" 2>&1
selecting default shared_buffers ... 400kB
selecting default timezone ... UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
child process was terminated by signal 7: Bus error
initdb: removing contents of data directory "/var/lib/postgresql/data/pgdata"
pg_ctl: database system initialization failed
running bootstrap script ... 2020-02-17 06:33:22,254 INFO: removing initialize key after failed attempt to bootstrap the cluster
2020-02-17 06:33:22,260 INFO: Postgres stop: success: True, signaled: False, block_callbacks: False
2020-02-17 06:33:22,260 INFO: renaming data directory to /var/lib/postgresql/data/pgdata_2020-02-17-06-33-22

POD configure for Patroni

      PG_PATRONI_CONFIG:                   bootstrap:
                                             dcs:
                                               synchronous_mode: $(PATRONI_SYNCHRONOUS_ENABLE)
                                               postgresql:
                                                 use_pg_rewind: true
                                                 parameters:
                                                   huge_pages: "off"
                                                   
                                             initdb:
                                             - auth-host: md5
                                             - auth-local: trust
                                             - encoding: UTF8
                                             - locale: en_US.UTF-8
                                             - data-checksums
                                             
                                             pg_hba:
                                             - host all all samehost md5
                                             - host all all ::0/0    md5
                                             - host all all all md5
                                             - host replication $(PATRONI_REPLICATION_USERNAME) all md5
                                             post_init: $(PG_SCRIPT_DIR)/pg_init.sh
                                           postgresql:
                                             callbacks:
                                               on_start: $(PG_SCRIPT_DIR)/patroni_callback.py
                                               on_stop: $(PG_SCRIPT_DIR)/patroni_callback.py
                                               on_role_change: $(PG_SCRIPT_DIR)/patroni_callback.py
                                             parameters:
                                               logging_collector: "off"
                                               huge_pages: "off"
                                          
                                           watchdog:
                                             mode: off

POD resource

    Limits:
      cpu:     1
      memory:  2560Mi
    Requests:
      cpu:      100m
      memory:   256Mi

[qingguee]

Update
This K8s BUG about hugepages which from kubernetes/kubernetes#71233

[adamcharnock]

I came across this issue too. This comment spelled it out for me, but it took me a little while to figure out how to disable huge pages for patroni during the database initialisation step (initdb).

In the end I mounted a custom postgresql.conf.sample into the postgresql container, which seems to have solved the issue. You can do this for versions 9.6-14 as follows (this is using Postgresql Operator):

apiVersion: v1
kind: ConfigMap
metadata:
  name: pg-init-config
data:
  postgresql.conf.sample: "huge_pages = off"
---
# Reference: https://postgres-operator.readthedocs.io/en/latest/reference/cluster_manifest/
apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
  ...
spec:
  ...
  postgresql:
    parameters:
      # Turn off huge pages.
      # Only applies after db has been initialised
      huge_pages: "off"

  # Also force this setting to be applied to the db
  # initialisation process (otherwise the process will fail
  # with a bus error).
  additionalVolumes:
    - name: pg-init-config-9
      mountPath: /usr/share/postgresql/9.6/postgresql.conf.sample
      subPath: postgresql.conf.sample
      volumeSource:
        configMap:
          name: pg-init-config
    - name: pg-init-config-10
      mountPath: /usr/share/postgresql/10/postgresql.conf.sample
      subPath: postgresql.conf.sample
      volumeSource:
        configMap:
          name: pg-init-config
    - name: pg-init-config-11
      mountPath: /usr/share/postgresql/11/postgresql.conf.sample
      subPath: postgresql.conf.sample
      volumeSource:
        configMap:
          name: pg-init-config
    - name: pg-init-config-12
      mountPath: /usr/share/postgresql/12/postgresql.conf.sample
      subPath: postgresql.conf.sample
      volumeSource:
        configMap:
          name: pg-init-config
    - name: pg-init-config-13
      mountPath: /usr/share/postgresql/13/postgresql.conf.sample
      subPath: postgresql.conf.sample
      volumeSource:
        configMap:
          name: pg-init-config
    - name: pg-init-config-14
      mountPath: /usr/share/postgresql/14/postgresql.conf.sample
      subPath: postgresql.conf.sample
      volumeSource:
        configMap:
          name: pg-init-config

Hopefully this saves someone some time.

A future step will be to figure out how to enable huge page support for my larger databases. Right now it looks like the resources: option in Postgres Operator only supports cpu and memory, not hugepages-1Gi / hugepages-2Mi, but I've yet to actually test this. If so I think that would make using huge pages impossible for now.

[David-Angel]

opencontainers/runtime-spec#1050