Postgres 9.6 / 9.6.5 - 'Invalid connection type' upon container up

[absolutejam]

Hey,

I'm trying to launch a Docker compose file for a project and I'm getting the following:

postgres_1  | LOG:  invalid connection type "entication"
postgres_1  | CONTEXT:  line 3 of configuration file "/var/lib/postgresql/data/pg_hba.conf"
postgres_1  | FATAL:  could not load pg_hba.conf
postgres_1  | LOG:  database system is shut down

When I check the file, the start looks like:

host all all all md5
thentication Configuration File
# ===================================================
#
# Refer to the "Client Authentication" section in the PostgreSQL
# documentation for a complete description of this file.  A short
# synopsis follows.
#

I've then tried manually creating a postgres container and I get this every time. I also tried removing all env vars, volumes, the images, etc. - Same deal.

Any ideas?

[tianon]

I can't seem to reproduce: 😞 😕

$ docker pull postgres:9.6
9.6: Pulling from library/postgres
Digest: sha256:144221a1e46af653bdda2bbb8e46ad3dd0c3f61ec0430030581031937f1d589d
Status: Image is up to date for postgres:9.6

$ docker run -dit --name postgres postgres:9.6
ae15509d88ed042a8001bb85beaf0f09abd590661302cb55140c60abe29f0433

$ docker exec -it postgres bash
root@ae15509d88ed:/# cat /var/lib/postgresql/data/pg_hba.conf
# PostgreSQL Client Authentication Configuration File
# ===================================================
#
# Refer to the "Client Authentication" section in the PostgreSQL
# documentation for a complete description of this file.  A short
# synopsis follows.
#
# This file controls: which hosts are allowed to connect, how clients
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access.  Records take one of these forms:
#
# local      DATABASE  USER  METHOD  [OPTIONS]
# host       DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
# hostssl    DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
# hostnossl  DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
# The first field is the connection type: "local" is a Unix-domain
# socket, "host" is either a plain or SSL-encrypted TCP/IP socket,
# "hostssl" is an SSL-encrypted TCP/IP socket, and "hostnossl" is a
# plain TCP/IP socket.
#
# DATABASE can be "all", "sameuser", "samerole", "replication", a
# database name, or a comma-separated list thereof. The "all"
# keyword does not match "replication". Access to replication
# must be enabled in a separate record (see example below).
#
# USER can be "all", a user name, a group name prefixed with "+", or a
# comma-separated list thereof.  In both the DATABASE and USER fields
# you can also write a file name prefixed with "@" to include names
# from a separate file.
#
# ADDRESS specifies the set of hosts the record matches.  It can be a
# host name, or it is made up of an IP address and a CIDR mask that is
# an integer (between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that
# specifies the number of significant bits in the mask.  A host name
# that starts with a dot (.) matches a suffix of the actual host name.
# Alternatively, you can write an IP address and netmask in separate
# columns to specify the set of hosts.  Instead of a CIDR-address, you
# can write "samehost" to match any of the server's own IP addresses,
# or "samenet" to match any address in any subnet that the server is
# directly connected to.
#
# METHOD can be "trust", "reject", "md5", "password", "gss", "sspi",
# "ident", "peer", "pam", "ldap", "radius" or "cert".  Note that
# "password" sends passwords in clear text; "md5" is preferred since
# it sends encrypted passwords.
#
# OPTIONS are a set of options for the authentication in the format
# NAME=VALUE.  The available options depend on the different
# authentication methods -- refer to the "Client Authentication"
# section in the documentation for a list of which options are
# available for which authentication methods.
#
# Database and user names containing spaces, commas, quotes and other
# special characters must be quoted.  Quoting one of the keywords
# "all", "sameuser", "samerole" or "replication" makes the name lose
# its special character, and just match a database or username with
# that name.
#
# This file is read on server startup and when the postmaster receives
# a SIGHUP signal.  If you edit the file on a running system, you have
# to SIGHUP the postmaster for the changes to take effect.  You can
# use "pg_ctl reload" to do that.

# Put your actual configuration here
# ----------------------------------
#
# If you want to allow non-local connections, you need to add more
# "host" records.  In that case you will also need to make PostgreSQL
# listen on a non-local interface via the listen_addresses
# configuration parameter, or via the -i or -h command line switches.

# CAUTION: Configuring the system for local "trust" authentication
# allows any local user to connect as any PostgreSQL user, including
# the database superuser.  If you do not trust all your local users,
# use another authentication method.


# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local   replication     postgres                                trust
#host    replication     postgres        127.0.0.1/32            trust
#host    replication     postgres        ::1/128                 trust

host all all all trust
root@ae15509d88ed:/# 

[absolutejam]

This is what I get when running docker run postgres:9.6

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
****************************************************
WARNING: No password has been set for the database.
         This will allow anyone with access to the
         Postgres port to access your database. In
         Docker's default configuration, this is
         effectively any other container on the same
         system.

         Use "-e POSTGRES_PASSWORD=password" to set
         it in "docker run".
****************************************************
waiting for server to start....LOG:  could not bind IPv6 socket: Cannot assign requested address
HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
LOG:  invalid connection type "entication"
CONTEXT:  line 3 of configuration file "/var/lib/postgresql/data/pg_hba.conf"
FATAL:  could not load pg_hba.conf
LOG:  database system is shut down
 stopped waiting
pg_ctl: could not start server
Examine the log output.

[jiminhsieh]

@tianon I found if you use this command docker logs -f {container_id}, you would see same logs that @absolutejam mentioned.

@absolutejam When you use docker run postgres:9.6, PostgreSQL would run in the background. You could try this command docker exec -it {container_id} bash, you would see PostgreSQL is running in the background.

About this log:

waiting for server to start....LOG: could not bind IPv6 socket: Cannot assign requested address

I could not figure out the reason of it.

[tianon]

@jiminhsieh but the logs @absolutejam mentioned were cut off -- mine are the full logs of a successful PostgreSQL startup 😕

I'd recommend completely removing your postgres images and re-pulling them (Docker occasionally suffers from on-disk corruption), and then trying to reproduce again.

As for could not bind IPv6 socket: Cannot assign requested address, is it possible that IPv6 is disabled on your host? (There were a few release of Docker where using --net=host containers could accidentally cause this to be configured, which broke quite a few things.)

Regardless, since we cannot reproduce the issue, I'm going to close this, and would recommend posting to the Docker Community Forums, the Docker Community Slack, or Stack Overflow for further debugging help, and filing a new issue if/when you have a simple reproducer to demonstrate an issue with this image. Thanks!

[bbratchiv]

If someone faces could not bind IPv6 socket: Cannot assign requested address issue and it's not connected with network, corrupted base images, then try the following. Check that you are not mounting a volume to postgres container, look at Dockerfile and compose file as well. Remove all mounts and try to rebuild the image. That fixed my problem.

[unlucio]

I'm facing the could not bind IPv6 socket: Cannot assign requested address as well.
----------------- config infos -------------------
Host: Darwin Asuna.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64 (macOS 10.13.4)

Docker: community edition - Version 18.04.0-ce-rc2-mac61 (23890)
(at this very moment I'm running the experimental version hoping it could help, but the same happens with the current stable one)

docker-compose:

db:
    network_mode: "bridge"
    image: postgres:9.6-alpine
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: docker
      POSTGRES_PASSWORD: docker
      POSTGRES_DB: docker

So nothing fancy other than the basic config needed to use it.
(but the same happens if I comment out everything except image: postgres:9.6-alpine, or use image postgres:9.6)

Output for docker-compose run db
(yes fig is actually /Users/<user>/local/bin/fig -> /usr/local/bin/docker-compose 'cose life is too short to type docker-compose)

➜  fig run db
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... sh: locale: not found
No usable system locales were found.
Use the option "--debug" to see details.
ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....LOG:  could not bind IPv6 socket: Address not available
HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
LOG:  database system was shut down at 2018-04-11 09:53:50 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
 done
server started
CREATE DATABASE

CREATE ROLE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down...LOG:  received fast shutdown request
.LOG:  aborting any active transactions
LOG:  autovacuum launcher shutting down
LOG:  shutting down
LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

LOG:  database system was shut down at 2018-04-11 09:53:52 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

----------------- /config infos -------------------

My 1st reaction was obviusly to check whatever I had ipv6 support enabled, only to find this bit of doc:
https://docs.docker.com/config/daemon/ipv6/
where Note: IPv6 networking is only supported on Docker daemons running on Linux hosts. makes me wonder if anyone was ever able to actually run this image on an OSX machine (and eventually how) since I see this kind of issue been around since a long time on this repo.

Some interesting bits I noticed when running it:

waiting for server to start....LOG:  could not bind IPv6 socket: Address not available

here is the pesky error message but:

[...]
LOG:  autovacuum launcher started
 done
server started                           <---- here it seems to be started any way(?)
CREATE DATABASE

CREATE ROLE

Only to shutdown right the way

waiting for server to shut down...LOG:  received fast shutdown request
[...]
 done
server stopped

but then it starts back up (again?)

PostgreSQL init process complete; ready for start up.

and down again

LOG:  database system was shut down at 2018-04-11 09:53:52 UTC

ending up with :

LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections            <------ wut? O.o
LOG:  autovacuum launcher started

end docker-compose doesn't quit, letting me think despite the errors stuff is actually running in that container.

Listing the active containers:

➜  docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS               NAMES
d502bb4fe7e5        postgres:9.6-alpine   "docker-entrypoint.s…"   13 minutes ago      Up 13 minutes       5432/tcp            b2baccountsapi_db_run_3

shows it running but the exported port doesn't seems to be connected, and checking inside the container:

➜  ~ docker exec -it  d502bb4fe7e5 bash
bash-4.3# netstat -plunt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      -
tcp        0      0 :::5432                 :::*                    LISTEN      -
bash-4.3# ps aux
PID   USER     TIME   COMMAND
    1 postgres   0:00 postgres
   48 postgres   0:00 postgres: checkpointer process
   49 postgres   0:00 postgres: writer process
   50 postgres   0:00 postgres: wal writer process
   51 postgres   0:00 postgres: autovacuum launcher process
   52 postgres   0:00 postgres: stats collector process
   97 root       0:00 bash

shows postgres active and listening (amusingly on ipv6 too after throwing the known error?), yet clients cannot connect to it.
I went thought all the usual voodoo: delete and re-install docker (multiple versions), delete and re-pull the base image (multiple versions multiple times), reboot my computer, etc but nothing changed.
At this point I'm pretty confused and I'm out of walls where to bang my head on to find a solution.

I'm just a simple developer so both my knowledge of postgres-administration and troubleshooting are limited (I'm just trying to have a postgres instace for my dev env, so time to troubleshoot something like this is very limited as well).

Can anyone perhaps suggest a postgres image that just works?
thanks for help.

[wglambert]

Closing as stated here #343 (comment) this isn't related to an error with the image and the environmental issue isn't reproducible so there's nothing more we can do.