Postgresql database schema creation fails when running w/o elasticsearch.

[mgetka]

Postgresql database schema creation fails when running w/o elasticsearch.

Description

Maintenance mode doesn't properly initialize postgresql database when the service is ran without elasticsearch. The issue manifests itself through FusionAuth's runtime user authentication failures while maintenance mode workflow is running. Everything works fine when running with elasticsearch.

Steps to reproduce

Steps to reproduce the behavior:

1. Start FusionAuth in development mode with an empty database and without elasticsearch.
2. Observe authentication errors in the database log

Expected behavior

Database schema is created via maintenance mode.

Platform

• Device: Desktop
• OS: 18.04.2-Ubuntu x64
• Docker: 19.03.8, build afacb8b7f0
• Image: fusionauth/fusionauth-app:1.16.0
• database image: postgres:9.6

Additional context

Postgresql log entries:

fusionauth_1  | ---------------------------------------------------------------------------------------------------------
fusionauth_1  | --------------------------------------- Entering Maintenance Mode ---------------------------------------
fusionauth_1  | ---------------------------------------------------------------------------------------------------------
fusionauth_1  | 
fusionauth_1  | May 10, 2020 6:31:21.934 PM INFO  io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
fusionauth_1  | May 10, 2020 6:31:21.934 PM INFO  io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration - Override property [fusionauth-app.url], set to [http://fusionauth:9011] from ENV.FUSIONAUTH_URL.
fusionauth_1  | May 10, 2020 6:31:21.934 PM INFO  io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration - Override property [fusionauth.runtime-mode], set to [Development] from ENV.FUSIONAUTH_RUNTIME_MODE.
fusionauth_1  | May 10, 2020 6:31:21.935 PM INFO  com.inversoft.maintenance.MaintenanceModePoller - Poller started to Wait for configuration to be completed.
fusionauth_1  | 10-May-2020 18:31:21.979 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-9011"]
fusionauth_1  | 10-May-2020 18:31:22.001 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-9013"]
fusionauth_1  | 10-May-2020 18:31:22.010 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-127.0.0.1-9019"]
fusionauth_1  | 10-May-2020 18:31:22.014 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 3177 ms
db_1          | LOG:  incomplete startup packet
db_1          | FATAL:  password authentication failed for user "fusionauth"
db_1          | DETAIL:  Role "fusionauth" does not exist.
db_1          | 	Connection matched pg_hba.conf line 95: "host all all all md5"
db_1          | LOG:  incomplete startup packet
db_1          | FATAL:  password authentication failed for user "fusionauth"
db_1          | DETAIL:  Role "fusionauth" does not exist.
db_1          | 	Connection matched pg_hba.conf line 95: "host all all all md5"
db_1          | LOG:  incomplete startup packet
db_1          | FATAL:  password authentication failed for user "fusionauth"
db_1          | DETAIL:  Role "fusionauth" does not exist.
db_1          | 	Connection matched pg_hba.conf line 95: "host all all all md5"
[...]

The problem can be reproduced with the following docker-compose file:

version: "3"

services:
  db:
    image: postgres:9.6
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres

  #search:
  #  image: docker.elastic.co/elasticsearch/elasticsearch:6.3.1
  #  environment:
  #    - cluster.name=fusionauth
  #    - bootstrap.memory_lock=true
  #    - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
  #  restart: unless-stopped
  #  ulimits:
  #    memlock:
  #      soft: -1
  #      hard: -1

  fusionauth:
    image: fusionauth/fusionauth-app:1.16.0
    environment:
      DATABASE_URL: jdbc:postgresql://db:5432/fusionauth
      DATABASE_ROOT_USER: postgres
      DATABASE_ROOT_PASSWORD: postgres
      DATABASE_USER: fusionauth
      DATABASE_PASSWORD: fusionauth
      #FUSIONAUTH_SEARCH_SERVERS: http://search:9200
      FUSIONAUTH_URL: http://fusionauth:9011
      FUSIONAUTH_RUNTIME_MODE: development
    ports:
      - 9011:9011

When elasticsearch related lines are uncommented the service starts correctly.

[codemonkeysoftware]

I'm seeing the same issue as well. I thought maybe the db needed time to start up before the connection (which would happen while waiting for the search server), but starting the db first and then the FusionAuth service does not change the outcome.

[mooreds]

There was a new version shipped 6 days ago: 798ead8ae175. https://hub.docker.com/r/fusionauth/fusionauth-app/tags

I just downloaded that version from a clean docker setup (removed all previous images related to FA) and was unable to replicate. Can you please try to replicate using this image and let me know if the problem still exists?

[codemonkeysoftware]

I was running on the lastest 1.16 tag, and I just blew away everything in docker and started over and had the same issue.

[cheelahim]

@lpdyck I have similar setup and it's working fine. But I explicitly specified the search engine type.

FUSIONAUTH_SEARCH_ENGINE_TYPE: database

[mooreds]

@lpdyck can you try @cheelahim 's workaround and let us know if that fixes it, please?

[mgetka]

I've checked with the following docker-compose file

version: "3"

services:
  db:
    image: postgres:9.6
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres

  #search:
  #  image: docker.elastic.co/elasticsearch/elasticsearch:6.3.1
  #  environment:
  #    - cluster.name=fusionauth
  #    - bootstrap.memory_lock=true
  #    - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
  #  restart: unless-stopped
  #  ulimits:
  #    memlock:
  #      soft: -1
  #      hard: -1

  fusionauth:
    image: fusionauth/fusionauth-app:1.16.0
    environment:
      DATABASE_URL: jdbc:postgresql://db:5432/fusionauth
      DATABASE_ROOT_USER: postgres
      DATABASE_ROOT_PASSWORD: postgres
      DATABASE_USER: fusionauth
      DATABASE_PASSWORD: fusionauth
      FUSIONAUTH_SEARCH_ENGINE_TYPE: database
      #FUSIONAUTH_SEARCH_SERVERS: http://search:9200
      FUSIONAUTH_URL: http://fusionauth:9011
      FUSIONAUTH_RUNTIME_MODE: development
    ports:
      - 9011:9011

and issue is still present. Tested with image 22fc55f73ef2 (digest sha256:798ead8ae1753c118ea82e4e7bc866847214d540a82b298e6bfda79c8ed2befd).

Update: The issue can be reporoduced in PWD environment with provided docker-compose file.

[trex]

@mgetka Prior to 1.16.0 we were entering into Silent Configuration Mode when the FUSIONAUTH_SEARCH_SERVERS, DATABASE_URL and DATABASE_ROOT_USER environment variables are set. This allows you to bypass the user-interactive Maintenance Mode.

We are still expecting the FUSIONAUTH_SEARCH_SERVERS variable in 1.16.0 even when we are configured to use the database as the search engine, FUSIONAUTH_SEARCH_ENGINE_TYPE=database.

This is a bug and we will get a patch released as soon as possible. This should be on the 1.16.1 release.

[robotdan]

Version 1.16.1 is released, feel free to try again and post your results.

[codemonkeysoftware]

Confirmed fixed