Abnormal Liquibase lock blocks app-autoscaler from starting

[cdlliuy]

App-autoscaler uses Liquibase to maintain the database change sets, and runs Liquibase process as a pre-start job.
Once a Liquibase process is running , it will insert a DB lock record , and then remove it with normal completion. But in some weird situation (i.e. liquibase update was interrupted ..) , the db lock won't be removed, and blocked the further autoscaler startup.

See detail in https://www.liquibase.org/documentation/databasechangeloglock_table.html

Currently , Liquibase don't have DB lock TTL implemented, so we need to workaround this with cmd listLocks and releaseLocks.

Details steps to reproduce and fix the problem:

step1: do a liquibase update manually , and ctrl+c to break its execution.

java -cp "$DB_DIR/target/lib/*" liquibase.integration.commandline.Main --url "$DBURI" --username=$USER --password=$PASSWORD --driver=org.postgresql.Driver --changeLogFile=$API_DIR/db/api.db.changelog.yml update
Starting Liquibase at Tue, 20 Aug 2019 05:05:11 UTC (version 3.6.3 built at 2019-01-29 11:34:48)
^Cautoscaler-api/1:/var/vcap/jobs/apiserver/bin# ^C

step2: now , a db lock was left in ICD database:

autoscaler-api/1:/var/vcap/jobs/apiserver/bin# java -cp "$DB_DIR/target/lib/*" liquibase.integration.commandline.Main --url "$DBURI" --username=$USER --password=$PASSWORD --driver=org.postgresql.Driver listLocks
Starting Liquibase at Tue, 20 Aug 2019 05:05:28 UTC (version 3.6.3 built at 2019-01-29 11:34:48)
Database change log locks for autoscaler@jdbc:postgresql://external-db.uaa.svc.cluster.local:31404/autoscaler?sslmode=require
 - autoscaler-api-1.autoscaler-api-set.cf.svc.cluster.local (172.30.51.196) at Aug 20, 2019, 5:05:12 AM
Liquibase command 'listLocks' was executed successfully.

step3: run the update cmd in step1 again. Now , it hung just as what we found in @travagli 's cluster

To fix it, we can execute "release lock" cmd as below:

autoscaler-api/1:/var/vcap/jobs/apiserver/bin# java -cp "$DB_DIR/target/lib/*" liquibase.integration.commandline.Main --url "$DBURI" --username=$USER --password=$PASSWORD --driver=org.postgresql.Driver releaselocks
Starting Liquibase at Tue, 20 Aug 2019 05:08:36 UTC (version 3.6.3 built at 2019-01-29 11:34:48)
Successfully released all database change log locks for 'autoscaler@jdbc:postgresql://external-db.uaa.svc.cluster.local:31404/autoscaler?sslmode=require'
Liquibase command 'releaselocks' was executed successfully.

autoscaler-api/1:/var/vcap/jobs/apiserver/bin# java -cp "$DB_DIR/target/lib/*" liquibase.integration.commandline.Main --url "$DBURI" --username=$USER --password=$PASSWORD --driver=org.postgresql.Driver listLocks
Starting Liquibase at Tue, 20 Aug 2019 05:08:47 UTC (version 3.6.3 built at 2019-01-29 11:34:48)
Database change log locks for autoscaler@jdbc:postgresql://external-db.uaa.svc.cluster.local:31404/autoscaler?sslmode=require
 - No locks
Liquibase command 'listLocks' was executed successfully.

autoscaler-api/1:/var/vcap/jobs/apiserver/bin# java -cp "$DB_DIR/target/lib/*" liquibase.integration.commandline.Main --url "$DBURI" --username=$USER --password=$PASSWORD --driver=org.postgresql.Driver --changeLogFile=$API_DIR/db/api.db.changelog.yml update
Starting Liquibase at Tue, 20 Aug 2019 05:08:52 UTC (version 3.6.3 built at 2019-01-29 11:34:48)
Liquibase: Update has been successful.

[garethjevans]

This should be fixed in the latest 3.1.1 release