java.lang.NullPointerException during DIFF operation

[NicolaSpreafico]

Liquibase Version

Starting Liquibase at 14:54:49 (version 4.2.2 #36 built at 2020-12-09 20:07+0000)
Liquibase Version: 4.2.2
Liquibase Community 4.2.2 by Datical
Running Java under /usr/lib/jvm/java-11-openjdk-amd64 (Version 11.0.9.1)

Command to generate snapshot

liquibase \
        --driver=org.postgresql.Driver \
        --classpath=postgresql-42.2.18.jar \
        --url="jdbc:postgresql://$SQL_IP/$SQL_DATABASE" \
        --username="$SQL_USERNAME" \
        --password="$SQL_PASSWORD" \
        --changeLogFile="$CHANGELOG_MASTER_FILE" \
        --liquibaseSchemaName=$SQL_LIQUIBASE_SCHEMA \
        --defaultSchemaName=$SQL_APPLICATION_SCHEMA \
        --contexts=$ENVIRONMENT \
        --outputFile="$STATUS_LOG_FOLDER/snapshot.after.json" \
        snapshot \
        --snapshotFormat=json

Command to generate diff

liquibase \
        --driver=org.postgresql.Driver \
        --classpath=postgresql-42.2.18.jar \
        --url="offline:postgresql?snapshot=$STATUS_LOG_FOLDER/snapshot.before.json" \
        --referenceUrl="jdbc:postgresql://$SQL_IP/$SQL_DATABASE" \
        --referenceusername="$SQL_USERNAME" \
        --referencepassword="$SQL_PASSWORD" \
        --referenceDefaultSchemaName=$SQL_APPLICATION_SCHEMA \
        --liquibaseSchemaName=$SQL_LIQUIBASE_SCHEMA \
        --contexts=$ENVIRONMENT \
        --outputFile="$STATUS_LOG_FOLDER/diff.txt" \
        diff

Thsi is the error I'm getting when launching DIFF command

[2021-01-15 14:53:41] FINE [liquibase.servicelocator] Loaded liquibase.diff.DiffGenerator instance liquibase.diff.core.StandardDiffGenerator
Unexpected error running Liquibase: java.lang.NullPointerException
For more information, please use the --logLevel flag
[2021-01-15 14:53:41] SEVERE [liquibase.integration] Unexpected error running Liquibase: java.lang.NullPointerException
liquibase.exception.LiquibaseException: liquibase.command.CommandExecutionException: java.lang.NullPointerException
        at liquibase.integration.commandline.CommandLineUtils.doDiff(CommandLineUtils.java:173)
        at liquibase.integration.commandline.Main.doMigration(Main.java:1480)
        at liquibase.integration.commandline.Main$1.lambda$run$0(Main.java:361)
        at liquibase.Scope.lambda$child$0(Scope.java:160)
        at liquibase.Scope.child(Scope.java:169)
        at liquibase.Scope.child(Scope.java:159)
        at liquibase.Scope.child(Scope.java:138)
        at liquibase.Scope.child(Scope.java:222)
        at liquibase.Scope.child(Scope.java:226)
        at liquibase.integration.commandline.Main$1.run(Main.java:360)
        at liquibase.integration.commandline.Main$1.run(Main.java:193)
        at liquibase.Scope.child(Scope.java:169)
        at liquibase.Scope.child(Scope.java:145)
        at liquibase.integration.commandline.Main.run(Main.java:193)
        at liquibase.integration.commandline.Main.main(Main.java:156)
Caused by: liquibase.command.CommandExecutionException: java.lang.NullPointerException
        at liquibase.command.AbstractCommand.execute(AbstractCommand.java:24)
        at liquibase.integration.commandline.CommandLineUtils.doDiff(CommandLineUtils.java:171)
        ... 14 more
Caused by: java.lang.NullPointerException
        at liquibase.diff.compare.core.IndexComparator$1.areEqual(IndexComparator.java:123)
        at liquibase.diff.ObjectDifferences.compare(ObjectDifferences.java:70)
        at liquibase.diff.ObjectDifferences.compare(ObjectDifferences.java:50)
        at liquibase.diff.compare.core.IndexComparator.findDifferences(IndexComparator.java:113)
        at liquibase.diff.compare.DatabaseObjectComparatorChain.findDifferences(DatabaseObjectComparatorChain.java:99)
        at liquibase.diff.compare.DatabaseObjectComparatorFactory.findDifferences(DatabaseObjectComparatorFactory.java:166)
        at liquibase.diff.core.StandardDiffGenerator.compareObjectType(StandardDiffGenerator.java:90)
        at liquibase.diff.core.StandardDiffGenerator.compare(StandardDiffGenerator.java:52)
        at liquibase.diff.DiffGeneratorFactory.compare(DiffGeneratorFactory.java:107)
        at liquibase.command.core.DiffCommand.createDiffResult(DiffCommand.java:152)
        at liquibase.command.core.DiffCommand.run(DiffCommand.java:136)
        at liquibase.command.AbstractCommand.execute(AbstractCommand.java:19)
        ... 15 more

Testing Criteria

This needs to be tested against AWS Aurora PostgreSQL 10.11 compatible.

Source Line: IndexComparator.java:123

I added the logLevel=DEBUG as suggested, but besided having the stacktrace I'm not getting additional information.

[molivasdat]

Thanks @NicolaSpreafico for bringing this to our attention. We should definitely fix the NPE.
Not sure if this is the issue or if you just have a typo in your recreation scenario.

In the snapshot you create you call it
--outputFile="$STATUS_LOG_FOLDER/snapshot.after.json" \
The file you create is called snapshot.after.json
and in the diff you use you call it.
--url="offline:postgresql?snapshot=$STATUS_LOG_FOLDER/snapshot.before.json" \
And the file you use to compare is called
snapshot.before.json

Is that the issue?

[NicolaSpreafico]

Hi @molivasdat,
there is indeed a typo, but it is inside this issue.

In my script I create a snapshot before and after the update operation, so that after the update I can diff the previous status of the database (snapshot.before.json) and the current status of the DB.

The snapshot.after.json is created as part of the logging but it is not considered for the diff operation

[keyset]

We're experiencing the same error and stacktrace as well, also on postgres and liquibase 4.2.2. Our database has several indexes containing computed columns; when I remove those indexes from the snapshot, I do not receive the error, and the diff command identifies the unexpected indexes and columns. As soon as I add one of the indexes back, the error recurs. @NicolaSpreafico does your database also contain indexes with computed columns? If not, I will create a separate issue.

[keyset]

@molivasdat @NicolaSpreafico Any insight on this bug?

[NicolaSpreafico]

Hi, my does not have computed column indexes.
Right now I'm simply waiting a reply in this issue for the resolution, for the moment I disabled the snapshot part in my script

[molivasdat]

Sorry I don't have an update on this. Other than the computed index column, I don't have a way to reproduce this one and see what is causing the NPE. @NicolaSpreafico It looks like one of your indexes is causing it. And in @keyset issue, it's based on a computed index that is causing the NPE.

[keyset]

@molivasdat should I create a separate bug report for this being caused by computed index columns, or is there one already filed that I can follow?

[molivasdat]

@keyset There is not one already. Is this Postgres 12 or higher?

Looking at something like this to cause an error.
https://dbfiddle.uk/?rdbms=postgres_12&fiddle=40e0511a30b0f4f6629e48861e69c29c
You are welcome to create another issue on specifically computed index columns.

[NicolaSpreafico]

@molivasdat I'm currently using Postgres13.

Is there a way to determine which index specifically is generating the issue? I tried to enable verbose log but besides having the stacktrace to identify where is thrown, I don't have runtime informations to pinpoint the exact index.

[molivasdat]

Hi @NicolaSpreafico I don't have a suggestion on how to turn on extra debugging for a diff. I would although look based on the line where the error is occuring an index based on a column that has a null name. Maybe look at the json file that is being generated for null values in index names?

[sync-by-unito]

➤ Mario Champion commented:

hey Mike Olivas and Wesley Willard / Nathan Voxland – can yall do quick chat to get an idea of the scope of this issue. that is, your gut take and then QA’s running PG 12 and 13 thru test-harness, especially using computed indexes, but also any new features in those versions. Then we use that output to scope and prioritize work. thanks!

[sync-by-unito]

➤ Mario Champion commented:

hey Erzsebet Carmean – did you see the example from above: “Looking at something like this to cause an error.
https://dbfiddle.uk/?rdbms=postgres_12&fiddle=40e0511a30b0f4f6629e48861e69c29c ( https://dbfiddle.uk/?rdbms=postgres_12&fiddle=40e0511a30b0f4f6629e48861e69c29c )
You are welcome to create another issue on specifically computed index columns.”

which has a code snippet that should be able to be used/converted to a changelog. at least i think so…

[sync-by-unito]

➤ Erzsebet Carmean commented:

Liquibase 4.2.2Postgres 13.2-alpinePostgres Driver 42.2.8Java 11Reproduced the NPE with Liquibase 4.2.2

• Execute a liquibase update to deploy changes in [^changelog.postgresql.sql]

  • liquibase update

• Execute a liquibase snapshot to JSON format

  • liquibase --outputFile=snapshot.json snapshot --snapshotFormat=JSON

• Execute a liquibase diff between the live database and the JSON snapshot

  • liquibase diff

The properties file needs to be configured with referenceURL to be the JSON snapshot. For example:
{CODE}
changeLogFile: changelog.postgresql.sql
url: jdbc:postgresql://localhost:5436/postgres
username: name
password: password
#referenceUrl: jdbc:postgresql://localhost:5432/dev
referenceUrl: offline:postgresql?snapshot=snapshot.json
referenceUsername: name
referencePassword: password
classpath: postgresql-42.2.8.jar
{CODE}

[sync-by-unito]

➤ Erzsebet Carmean commented:

Liquibase 4.3.2-LB-1115-SNAPSHOT #1

Fix will be in 4.4.0Verify the diff of an online and offline JSON snapshot does not throw an NPE. PASS

I verified the fix manually and also added a functional test. The PR for the Liquibase internal functional test repo here: https://github.com/Datical/liquibase-pro-tests/pull/127 ( https://github.com/Datical/liquibase-pro-tests/pull/127 ).

Looks good! Thanks for the help, Tsvi Zandany!

[nvoxland]

Fixed in #1758 in 4.4.0