Upgrade fails from 0.28.1-alpha to 0.29.18.alpha

[zestyping]

Environment

• Airbyte version: 0.28.1-alpha
• OS Version / Instance: AWS EC2
• Deployment: Docker
• Source Connector and version: n/a
• Destination Connector and version: n/a
• Severity: High
• Step where error happened: Upgrade Airbyte

Current Behavior

After following the upgrade steps described at https://docs.airbyte.io/operator-guides/upgrading-airbyte (docker-compose down, then git pull, then docker-compose up), attempting to load the Airbyte dashboard just shows a blank page. In the Dev Tools console, we see the error message:

Error: Version mismatch between 0.29.18-alpha and 0.28.1-alpha.
Please upgrade or reset your Airbyte Database, see more at https://docs.airbyte.io/operator-guides/upgrading-airbyte

Expected Behavior

After version 0.27.0 was released, Airbyte migrated its internal configuration automatically when upgrading. It usually runs the migration automatically and then starts up without incident.

Logs

Based on the logs, it looks like it did actually try to run a migration, but failed due to a JSON validation problem.

2021-09-16 18:45:26 ERROR i.a.s.ServerApp(runAutomaticMigration):284 - {workspace_app_root=/tmp/workspace/server/logs} - Automatic Migration failed 
java.lang.IllegalArgumentException: Input data schema does not match declared input schema ATTEMPTS.
	at io.airbyte.migrate.Migrate.lambda$runMigration$1(Migrate.java:140) ~[io.airbyte-airbyte-migration-0.29.18-alpha.jar:?]
	at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:441) ~[?:?]
	at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
	at io.airbyte.migrate.migrations.MigrationV0_29_0.migrate(MigrationV0_29_0.java:110) ~[io.airbyte-airbyte-migration-0.29.18-alpha.jar:?]
	at io.airbyte.migrate.MigrateWithMetadata.migrate(MigrateWithMetadata.java:52) ~[io.airbyte-airbyte-migration-0.29.18-alpha.jar:?]
	at io.airbyte.migrate.Migrate.runMigration(Migrate.java:150) ~[io.airbyte-airbyte-migration-0.29.18-alpha.jar:?]
	at io.airbyte.migrate.Migrate.run(Migrate.java:113) ~[io.airbyte-airbyte-migration-0.29.18-alpha.jar:?]
	at io.airbyte.migrate.MigrationRunner.run(MigrationRunner.java:76) ~[io.airbyte-airbyte-migration-0.29.18-alpha.jar:?]
	at io.airbyte.server.RunMigration.run(RunMigration.java:79) ~[io.airbyte-airbyte-server-0.29.18-alpha.jar:?]
	at io.airbyte.server.ServerApp.runAutomaticMigration(ServerApp.java:282) [io.airbyte-airbyte-server-0.29.18-alpha.jar:?]
	at io.airbyte.server.ServerApp.getServer(ServerApp.java:234) [io.airbyte-airbyte-server-0.29.18-alpha.jar:?]
	at io.airbyte.server.ServerApp.main(ServerApp.java:263) [io.airbyte-airbyte-server-0.29.18-alpha.jar:?]
Caused by: io.airbyte.validation.json.JsonValidationException: json schema validation failed. 
errors: $.temporal_workflow_id: is not defined in the schema and the schema does not allow additional properties 
schema: 
{
  "$schema" : "http://json-schema.org/draft-07/schema#",
  "title" : "Attempts",
  "description" : "representation of a attempts record as created in schema.sql",
  "type" : "object",
  "required" : [ "id", "job_id", "attempt_number", "log_path", "status", "created_at", "updated_at" ],
  "additionalProperties" : false,
  "properties" : {
    "id" : {
      "type" : "number"
    },
    "job_id" : {
      "type" : "number"
    },
    "attempt_number" : {
      "type" : "number"
    },
    "log_path" : {
      "type" : "string"
    },
    "output" : {
      "type" : [ "null", "object" ]
    },
    "status" : {
      "type" : "string"
    },
    "created_at" : {
      "type" : "string",
      "description" : "Integration specific blob. Must be a valid JSON string."
    },
    "updated_at" : {
      "type" : "string"
    },
    "ended_at" : {
      "type" : [ "null", "string" ]
    }
  }
} 
object: 
{
  "id" : 1,
  "job_id" : 1,
  "attempt_number" : 0,
  "log_path" : "/tmp/workspace/1/0/logs.log",
  "output" : null,
  "status" : "running",
  "created_at" : "2021-07-27T17:31:55.394428Z",
  "updated_at" : "2021-07-27T17:31:55.394428Z",
  "ended_at" : null,
  "temporal_workflow_id" : null
}

Steps to Reproduce

docker-compose down
git pull
docker-compose up

[tuliren]

Working on a solution. Looks like we need to release a new version to fix this.

[zestyping]

Awesome, thank you @tuliren!

[tuliren]

I am releasing a new version that should fix this.

[tuliren]

@zestyping, could you try the latest release: v0.29.19-alpha? Thank you.

[zestyping]

It worked! Thank you.

[tuliren]

Great! Thank you for the update. Sorry about the inconvenience.