Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[upgrade] Migrations from v0.44.2 are broken by the recent refactor #36855

Open
killthekitten opened this issue Apr 5, 2024 · 11 comments
Open
Labels
area/platform issues related to the platform community helm step/upgrade team/deployments type/bug Something isn't working

Comments

@killthekitten
Copy link
Contributor

Platform Version

0.44.0, 0.57.1

What step the error happened?

Upgrading the Platform or Helm Chart

Revelant information

We are upgrading from v0.44.0 to v0.57.1 and encountered an error in V0_44_4_002__MigrateNonBreakingChangeToEnum migration that stops the bootloader process:

SQL [select id from connection where non_breaking_change_preference = cast(? as "non_breaking_change_preference_type")]; ERROR: operator does not exist: character varying = non_breaking_change_preference_type

Hint: No operator matches the given name and argument types. You might need to add explicit type casts.

It comes from a typecast in migrateExistingDisableConnection. The migration was changed a few months ago in a mass refactor related to Micronaut, specifically this change:

airbytehq/airbyte-platform@75a3855#diff-65ad779d71239248181343fa5d51e662439bcf7b2b4ba2ea7073606210d5fb82R96-R98

The new code assumes that the column is of a type non_breaking_change_preference_type while it is actually varchar.

FYI @jdpgrailsdev @jpefaur

Relevant log output

airbyte-bootloader                | 2024-04-04 16:14:19 INFO o.f.c.i.l.s.Slf4jLog(info):37 - Successfully validated 111 migrations (execution time 00:00.052s)
airbyte-bootloader                | 2024-04-04 16:14:19 INFO o.f.c.i.l.s.Slf4jLog(info):37 - Current version of schema "public": 0.44.4.001
airbyte-bootloader                | 2024-04-04 16:14:19 INFO o.f.c.i.l.s.Slf4jLog(info):37 - Migrating schema "public" to version "0.44.4.002 - MigrateNonBreakingChangeToEnum"
airbyte-bootloader                | 2024-04-04 16:14:19 ERROR o.f.c.i.l.s.Slf4jLog(error):45 - Migration of schema "public" to version "0.44.4.002 - MigrateNonBreakingChangeToEnum" failed! Changes successfully rolled back.
airbyte-bootloader                | 2024-04-04 16:14:19 ERROR i.a.b.Application(main):25 - Unable to bootstrap Airbyte environment.
airbyte-bootloader                | org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: Migration failed !
airbyte-bootloader                |     at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:382) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.command.DbMigrate.lambda$applyMigrations$1(DbMigrate.java:272) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.jdbc.TransactionalExecutionTemplate.execute(TransactionalExecutionTemplate.java:55) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:271) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:244) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.command.DbMigrate.lambda$migrateAll$0(DbMigrate.java:139) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:73) ~[flyway-database-postgresql-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.database.postgresql.PostgreSQLAdvisoryLockTemplate.lambda$execute$0(PostgreSQLAdvisoryLockTemplate.java:56) ~[flyway-database-postgresql-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.jdbc.TransactionalExecutionTemplate.execute(TransactionalExecutionTemplate.java:55) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:56) ~[flyway-database-postgresql-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:96) ~[flyway-database-postgresql-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:144) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:139) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:97) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.Flyway.lambda$migrate$0(Flyway.java:202) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:200) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.Flyway.migrate(Flyway.java:147) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at io.airbyte.db.instance.FlywayDatabaseMigrator.migrate(FlywayDatabaseMigrator.java:45) ~[io.airbyte.airbyte-db-db-lib-0.57.1.jar:?]
airbyte-bootloader                |     at io.airbyte.bootloader.Bootloader.runFlywayMigration(Bootloader.java:270) ~[io.airbyte-airbyte-bootloader-0.57.1.jar:?]
airbyte-bootloader                |     at io.airbyte.bootloader.Bootloader.load(Bootloader.java:117) ~[io.airbyte-airbyte-bootloader-0.57.1.jar:?]
airbyte-bootloader                |     at io.airbyte.bootloader.Application.main(Application.java:22) [io.airbyte-airbyte-bootloader-0.57.1.jar:?]
airbyte-bootloader                | Caused by: org.flywaydb.core.api.FlywayException: Migration failed !
airbyte-bootloader                |     at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.executeOnce(JavaMigrationExecutor.java:76) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.lambda$execute$0(JavaMigrationExecutor.java:54) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.execute(JavaMigrationExecutor.java:53) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:374) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     ... 20 more
airbyte-bootloader                | Caused by: org.jooq.exception.DataAccessException: SQL [select id from connection where non_breaking_change_preference = cast(? as "non_breaking_change_preference_type")]; ERROR: operator does not exist: character varying = non_breaking_change_preference_type
airbyte-bootloader                |   Hint: No operator matches the given name and argument types. You might need to add explicit type casts.
airbyte-bootloader                |   Position: 64
airbyte-bootloader                |     at org.jooq_3.19.1.POSTGRES.debug(Unknown Source) ~[?:?]
airbyte-bootloader                |     at org.jooq.impl.Tools.translate(Tools.java:3528) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.Tools.translate(Tools.java:3516) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:812) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:362) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:301) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.SelectImpl.fetchLazy(SelectImpl.java:3251) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.ResultQueryTrait.lambda$fetchStream$1(ResultQueryTrait.java:329) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at java.base/java.util.stream.AbstractPipeline.sourceSpliterator(AbstractPipeline.java:405) ~[?:?]
airbyte-bootloader                |     at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
airbyte-bootloader                |     at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
airbyte-bootloader                |     at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
airbyte-bootloader                |     at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
airbyte-bootloader                |     at io.airbyte.db.instance.configs.migrations.V0_44_4_002__MigrateNonBreakingChangeToEnum.migrateExistingDisableConnection(V0_44_4_002__MigrateNonBreakingChangeToEnum.java:101) ~[io.airbyte.airbyte-db-db-lib-0.57.1.jar:?]
airbyte-bootloader                |     at io.airbyte.db.instance.configs.migrations.V0_44_4_002__MigrateNonBreakingChangeToEnum.migrate(V0_44_4_002__MigrateNonBreakingChangeToEnum.java:60) ~[io.airbyte.airbyte-db-db-lib-0.57.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.executeOnce(JavaMigrationExecutor.java:62) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.lambda$execute$0(JavaMigrationExecutor.java:54) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.execute(JavaMigrationExecutor.java:53) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:374) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     ... 20 more
airbyte-bootloader                | Caused by: org.postgresql.util.PSQLException: ERROR: operator does not exist: character varying = non_breaking_change_preference_type
airbyte-bootloader                |   Hint: No operator matches the given name and argument types. You might need to add explicit type casts.
airbyte-bootloader                |   Position: 64
airbyte-bootloader                |     at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2712) ~[postgresql-42.7.1.jar:42.7.1]
airbyte-bootloader                |     at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2400) ~[postgresql-42.7.1.jar:42.7.1]
airbyte-bootloader                |     at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:367) ~[postgresql-42.7.1.jar:42.7.1]
airbyte-bootloader                |     at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498) ~[postgresql-42.7.1.jar:42.7.1]
airbyte-bootloader                |     at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415) ~[postgresql-42.7.1.jar:42.7.1]
airbyte-bootloader                |     at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190) ~[postgresql-42.7.1.jar:42.7.1]
airbyte-bootloader                |     at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:177) ~[postgresql-42.7.1.jar:42.7.1]
airbyte-bootloader                |     at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-5.1.0.jar:?]
airbyte-bootloader                |     at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-5.1.0.jar:?]
airbyte-bootloader                |     at org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:219) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.Tools.executeStatementAndGetFirstResultSet(Tools.java:4848) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:236) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:348) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:301) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.SelectImpl.fetchLazy(SelectImpl.java:3251) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at org.jooq.impl.ResultQueryTrait.lambda$fetchStream$1(ResultQueryTrait.java:329) ~[jooq-3.19.1.jar:?]
airbyte-bootloader                |     at java.base/java.util.stream.AbstractPipeline.sourceSpliterator(AbstractPipeline.java:405) ~[?:?]
airbyte-bootloader                |     at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
airbyte-bootloader                |     at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
airbyte-bootloader                |     at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
airbyte-bootloader                |     at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
airbyte-bootloader                |     at io.airbyte.db.instance.configs.migrations.V0_44_4_002__MigrateNonBreakingChangeToEnum.migrateExistingDisableConnection(V0_44_4_002__MigrateNonBreakingChangeToEnum.java:101) ~[io.airbyte.airbyte-db-db-lib-0.57.1.jar:?]
airbyte-bootloader                |     at io.airbyte.db.instance.configs.migrations.V0_44_4_002__MigrateNonBreakingChangeToEnum.migrate(V0_44_4_002__MigrateNonBreakingChangeToEnum.java:60) ~[io.airbyte.airbyte-db-db-lib-0.57.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.executeOnce(JavaMigrationExecutor.java:62) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.lambda$execute$0(JavaMigrationExecutor.java:54) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.execute(JavaMigrationExecutor.java:53) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:374) ~[flyway-core-10.7.1.jar:?]
airbyte-bootloader                |     ... 20 more
airbyte-bootloader                | INFO StatusConsoleListener Loading mask data from '/seed/specs_secrets_mask.yaml
airbyte-bootloader                | WARN StatusConsoleListener Unable to register Log4j shutdown hook because JVM is shutting down. Using SimpleLogger
@killthekitten
Copy link
Contributor Author

killthekitten commented Apr 5, 2024

My current workaround was to change the column type manually to make migrations pass:

begin;
alter table connection alter column non_breaking_change_preference drop
not null;
alter table connection alter column non_breaking_change_preference drop
default;
alter table connection alter column non_breaking_change_preference TYPE
non_breaking_change_preference_type USING cast(non_breaking_change_preference as "non_breaking_change_preference_type");
alter table connection alter column non_breaking_change_preference set not null;
alter table connection alter column non_breaking_change_preference set default 'ignore'::non_breaking_change_preference_type;
commit;

Make sure to revert the changes after the migration is done (I'm not sure this is the right way):

begin;
alter table connection alter column non_breaking_change_preference drop
not null;
alter table connection alter column non_breaking_change_preference drop
default;
alter table connection alter column non_breaking_change_preference TYPE text;
alter table connection alter column non_breaking_change_preference set not null;
alter table connection alter column non_breaking_change_preference set default 'ignore';
commit;

Update: typos fixed according to comments from @Dirkmal and @linuxs

@marcosmarxm
Copy link
Member

@killthekitten are you trying to migrate directly from 0.44 to latest?

@marcosmarxm marcosmarxm changed the title Migrations from v0.44.2 are broken by the recent refactor [upgrade] Migrations from v0.44.2 are broken by the recent refactor Apr 10, 2024
@killthekitten
Copy link
Contributor Author

@marcosmarxm yes, that's correct, directly from 0.44 to latest!

To be clear, we have already finished the migration using the workaround I posted above.

@adam-bloom
Copy link
Contributor

I hit this as well. thanks for the workaround @killthekitten! I was able to create new connections without applying the workaround "undo", so I skipped that part

@dcondePEAX
Copy link

@killthekitten I had the same issue when updating from 0.44.0 to latest. Solved it by updating to 0.51.0 first, then updating again to the latest version.

@killthekitten
Copy link
Contributor Author

killthekitten commented Apr 26, 2024

@dcondePEAX nice workaround, it haven't crossed my mind. My thinking was that intermediate versions might introduce even more bugs, but good to know it worked!

@Dirkmal
Copy link

Dirkmal commented May 26, 2024

This workaround is a life-saver, I have spent the past 24 hours struggling to bring our airbyte up after updating.

There is a small error in the last alter statement, see correction here:

alter table connection alter column non_breaking_change_preference set default 'ignore'::non_breaking_change_preference_type;

@killthekitten
Copy link
Contributor Author

@Dirkmal thanks for the note, I edited my comment! I typed it out from memory looks like 😅

@happythenewsad
Copy link

Is there a plan to fix the install script, so that users don't have to individually discover this issue and manually address?

@linuxs
Copy link

linuxs commented Sep 13, 2024

begin;
alter table connection alter column non_breaking_change_preference drop
not null;
alter table connection alter column non_breaking_change_preference drop
default;
alter table connection alter column non_breaking_change_preference TYPE
non_breaking_change_preference_type text;
alter table connection alter column non_breaking_change_preference set not null;
alter table connection alter column non_breaking_change_preference set default 'ignore';
commit;

Correct code to undo is:

begin;
alter table connection alter column non_breaking_change_preference drop
not null;
alter table connection alter column non_breaking_change_preference drop
default;
alter table connection alter column non_breaking_change_preference TYPE text;
alter table connection alter column non_breaking_change_preference set not null;
alter table connection alter column non_breaking_change_preference set default 'ignore';
commit;

@killthekitten
Copy link
Contributor Author

@linuxs edited again, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform issues related to the platform community helm step/upgrade team/deployments type/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

8 participants