Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added support for non-transactional SQL migrations #1027

Closed
wants to merge 2 commits into from

Conversation

Omie
Copy link

@Omie Omie commented May 29, 2015

fixes #1026

  • added new Resolver and Executor classes
  • NTSqlMigrationResolver works on the files prefixed with "NTV", currently hardcoded. (Looking forward for suggestions over this)
  • Added the new resolver in migrationResolvers while creating CompositeMigrationResolver instance
  • added small test for new resolver

Omie added 2 commits May 28, 2015 17:00
- added new Resolver and Executor classes
- NTSqlMigrationResolver works on the files prefixed with "NTV",
  currently hardcoded

- added small test for new resolver
@ccouturi
Copy link

ccouturi commented Jun 5, 2015

+1 for this feature.

@eemmiirr
Copy link

+1

4 similar comments
@Nthalk
Copy link

Nthalk commented Jul 31, 2015

+1

@0x6e6562
Copy link

+1

@baekdahl
Copy link

+1

@dannybusch
Copy link

+1

@dannybusch
Copy link

when will this feature be merged into flyway?

@0x6e6562
Copy link

For anybody that is looking for a quick and dirty way to mix in some non-transactional migrations into an existing SQL-based patch series, I knocked out the following shim:

public abstract class AbstractResolvedMigration implements ResolvedMigration {

  private final String version;
  private final String description;

  public AbstractResolvedMigration() {

    String className = this.getClass().getSimpleName();
    String template = "V(\\d+)__(.*)";
    Pattern pattern = Pattern.compile(template);
    Matcher matcher = pattern.matcher(className);
    if (matcher.matches()) {
      this.version = matcher.group(1);
      this.description = matcher.group(2).replaceAll("_", " ");
    } else {
      throw new RuntimeException("Bogus migration name: " + className);
    }

  }

  @Override
  public MigrationVersion getVersion() {
    return MigrationVersion.fromVersion(version);
  }

  @Override
  public String getDescription() {
    return description;
  }

  @Override
  public Integer getChecksum() {
    final CRC32 crc32 = new CRC32();
    crc32.update(getScript().getBytes());
    return (int) crc32.getValue();
  }

  @Override
  public MigrationType getType() {
    return MigrationType.CUSTOM;
  }

  @Override
  public String getPhysicalLocation() {
    return "virtual://loaction";
  }

  @Override
  public MigrationExecutor getExecutor() {
    return new MigrationExecutor() {

      @Override
      public void execute(Connection connection) {
        PreparedStatement stmt;
        try {
          stmt = connection.prepareStatement(getScript());
          stmt.execute();
        } catch (SQLException e) {
          throw new RuntimeException(e);
        }
      }

      @Override
      public boolean executeInTransaction() {
        return false;
      }

    };
  }
}

And then if I need to execute something outside of a TX, I subclass the base class to supply the patch specific SQL. For example, say I have a V50 SQL migration that requires a new type in Postgres, I create a V49 custom migration that patches the DDL in a non-TX migration in order that the subsequent V50 migration that depends on the new type can execute:

public class V49__some_meaningful_description extends AbstractResolvedMigration {

  @Override
  public String getScript() {
    return "ALTER TYPE foo ADD VALUE 'BAR' AFTER 'BAZ';";
  }

}

And then to mix the V49 migration into the execution series, you need to add a bit of resolver glue and then tell Flyway about it at runtime:

public class NonTransactionalMigrationResolver implements MigrationResolver {

  @Override
  public List<ResolvedMigration> resolveMigrations() {

    List<ResolvedMigration> resolvedMigrations = new ArrayList<>();

    resolvedMigrations.add(new V49__some_meaningful_description());
    // add any other non-TX migrations to this list

    return resolvedMigrations;
  }
}

Hope this can help somebody until such a time that there is a less hacky way to do this kind of thing.

@hendley
Copy link

hendley commented Oct 14, 2015

+1

2 similar comments
@eemmiirr
Copy link

+1

@mjallday
Copy link

👍

@tomdcc
Copy link

tomdcc commented Nov 11, 2015

+1

@axelfontaine Is there anything we can do to help get this PR in shape to be merged?

@akopts
Copy link

akopts commented Dec 1, 2015

+1

5 similar comments
@mr-zhukov
Copy link

+1

@skrasovsky
Copy link

+1

@skozlov
Copy link

skozlov commented Dec 1, 2015

+1

@irwinm
Copy link

irwinm commented Dec 1, 2015

+1

@hhware
Copy link

hhware commented Dec 22, 2015

+1

@brckner
Copy link

brckner commented Feb 22, 2016

When would this feature be merged into master? We have currently issues too, where we need migrations to run in an non transactional context.

@falschparker82
Copy link

+1

1 similar comment
@karlvlam
Copy link

+1

@karlvlam
Copy link

any updates for this issue?

@anthonylau
Copy link

+1

2 similar comments
@VLMH
Copy link

VLMH commented May 18, 2016

+1

@jjbubudi
Copy link

+1

@lukasniemeier-zalando
Copy link

@axelfontaine dies this change has a Chance of being merged?

@morero
Copy link

morero commented Jun 8, 2016

+1

@j-lindblom
Copy link

@axelfontaine Would you consider merging this PR?

@Trekoid
Copy link

Trekoid commented Jul 21, 2016

I wouldn't want non-transactional migrations to have a different runtime order than the regular V1_Description.sql files. I would prefer something like V53_NT__Description.sql or V53__NT_Description.sql so they would run in line with other transactional migrations.

@Omie
Copy link
Author

Omie commented Jul 22, 2016

@Trekoid this patch does maintain the order. V1_foo.sql, NTV2_bar.sql and V3_baz.sql files will get executed in foo, bar, baz order.

On the other note, this PR is vastly out of date, there are hundreds of commits in between and this will probably require work from scratch.

@davidvuong
Copy link

Hey @Omie,

Just to add to @Trekoid's comment:

My IDE orders files alphanumerically and having it named NTVxx when all other flyway migrations areVxx makes it a little hard to navigate. I understand that flyway will run these migrations in order but editors won't display them in order (which can be a bit difficult to grok if you have lots of migrations).

Just my 2 cents. Awesome PR BTW. I'd love to see this finally get merged soon.

@Trekoid
Copy link

Trekoid commented Nov 28, 2016

@davidvuong your should be happy to hear the company I work for co-sponsored the development of non-transactional migrations with BoxFuse, so expect it soon in release 4.1. The solution @axelfontaine and the team at BoxFuse created requires no special file naming. It will auto detect the migration and run it appropriately.

@davidvuong
Copy link

davidvuong commented Nov 29, 2016

@Trekoid That's awesome. Looking forward to it!

@brckner
Copy link

brckner commented Nov 30, 2016

Hey, I've also worked on a non transactional resolver for flyway. It scans for sql files with suffix .ntx.sql and executes them in a nontransactional context. It works only with flyway maven plugin... so you can add it as a dependency in maven pom and add the custom resolver in the flyway-maven-plugin. if anyone is interested, i'll put it on gitlab and maven central.

@arturaz
Copy link

arturaz commented Jan 30, 2017

Anything?

@axelfontaine
Copy link
Contributor

This has been implemented differently using auto-detection (PostgreSQL only for now). Closing.

@mohammad-sk
Copy link

mohammad-sk commented Mar 2, 2021

@andrewlmurray Pls help
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-03-02 17:21:39.233 ERROR [identity-service,,,] 5621 --- [ main] o.s.boot.SpringApplication : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Unable to parse statement in db/migration/V1_1__Identity_Service_dml.sql at line 1 col 1: No value provided for placeholder: ${USER_NAME}. Check your configuration!
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1108) ~[spring-context-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:868) ~[spring-context-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) ~[spring-boot-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
.service.identity.ServiceApplication.main(ServiceApplication.java:28) ~[classes!/:sp09-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) ~[identity-service-sp09-SNAPSHOT.jar:sp09-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) ~[identity-service-sp09-SNAPSHOT.jar:sp09-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) ~[identity-service-sp09-SNAPSHOT.jar:sp09-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
Caused by: org.flywaydb.core.api.FlywayException: Unable to parse statement in db/migration/V1_1__Identity_Service_dml.sql at line 1 col 1: No value provided for placeholder: ${USER_NAME}. Check your configuration!
at org.flywaydb.core.internal.parser.Parser.getNextStatement(Parser.java:284) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.parser.Parser.access$000(Parser.java:44) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.parser.Parser$ParserSqlStatementIterator.(Parser.java:627) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.parser.Parser.parse(Parser.java:116) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.sqlscript.ParserSqlScript.parse(ParserSqlScript.java:76) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.sqlscript.ParserSqlScript.validate(ParserSqlScript.java:124) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.sqlscript.ParserSqlScript.executeInTransaction(ParserSqlScript.java:187) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.canExecuteInTransaction(SqlMigrationExecutor.java:75) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.command.DbMigrate.isExecuteGroupInTransaction(DbMigrate.java:312) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:275) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:244) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.command.DbMigrate.access$100(DbMigrate.java:54) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.command.DbMigrate$2.call(DbMigrate.java:162) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.command.DbMigrate$2.call(DbMigrate.java:159) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.database.sqlserver.SQLServerApplicationLockTemplate.execute(SQLServerApplicationLockTemplate.java:62) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.database.sqlserver.SQLServerConnection.lock(SQLServerConnection.java:87) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:139) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:159) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:137) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.Flyway$1.execute(Flyway.java:189) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.Flyway$1.execute(Flyway.java:149) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.Flyway.execute(Flyway.java:511) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.Flyway.migrate(Flyway.java:149) ~[flyway-core-6.0.8.jar!/:na]
at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:65) ~[spring-boot-autoconfigure-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1855) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1792) ~[spring-beans-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
... 26 common frames omitted
Caused by: org.flywaydb.core.api.FlywayException: No value provided for placeholder: ${USER_NAME}. Check your configuration!
at org.flywaydb.core.internal.parser.PlaceholderReplacingReader.read(PlaceholderReplacingReader.java:105) ~[flyway-core-6.0.8.jar!/:na]
at java.base/java.io.FilterReader.read(FilterReader.java:65) ~[na:na]
at org.flywaydb.core.internal.parser.RecordingReader.read(RecordingReader.java:33) ~[flyway-core-6.0.8.jar!/:na]
at java.base/java.io.FilterReader.read(FilterReader.java:65) ~[na:na]
at org.flywaydb.core.internal.parser.PeekingReader.refillPeekBuffer(PeekingReader.java:70) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.parser.PeekingReader.peek(PeekingReader.java:59) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.parser.PeekingReader.peekWhitespace(PeekingReader.java:123) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.parser.PeekingReader.readWhitespace(PeekingReader.java:428) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.parser.Parser.readToken(Parser.java:484) ~[flyway-core-6.0.8.jar!/:na]
at org.flywaydb.core.internal.parser.Parser.getNextStatement(Parser.java:162) ~[flyway-core-6.0.8.jar!/:na]
... 51 common frames omitted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

support non transactional migrations in SQL api