Apache Phoenix Support

[jmahonin]

Updated PR, reference issue:
#929

[anirudha13]

@jmahonin I got the failing unit test, MigrationTestCase#validateClean to pass by renaming the file, PhoenixCheckValidate1__First.sql to CheckValidate1__First.sql . Since the sqlMigrationPrefix is set as "CheckValidate" in the unit test.

[jmahonin]

@anirudha13 Thanks. That's likely the only change in the new PR, but I realized I pushed the last one in error. Can you verify the new one if you have a chance?

[anirudha13]

@jmahonin
Attaching a diff with the changes that I made to make sure that unit test passed, along with the other unit tests as well. Just look at the changes to files, flyway-core/src/test/java/org/flywaydb/core/migration/MigrationTestCase.java and flyway-core/src/test/java/org/flywaydb/core/internal/dbsupport/phoenix/PhoenixMigrationMediumTest.java

diff --git a/flyway-core/src/test/java/org/flywaydb/core/internal/dbsupport/phoenix/PhoenixMigrationMediumTest.java b/flyway-core/src/test/java/org/flywaydb/core/internal/db
index 7be4e16..9cce0d3 100644
--- a/flyway-core/src/test/java/org/flywaydb/core/internal/dbsupport/phoenix/PhoenixMigrationMediumTest.java
+++ b/flyway-core/src/test/java/org/flywaydb/core/internal/dbsupport/phoenix/PhoenixMigrationMediumTest.java
@@ -52,11 +52,6 @@ public class PhoenixMigrationMediumTest extends MigrationTestCase {
         return "migration/dbsupport/phoenix/sql/quote";
     }

-    @Override
-    protected String getCheckValidateSqlPrefix() {
-        return "PhoenixCheckValidate";
-    }
-
     @BeforeClass
     public static void beforeClassSetUp() throws Exception {
         // Startup HBase in-memory cluster
@@ -215,4 +210,4 @@ public class PhoenixMigrationMediumTest extends MigrationTestCase {
     public void setCurrentSchema() throws Exception {}


-}
+}
\ No newline at end of file
diff --git a/flyway-core/src/test/java/org/flywaydb/core/migration/MigrationTestCase.java b/flyway-core/src/test/java/org/flywaydb/core/migration/MigrationTestCase.java
index 5d41d45..500e3b5 100644
--- a/flyway-core/src/test/java/org/flywaydb/core/migration/MigrationTestCase.java
+++ b/flyway-core/src/test/java/org/flywaydb/core/migration/MigrationTestCase.java
@@ -263,14 +263,10 @@ public abstract class MigrationTestCase {

         flyway.setValidateOnMigrate(true);
         flyway.setCleanOnValidationError(true);
-        flyway.setSqlMigrationPrefix(getCheckValidateSqlPrefix());
+        flyway.setSqlMigrationPrefix("CheckValidate");
         assertEquals(1, flyway.migrate());
     }

-    protected String getCheckValidateSqlPrefix() {
-        return "CheckValidate";
-    }
-
     @Test
     public void failedMigration() throws Exception {
         String tableName = "before_the_error";

[nycjay]

Any idea when Phoenix support will make it into a Flyway release?

[jmahonin]

I've just re-upped this PR with a new version that fixes unit tests, and uses an updated Phoenix / HBase version for the tests as well.

@anirudha13 @nycjay If you guys can give this branch a test drive and see if it works for you, that'd be awesome.

@axelfontaine Please let me know if you'd like any changes made

[nycjay]

@jmahonin thanks. still have a bit of setup work to do on our end, just getting started with phoenix. will reply back when we a chance to test, but it may be a couple of weeks.

[axelfontaine]

Any updates on this?

[jmahonin]

We've been using this in production for months without issue, and the PR has been ready to go for months. It looks like there's some new merge conflicts that we need to resolve though. Do you have any specific comments with the PR?

[axelfontaine]

Thanks. Could you make it mergeable again?

Also, a few things seem problematic:

• why can't Phoenix tests be run by default?
• why did you need to make changes to MetaDataTableImpl? Is it only for the update checksum stuff? Doesn't the existing syntax work?
• why the changes in MigrationTestCase?
• why did you duplicate all sql files in src/test/resources/migration/dbsupport?

Cheers
Axel

[jmahonin]

I'll work on the merge issues.

Issue responses:

1. I think previously the Phoenix tests were being run in the 'test' phase, but setting up the in-memory HBase cluster is a bit slow and computationally expensive, so I had them disabled by default. It appears now they are running in the 'verify' phase, so running by default sounds good to me, if you'd prefer that.

2. The MetaDataTableImpl changes are specifically to support Phoenix syntax. It doesn't support UPDATE, only UPSERT. All of the changes in the core files are to support loading a modified syntax.

3. Most of the changes in the MigrationTestCase don't appear to be needed anymore, now that there are overridable paths, however I have a caveat in the last paragraph.

4. None of the dbsupport files are "duplicated", they're all copied and modified to work with Phoenix syntax. If I recall, mostly it was things like replacing UPDATE with UPSERT, modifying PRIMARY KEY syntax, and changing data types where needed (INT -> INTEGER).

While I have your attention though, on the first iteration of this PR, I was getting a "SCHEMA" applied migration type for the first entry of the migration table, and then subsequent "SQL" types after that for the real migrations. If I recall, it's triggered by coding around the fact that a schema doesn't exist in Phoenix until a table is created. Everything continued to work properly with this "SCHEMA" entry, although I had to override several of the unit tests that assumed the first entry was 'SQL'.

I'm curious if this "SCHEMA" migration type is expected behaviour, or is this something that I should try and avoid having completely? I couldn't really work out the reasoning behind if from the code and comments alone.

[axelfontaine]

1. db tests are being run in the "integration-test" phase
2. maybe this should be abstracted by dbsupport instead
3. ok
4. ok

SCHEMA is used when Flyway creates the schema. It is absent when the schema already existed before Flyway ran the first time.

[jmahonin]

Back to mergeable.

Re: 1, I've left it in its own profile. The integration tests take around 5m to run, but if you'd like them in the EmbeddedDBTest profile, let me know

Re: 2, I can try roll that into this patch if you'd prefer as well.

[ndimiduk]

Bump. Any activity here? I'd love to make use of this.

[axelfontaine]

Thank you Josh for all the work you put into this! Merged.

[jmahonin]

Thanks @axelfontaine !

Let me know if there's any further cleanup work you like done. It looks like you've just merged the docs branch in as well, I'll take a once-over to see if there's anything changed there as well, but I suspect it's fine.

[axelfontaine]

Is there a single jar (no transitive deps) for the jdbc driver that could be shipped out of the box with the Flyway command-line tool?

[jmahonin]

There is an assembly client JAR which is self-contained, but not published to maven, available from the Phoenix downloads page [1].

Unfortunately, the client JAR must always be at least the same version or newer than the corresponding server JAR, and to add to the complexity, there are three supported versions of HBase, each with their own client JAR.

As well, the Phoenix release cycle is pretty active, so it may be difficult to ship up-to-date versions. Would it be possible instead to package installation instructions specifically for the Phoenix JAR? Most Phoenix users are pretty accustomed to grabbing and moving around the JAR files as necessary anyway.

[1] https://phoenix.apache.org/download.html