-
Notifications
You must be signed in to change notification settings - Fork 123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MySQL support for Nessie(JDBC based) #8483
Conversation
Hi @vyj7 thanks for contributing to Nessie! It turns out, I was myself planning to introduce support for MySQL. I'm afraid this will require a lot more changes compared to what you have in this PR. Are you willing to continue working on this (in which case, I could give you some pointers), or should I take it over from here? Just le me know :-) Thanks! (Note: you also would need to sign our CLA.) |
@adutra Yes I would love to work on this. Can you guide me what else I am missing.If you can give me a small walkthrough or doc how code is structured i can pick up |
Hi @vyj7 we're ready to work on this. We decided that it's not safe for Nessie to be distributed with the MySQL driver. But there is one thing we can do: offer support for MariaDB instead, and use the MariaDB driver to connect to either MariaDB or MySQL. So, if you are still OK to work on this, here are the first steps:
mariadb-java-client = { module = "org.mariadb.jdbc:mariadb-java-client", version = "3.3.3" }
{
"moduleLicense": "GNU Lesser General Public License Version 2.1"
},
{ "bundleName" : "lgpl21", "licenseNamePattern" : "LGPL-2.1" },
intTestRuntimeOnly(libs.mariadb.java.client)
static class MariaDBDatabaseSpecific implements DatabaseSpecific {
private static final String VARCHAR = "VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin";
private static final String TEXT = "TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin";
private static final String MYSQL_CONSTRAINT_VIOLATION_SQL_STATE = "23000";
private final Map<JdbcColumnType, String> typeMap;
private final Map<JdbcColumnType, Integer> typeIdMap;
MariaDBDatabaseSpecific() {
typeMap = new EnumMap<>(JdbcColumnType.class);
typeIdMap = new EnumMap<>(JdbcColumnType.class);
typeMap.put(JdbcColumnType.NAME, VARCHAR);
typeIdMap.put(JdbcColumnType.NAME, Types.VARCHAR);
typeMap.put(JdbcColumnType.OBJ_ID, VARCHAR);
typeIdMap.put(JdbcColumnType.OBJ_ID, Types.VARCHAR);
typeMap.put(JdbcColumnType.OBJ_ID_LIST, TEXT);
typeIdMap.put(JdbcColumnType.OBJ_ID_LIST, Types.VARCHAR);
typeMap.put(JdbcColumnType.BOOL, "BIT(1)");
typeIdMap.put(JdbcColumnType.BOOL, Types.BIT);
typeMap.put(JdbcColumnType.VARBINARY, "BLOB");
typeIdMap.put(JdbcColumnType.VARBINARY, Types.BLOB);
typeMap.put(JdbcColumnType.BIGINT, "BIGINT");
typeIdMap.put(JdbcColumnType.BIGINT, Types.BIGINT);
typeMap.put(JdbcColumnType.VARCHAR, VARCHAR);
typeIdMap.put(JdbcColumnType.VARCHAR, Types.VARCHAR);
}
@Override
public Map<JdbcColumnType, String> columnTypes() {
return typeMap;
}
@Override
public Map<JdbcColumnType, Integer> columnTypeIds() {
return typeIdMap;
}
@Override
public boolean isConstraintViolation(SQLException e) {
return MYSQL_CONSTRAINT_VIOLATION_SQL_STATE.equals(e.getSQLState());
}
@Override
public String wrapInsert(String sql) {
return sql.replace("INSERT INTO", "INSERT IGNORE INTO");
}
} Once you're done we can tackle the tests. Good luck :-) |
@adutra yes I think mariaDB driver should also work for mysql. I will work on this. |
@vyj7 I think you accidentally closed this PR? |
I removed changes done for mysql. I will push changes on this branch only. |
@adutra I couldn't find gradle/license/allowed-licenses.json and gradle/license/allowed-licenses.json in project |
@vyj7 for the tests, here's what you need to do:
|
...d/storage/jdbc/src/main/java/org/projectnessie/versioned/storage/jdbc/DatabaseSpecifics.java
Show resolved
Hide resolved
Also: |
Not in this PR please :-) Let's tackle just the persist layer, the rest is much more involved and will be done in a follow-up PR. |
Support to detect JDBC backend for mysql and mariaDb
...kus-common/src/main/java/org/projectnessie/quarkus/providers/storage/JdbcBackendBuilder.java
Outdated
Show resolved
Hide resolved
@adutra how should i run this code in order to test it with a local mysql/maria-Db image. and then ran task in nessie Quarkus - quarkusRun |
@vyj7 you don't need to run the server, all you need to do is run the JDBC persistence tests:
If the build passes once you've added the new tests, then you are done :-) |
Added TCs for maria DB and mySQL
@adutra I have committed the TCs. All the testcases actually failed for mariaDB(TCs are failing) and mySQL(driver not found). I am debugging what are the causes of it. So far the test containers are spawning correctly. |
...s/src/main/java/org/projectnessie/versioned/storage/jdbctests/MariaDbBackendTestFactory.java
Outdated
Show resolved
Hide resolved
...sts/src/main/java/org/projectnessie/versioned/storage/jdbctests/MySQLBackendTestFactory.java
Outdated
Show resolved
Hide resolved
...kus-common/src/main/java/org/projectnessie/quarkus/providers/storage/JdbcBackendBuilder.java
Outdated
Show resolved
Hide resolved
@adutra Caused by: java.lang.ClassNotFoundException: org.projectnessie.nessie.relocated.protobuf.UnsafeByteOperations |
Modified MySQL testcontainer to use mariaDb driver
Yes this happens sometimes in your IDE. You need to build the
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@vyj7 You have a few test failures in AbstractTestJdbcBackendFactory.incompatibleTableSchema()
.
I think you can easily fix them by replacing all occurrences of VARCHAR
with VARCHAR(255)
.
...sts/src/main/java/org/projectnessie/versioned/storage/jdbctests/MySQLBackendTestFactory.java
Outdated
Show resolved
Hide resolved
Modified AbstractTestJdbcBackendFactory varchar to varchar(255)
@adutra All testcases passed after the changes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you so much for your contribution @vyj7 !
Pleasure is mine. Really enjoyed working on this. :-) |
Added support to use mySQL database for nessie.