Skip to content

Commit

Permalink
Nessie server: add support for MariaDB and MySQL backends (#8544)
Browse files Browse the repository at this point in the history
  • Loading branch information
adutra committed May 23, 2024
1 parent c1c97e5 commit 41c5880
Show file tree
Hide file tree
Showing 28 changed files with 601 additions and 59 deletions.
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,21 @@ as necessary. Empty sections will not end in the release notes.
- New Nessie CLI tool + REPL, replacing the old Python based CLI, based on Java.
SQL-ish syntax, built-in online `HELP` command, auto-completion of commands, keywords
and reference names, syntax highlighting, paging of long results, command history.
- Nessie now includes built-in support for MariaDB, with full compatibility with MySQL servers. New
users wishing to try MariaDB (or MySQL) should:
1. Specify the new configuration property: `nessie.version.store.persist.jdbc.datasource=mariadb`;
2. Provide all the MariaDB (or MySQL) connection details using `quarkus.datasource.mariadb.*`
configuration properties.

### Upgrade notes

- Due to the newly-introduced support for MariaDB, existing PostgreSQL users can continue to use
their current JDBC configuration, but are encouraged to update it as follows:
1. Specify the new configuration property:
`nessie.version.store.persist.jdbc.datasource=postgresql`;
2. Migrate any property under `quarkus.datasource.*` to `quarkus.datasource.postgresql.*`. Support
for the old `quarkus.datasource.*` properties will be removed in a future release.

### Breaking changes

- `nessie-quarkus-cli`, the low-level tool to for example export/import Nessie repositories, has been renamed
Expand Down
34 changes: 34 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,32 @@ com.github.jnr:jnr-x86asm
OTHER DEALINGS IN THE SOFTWARE.


---
com.github.waffle:waffle-jna

MIT License

Copyright (c) 2010-2023 The Waffle Project Contributors: https://github.com/Waffle/waffle/graphs/contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


---
com.google.protobuf:protobuf-java

Expand Down Expand Up @@ -1786,6 +1812,14 @@ FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.


---
org.mariadb.jdbc:mariadb-java-client

GNU LESSER GENERAL PUBLIC LICENSE (LGPL) v. 2.1

Copyright (c) 2012-2014 Monty Program Ab
Copyright (c) 2015-2024 MariaDB Corporation Ab

---
postcss

Expand Down
2 changes: 1 addition & 1 deletion gradle/libs.versions.toml
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,8 @@ junit-jupiter-params = { module = "org.junit.jupiter:junit-jupiter-params" }
junit-platform-reporting = { module = "org.junit.platform:junit-platform-reporting" }
logback-classic = { module = "ch.qos.logback:logback-classic", version.ref = "logback" }
keycloak-admin-client = { module = "org.keycloak:keycloak-admin-client", version.ref = "keycloak" }
maven-resolver-supplier = { module = "org.apache.maven.resolver:maven-resolver-supplier", version.ref = "mavenResolver" }
mariadb-java-client = { module = "org.mariadb.jdbc:mariadb-java-client", version = "3.4.0" }
maven-resolver-supplier = { module = "org.apache.maven.resolver:maven-resolver-supplier", version.ref = "mavenResolver" }
micrometer-core = { module = "io.micrometer:micrometer-core", version = "1.13.0" }
microprofile-contextpropagation-api = { module = "org.eclipse.microprofile.context-propagation:microprofile-context-propagation-api", version = "1.3" }
microprofile-openapi = { module = "org.eclipse.microprofile.openapi:microprofile-openapi-api", version = "3.1.1" }
Expand Down
3 changes: 3 additions & 0 deletions gradle/license/allowed-licenses.json
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,9 @@
{
"moduleLicense": "GNU General Public License, Version 2 with the GNU Classpath Exception"
},
{
"moduleLicense": "GNU Lesser General Public License Version 2.1"
},
{
"moduleLicense": "Go License"
},
Expand Down
1 change: 1 addition & 0 deletions gradle/license/normalizer-bundle.json
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@
{ "bundleName" : "lgpl21", "licenseNamePattern" : "GNU LESSER GENERAL PUBLIC LICENSE, Version 2.1" },
{ "bundleName" : "lgpl21", "licenseNamePattern" : "LGPL, version 2.1" },
{ "bundleName" : "lgpl21", "licenseNamePattern" : "LGPL 2.1" },
{ "bundleName" : "lgpl21", "licenseNamePattern" : "LGPL-2.1" },
{ "bundleName" : "lgpl21", "licenseNamePattern" : "lgpl" },

{ "bundleName" : "lgpl3", "licenseNamePattern" : "Lesser General Public License, version 3 or greater" },
Expand Down
1 change: 1 addition & 0 deletions servers/quarkus-common/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ dependencies {
implementation("io.quarkus:quarkus-hibernate-validator")
implementation("io.quarkus:quarkus-agroal")
implementation("io.quarkus:quarkus-jdbc-postgresql")
implementation("io.quarkus:quarkus-jdbc-mariadb")
implementation("io.quarkus:quarkus-opentelemetry")
implementation("io.quarkus:quarkus-micrometer")
implementation(enforcedPlatform(libs.quarkus.amazon.services.bom))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,55 @@

/**
* Setting {@code nessie.version.store.type=JDBC} enables transactional/RDBMS as the version store
* used by the Nessie server. Configuration of the datastore will be done by Quarkus and depends on
* many factors, such as the actual database in use. A complete set of JDBC configuration options
* can be found on <a href="https://quarkus.io/guides/datasource">quarkus.io</a>.
* used by the Nessie server.
*
* <p>Configuration of the datastore will be done by Quarkus and depends on many factors, such as
* the actual database to use. The property {@code nessie.version.store.persist.jdbc.datasource}
* will be used to select one of the built-in datasources; currently supported values are: {@code
* postgresql} (which activates the PostgresQL driver) and {@code mariadb} (which activates the
* MariaDB driver, compatible with MySQL servers).
*
* <p>For example, to configure a PostgresQL connection, the following configuration should be used:
*
* <ul>
* <li>{@code nessie.version.store.type=JDBC}
* <li>{@code nessie.version.store.persist.jdbc.datasource=postgresql}
* <li>{@code quarkus.datasource.postgresql.jdbc.url=jdbc:postgresql://localhost:5432/my_database}
* <li>{@code quarkus.datasource.postgresql.username=<your username>}
* <li>{@code quarkus.datasource.postgresql.password=<your password>}
* <li>Other PostgresQL-specific properties can be set using {@code
* quarkus.datasource.postgresql.*}
* </ul>
*
* <p>To connect to a MariaDB database instead, the following configuration should be used:
*
* <ul>
* <li>{@code nessie.version.store.type=JDBC}
* <li>{@code nessie.version.store.persist.jdbc.datasource=mariadb}
* <li>{@code quarkus.datasource.mariadb.jdbc.url=jdbc:mariadb://localhost:3306/my_database}
* <li>{@code quarkus.datasource.mariadb.username=<your username>}
* <li>{@code quarkus.datasource.mariadb.password=<your password>}
* <li>Other MariaDB-specific properties can be set using {@code quarkus.datasource.mariadb.*}
* </ul>
*
* <p>A complete set of JDBC configuration options can be found on <a
* href="https://quarkus.io/guides/datasource">quarkus.io</a>.
*/
@StaticInitSafe
@ConfigMapping(prefix = "nessie.version.store.persist.jdbc")
public interface QuarkusJdbcConfig extends JdbcBackendBaseConfig {

/**
* The name of the datasource to use. Must correspond to a configured datasource under {@code
* quarkus.datasource.<name>}. Supported values are: {@code postgresql} and {@code mariadb}. If
* not provided, the default Quarkus datasource, defined using the {@code quarkus.datasource.*}
* configuration keys, will be used (the corresponding driver is PostgresQL). Note that it is
* recommended to define "named" JDBC datasources, see <a
* href="https://quarkus.io/guides/datasource#jdbc-configuration">Quarkus JDBC config
* reference</a>.
*/
Optional<String> datasource();

@Override
Optional<String> catalog();

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
/*
* Copyright (C) 2024 Dremio
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.projectnessie.quarkus.config.datasource;

import io.smallrye.config.ConfigSourceInterceptor;
import io.smallrye.config.ConfigSourceInterceptorContext;
import io.smallrye.config.ConfigValue;
import io.smallrye.config.ConfigValue.ConfigValueBuilder;
import org.projectnessie.quarkus.config.VersionStoreConfig.VersionStoreType;
import org.projectnessie.quarkus.providers.storage.JdbcBackendBuilder;

/**
* Activates a data source based on the current Nessie configuration under {@code
* nessie.version.store.persist.jdbc.datasource}, and deactivates all other data sources.
*
* <p>If the version store type is not JDBC, all data sources are deactivated.
*/
public class DataSourceActivator implements ConfigSourceInterceptor {

/**
* The default ordinal to use for modified {@code quarkus.datasource.*.active} properties. It must
* be higher than the ordinal of the application.properties classpath config source (250) but
* lower than the ordinal of the user-supplied application.properties (260).
*/
private static final int DEFAULT_ORDINAL = 251;

private static String activeDataSourceName;
private static VersionStoreType versionStoreType;

@Override
public ConfigValue getValue(ConfigSourceInterceptorContext context, String name) {
ConfigValue value = context.proceed(name);
if (name.startsWith("quarkus.datasource.") && name.endsWith(".active")) {
boolean active =
versionStoreType(context) == VersionStoreType.JDBC
&& dataSourceName(name).equals(activeDataSourceName(context));
if (value == null
|| value.getValue() == null
|| active != Boolean.parseBoolean(value.getValue())) {
value = newConfigValue(value, active);
}
}
return value;
}

private static ConfigValue newConfigValue(ConfigValue current, boolean active) {
ConfigValueBuilder builder = current == null ? ConfigValue.builder() : current.from();
int ordinal = current == null ? DEFAULT_ORDINAL : current.getConfigSourceOrdinal() + 1;
return builder.withConfigSourceOrdinal(ordinal).withValue(active ? "true" : "false").build();
}

private static synchronized VersionStoreType versionStoreType(
ConfigSourceInterceptorContext context) {
if (versionStoreType == null) {
ConfigValue value = context.proceed("nessie.version.store.type");
versionStoreType =
value == null || value.getValue() == null
? VersionStoreType.IN_MEMORY
: VersionStoreType.valueOf(value.getValue());
}
return versionStoreType;
}

private static String dataSourceName(String property) {
if (property.equals("quarkus.datasource.active")) {
return JdbcBackendBuilder.DEFAULT_DATA_SOURCE_NAME;
}
String dataSourceName =
property.substring("quarkus.datasource.".length(), property.length() - ".active".length());
return JdbcBackendBuilder.unquoteDataSourceName(dataSourceName);
}

private static synchronized String activeDataSourceName(ConfigSourceInterceptorContext context) {
if (activeDataSourceName == null) {
ConfigValue value = context.proceed("nessie.version.store.persist.jdbc.datasource");
activeDataSourceName =
value == null || value.getValue() == null
? JdbcBackendBuilder.DEFAULT_DATA_SOURCE_NAME
: JdbcBackendBuilder.unquoteDataSourceName(value.getValue());
}
return activeDataSourceName;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -17,43 +17,114 @@

import static org.projectnessie.quarkus.config.VersionStoreConfig.VersionStoreType.JDBC;

import io.agroal.api.AgroalDataSource;
import io.quarkus.agroal.runtime.UnconfiguredDataSource;
import io.quarkus.arc.All;
import io.quarkus.arc.InstanceHandle;
import io.quarkus.datasource.common.runtime.DatabaseKind;
import io.quarkus.datasource.runtime.DataSourceBuildTimeConfig;
import io.quarkus.datasource.runtime.DataSourcesBuildTimeConfig;
import jakarta.annotation.PostConstruct;
import jakarta.enterprise.context.Dependent;
import jakarta.inject.Inject;
import org.eclipse.microprofile.config.inject.ConfigProperty;
import java.util.List;
import javax.sql.DataSource;
import org.projectnessie.quarkus.config.QuarkusJdbcConfig;
import org.projectnessie.quarkus.providers.versionstore.StoreType;
import org.projectnessie.versioned.storage.common.persist.Backend;
import org.projectnessie.versioned.storage.jdbc.JdbcBackendConfig;
import org.projectnessie.versioned.storage.jdbc.JdbcBackendFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@StoreType(JDBC)
@Dependent
public class JdbcBackendBuilder implements BackendBuilder {

@SuppressWarnings("CdiInjectionPointsInspection")
@Inject
AgroalDataSource dataSource;
/**
* The name of the default datasource. Corresponds to the default map key in {@link
* DataSourcesBuildTimeConfig#dataSources()}.
*/
public static final String DEFAULT_DATA_SOURCE_NAME = "<default>";

private static final Logger LOGGER = LoggerFactory.getLogger(JdbcBackendBuilder.class);

@Inject DataSourcesBuildTimeConfig dataSourcesConfig;

@Inject
@ConfigProperty(name = "quarkus.datasource.db-kind")
String databaseKind;
@All
@SuppressWarnings("CdiInjectionPointsInspection")
List<InstanceHandle<DataSource>> dataSources;

@Inject QuarkusJdbcConfig config;

@PostConstruct
public void checkDataSourcesConfiguration() {
dataSourcesConfig.dataSources().forEach(this::checkDatabaseKind);
}

@Override
public Backend buildBackend() {
if (!DatabaseKind.isPostgreSQL(databaseKind) && !DatabaseKind.isH2(databaseKind)) {
DataSource dataSource = selectDataSource();
JdbcBackendConfig c = JdbcBackendConfig.builder().from(config).dataSource(dataSource).build();
return new JdbcBackendFactory().buildBackend(c);
}

public static String unquoteDataSourceName(String dataSourceName) {
if (dataSourceName.startsWith("\"") && dataSourceName.endsWith("\"")) {
dataSourceName = dataSourceName.substring(1, dataSourceName.length() - 1);
}
return dataSourceName;
}

private void checkDatabaseKind(String dataSourceName, DataSourceBuildTimeConfig config) {
if (config.dbKind().isEmpty()) {
throw new IllegalArgumentException(
"Database kind not configured for datasource " + dataSourceName);
}
String databaseKind = config.dbKind().get();
if (!DatabaseKind.isPostgreSQL(databaseKind)
&& !DatabaseKind.isH2(databaseKind)
&& !DatabaseKind.isMariaDB(databaseKind)) {
throw new IllegalArgumentException(
"Database kind is configured to '"
"Database kind for datasource "
+ dataSourceName
+ " is configured to '"
+ databaseKind
+ "', which Nessie does not support yet, PostgreSQL, H2, MariaDb and MySQL(via mariaDb driver) are supported. "
+ "', which Nessie does not support yet; "
+ "currently PostgreSQL, H2, MariaDB (and MySQL via MariaDB driver) are supported. "
+ "Feel free to raise a pull request to support your database of choice.");
}
}

JdbcBackendFactory factory = new JdbcBackendFactory();
JdbcBackendConfig c = JdbcBackendConfig.builder().from(config).dataSource(dataSource).build();
return factory.buildBackend(c);
private DataSource selectDataSource() {
String dataSourceName =
config
.datasource()
.map(JdbcBackendBuilder::unquoteDataSourceName)
.orElse(DEFAULT_DATA_SOURCE_NAME);
DataSource dataSource = findDataSourceByName(dataSourceName);
if (dataSource instanceof UnconfiguredDataSource e) {
e.throwException();
}
if (dataSourceName.equals(DEFAULT_DATA_SOURCE_NAME)) {
LOGGER.warn(
"Using legacy datasource configuration under quarkus.datasource.*: "
+ "please migrate to quarkus.datasource.postgresql.* and "
+ "set nessie.version.store.persist.jdbc.datasource=postgresql");
} else {
LOGGER.info("Selected datasource: {}", dataSourceName);
}
return dataSource;
}

private DataSource findDataSourceByName(String dataSourceName) {
for (InstanceHandle<DataSource> handle : dataSources) {
String name = handle.getBean().getName();
name = name == null ? DEFAULT_DATA_SOURCE_NAME : unquoteDataSourceName(name);
if (name.equals(dataSourceName)) {
return handle.get();
}
}
throw new IllegalStateException("No datasource configured with name: " + dataSourceName);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@
#

org.projectnessie.quarkus.config.OpenTelemetryConfigSourceInterceptor
org.projectnessie.quarkus.config.datasource.DataSourceActivator
Loading

0 comments on commit 41c5880

Please sign in to comment.