Skip to content

Commit

Permalink
Added native support for Kudu extension apache#1097
Browse files Browse the repository at this point in the history
  • Loading branch information
aldettinger committed Apr 22, 2020
1 parent 1abe10c commit 612f7e5
Show file tree
Hide file tree
Showing 20 changed files with 522 additions and 91 deletions.
3 changes: 2 additions & 1 deletion .github/workflows/pr-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -206,11 +206,12 @@ jobs:
sap-netweaver
servicenow
slack
- category: SQL
- category: Database
test-modules: >
couchdb
influxdb
jdbc
kudu
mongodb
sql
- category: Social
Expand Down
4 changes: 2 additions & 2 deletions docs/modules/ROOT/pages/list-of-camel-quarkus-extensions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -410,8 +410,8 @@ Level | Since | Description
Stable | 1.0.0-M6 | The Kubernetes Services component provides a producer to execute service operations and a consumer to consume service events.

| link:https://camel.apache.org/components/latest/kudu-component.html[Kudu] (camel-quarkus-kudu) +
`kudu:host:port/tableName` | JVM +
Preview | 1.0.0-M6 | Represents a Kudu endpoint. A kudu endpoint allows you to interact with Apache Kudu, a free and open source column-oriented data store of the Apache Hadoop ecosystem.
`kudu:host:port/tableName` | Native +
Stable | 1.0.0-M6 | Represents a Kudu endpoint. A kudu endpoint allows you to interact with Apache Kudu, a free and open source column-oriented data store of the Apache Hadoop ecosystem.

| link:https://camel.apache.org/components/latest/log-component.html[Log] (camel-quarkus-log) +
`log:loggerName` | Native +
Expand Down

This file was deleted.

2 changes: 2 additions & 0 deletions extensions-jvm/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@
<module>google-pubsub</module>
<module>groovy</module>
<module>grpc</module>
<module>influxdb</module>
<module>kubernetes</module>
<module>kudu</module>
<module>mongodb-gridfs</module>
<module>nitrite</module>
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,10 @@
package org.apache.camel.quarkus.component.kudu.deployment;

import io.quarkus.deployment.annotations.BuildStep;
import io.quarkus.deployment.annotations.ExecutionTime;
import io.quarkus.deployment.annotations.Record;
import io.quarkus.deployment.builditem.ExtensionSslNativeSupportBuildItem;
import io.quarkus.deployment.builditem.FeatureBuildItem;
import io.quarkus.deployment.pkg.steps.NativeBuild;
import org.apache.camel.quarkus.core.JvmOnlyRecorder;
import org.jboss.logging.Logger;

class KuduProcessor {
private static final Logger LOG = Logger.getLogger(KuduProcessor.class);

private static final String FEATURE = "camel-kudu";

Expand All @@ -34,14 +29,9 @@ FeatureBuildItem feature() {
return new FeatureBuildItem(FEATURE);
}

/**
* Remove this once this extension starts supporting the native mode.
*/
@BuildStep(onlyIf = NativeBuild.class)
@Record(value = ExecutionTime.RUNTIME_INIT)
void warnJvmInNative(JvmOnlyRecorder recorder) {
JvmOnlyRecorder.warnJvmInNative(LOG, FEATURE); // warn at build time
recorder.warnJvmInNative(FEATURE); // warn at runtime
@BuildStep
ExtensionSslNativeSupportBuildItem activateSslNativeSupport() {
return new ExtensionSslNativeSupportBuildItem(FEATURE);
}

}
1 change: 0 additions & 1 deletion extensions-jvm/kudu/pom.xml → extensions/kudu/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,5 @@
<modules>
<module>deployment</module>
<module>runtime</module>
<module>integration-test</module>
</modules>
</project>
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,13 @@

---
name: "Kudu"
description: "Represents a Kudu endpoint. A kudu endpoint allows you to interact with Apache Kudu, a free and open source column-oriented data store of the Apache Hadoop ecosystem."
description: "Offer interaction with Apache Kudu, a free and open source column-oriented data store of the Apache Hadoop ecosystem."
metadata:
unlisted: true
keywords:
- "camel"
- "cloud"
- "database"
- "iot"
guide: "https://camel.apache.org/components/latest/kudu-component.html"
guide: "https://quarkus.io/guides/camel"
categories:
- "integration"
status: "preview"
1 change: 1 addition & 0 deletions extensions/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@
<module>kafka</module>
<module>kotlin</module>
<module>kubernetes</module>
<module>kudu</module>
<module>log</module>
<module>lzf</module>
<module>mail</module>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-kudu-parent</artifactId>
<artifactId>camel-quarkus-integration-tests</artifactId>
<version>1.1.0-SNAPSHOT</version>
</parent>

<artifactId>camel-quarkus-kudu-integration-test</artifactId>
<name>Camel Quarkus :: Kudu :: Integration Test</name>
<artifactId>camel-quarkus-integration-test-kudu</artifactId>
<name>Camel Quarkus :: Integration Test :: Kudu</name>
<description>Integration tests for Camel Quarkus Kudu extension</description>

<properties>
Expand All @@ -39,14 +39,38 @@
<mvnd.builder.rule>camel-quarkus-kudu-deployment,camel-quarkus-support-policy-deployment</mvnd.builder.rule>
</properties>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-bom-test</artifactId>
<version>${project.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<dependencies>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-resteasy</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-kudu</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-resteasy</artifactId>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-bean</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-direct</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-support-commons-logging</artifactId>
</dependency>

<!-- test dependencies -->
Expand All @@ -60,6 +84,13 @@
<artifactId>rest-assured</artifactId>
<scope>test</scope>
</dependency>

<!-- test dependencies - camel-quarkus -->
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-integration-testcontainers-support</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand All @@ -77,4 +108,33 @@
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>native</id>
<activation>
<property>
<name>native</name>
</property>
</activation>
<properties>
<quarkus.package.type>native</quarkus.package.type>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.camel.quarkus.component.kudu.it;

import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.net.InetAddress;

import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.event.Observes;

import io.quarkus.runtime.StartupEvent;
import org.jboss.logging.Logger;

/**
* In order to run Kudu integration tests, {@KuduResource}, {@code KuduTest} and {@code KuduIT} should have access to:
* 1) A Kudu master server needed to create a table and also to obtain the host/port of the associated tablet server
* 2) A Kudu tablet server needed to insert and scan records
*
* As such, one solution could be to use a custom setup where Kudu servers run on the same network than integration
* tests.
* Another solution could be to use the container based setup where Kudu servers are managed by
* {@code KuduTestResource}.
*
* A) How to run integration tests against a custom setup:
* Comment @Disabled and @DisabledOnNativeImage annotations from {@code KuduTest} and {@code KuduIT}.
* Install Kudu master and tablet servers on the same network than integration tests.
* Configure "camel.kudu.test.master.rpc-authority" in "application.properties", for instance:
* camel.kudu.test.master.rpc-authority=kudu-master-hostname:7051
* Run integration tests with mvn clean integration-test -P native
*
* B) How to run integration tests against the container based setup when NOT running on top of OpenJDK 8:
* Comment @Disabled and @DisabledOnNativeImage annotations from {@code KuduTest} and {@code KuduIT}.
* Override the ip resolution of the host "kudu-tserver" to 127.0.0.1, e.g. by adding an entry in /etc/hosts file as
* below:
* 127.0.0.1 kudu-tserver
* Run integration tests with mvn clean integration-test -P native
*
* C) How to run integration tests against the container based setup when running on top of OpenJDK 8:
* Comment @Disabled and @DisabledOnNativeImage annotations from {@code KuduTest} and {@code KuduIT}.
* No extra setup is needed as {@code overrideKuduTabletServerResolutionInInetAddressCache} takes care of redirecting
* the Kudu tablet server traffic toward localhost
* Run integration tests with mvn clean integration-test -P native
*/
@ApplicationScoped
public class KuduInfrastructureTestHelper {

private static final Logger LOG = Logger.getLogger(KuduInfrastructureTestHelper.class);
static final String KUDU_TABLET_SERVER_HOSTNAME = "kudu-tserver";
public static final String KUDU_AUTHORITY_CONFIG_KEY = "camel.kudu.test.master.rpc-authority";

void onStart(@Observes StartupEvent ev) {
LOG.info("Attempting to override the kudu tablet server hostname resolution on application startup");
KuduInfrastructureTestHelper.overrideTabletServerHostnameResolution();
}

public static void overrideTabletServerHostnameResolution() {
try {
Field field = InetAddress.class.getDeclaredField("addressCache");
field.setAccessible(true);
Object addressCache = field.get(null);

Method put = addressCache.getClass().getMethod("put", String.class, InetAddress[].class);
put.setAccessible(true);
put.invoke(addressCache, KUDU_TABLET_SERVER_HOSTNAME, (Object[]) InetAddress.getAllByName("localhost"));
} catch (Exception ex) {
final String msg = "Can't override the kudu tablet server hostname resolution when not running on top of OpenJDK 8";
LOG.error(msg, ex);
}
}
}

0 comments on commit 612f7e5

Please sign in to comment.