Skip to content

Commit

Permalink
Reimplement creation of minio client to avoid minio extension limitat…
Browse files Browse the repository at this point in the history
…ions #2134
  • Loading branch information
JiriOndrusek authored and jamesnetherton committed Jan 20, 2021
1 parent 5fe7b91 commit 6648464
Show file tree
Hide file tree
Showing 6 changed files with 100 additions and 76 deletions.
21 changes: 15 additions & 6 deletions docs/modules/ROOT/pages/reference/extensions/minio.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,22 @@ Please refer to the above link for usage and configuration details.

Check the xref:user-guide/index.adoc[User guide] for more information about writing Camel Quarkus applications.

== Camel Quarkus limitations
== Additional Camel Quarkus configuration

This extension leverages the Quarkiverse Minio. Standard application.properties mechanism is used to define connection (see http://github.com/quarkiverse/quarkiverse-minio#configuration-reference[documentation]).
Depending on Minio configuration, this extension may require SSL encryption on its connections. In such cases, you will need
to add `quarkus.ssl.native=true` to your `application.properties`.
See also https://quarkus.io/guides/native-and-ssl[Quarkus native SSL guide] and xref:user-guide/native-mode.adoc[Native mode]
section of Camel Quarkus user guide.

It is mandatory to configure the Minio client in properties and Camel will autowire client into the Minio component.
Configuration via application.properties allows definition of only one minio client, therefore it isn't possible to define several different minio endpoints, which run together.
There are two different configuration approaches:

Please upvote https://github.com/quarkiverse/quarkiverse-minio/issues/22[this issue]
if you do not like the present state.
* Minio client can be defined via quarkus properties leveraging the Quarkiverse Minio (see http://github.com/quarkiverse/quarkiverse-minio#configuration-reference[documentation]).
Camel will autowire client into the Minio component.
This configuration allows definition of only one minio client, therefore it isn't possible to define several different minio endpoints, which run together.

* Provide client/clients for camel registry (e.g. CDI producer/bean) and reference them from endpoint.
[source,properties]
----
minio:foo?minioClient=#minioClient
----

16 changes: 16 additions & 0 deletions extensions/minio/runtime/src/main/doc/configuration.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
Depending on Minio configuration, this extension may require SSL encryption on its connections. In such cases, you will need
to add `quarkus.ssl.native=true` to your `application.properties`.
See also https://quarkus.io/guides/native-and-ssl[Quarkus native SSL guide] and xref:user-guide/native-mode.adoc[Native mode]
section of Camel Quarkus user guide.

There are two different configuration approaches:

* Minio client can be defined via quarkus properties leveraging the Quarkiverse Minio (see http://github.com/quarkiverse/quarkiverse-minio#configuration-reference[documentation]).
Camel will autowire client into the Minio component.
This configuration allows definition of only one minio client, therefore it isn't possible to define several different minio endpoints, which run together.
* Provide client/clients for camel registry (e.g. CDI producer/bean) and reference them from endpoint.
[source,properties]
----
minio:foo?minioClient=#minioClient
----
7 changes: 0 additions & 7 deletions extensions/minio/runtime/src/main/doc/limitations.adoc

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.camel.quarkus.component.minio.it;

import javax.enterprise.context.ApplicationScoped;
import javax.inject.Named;
import javax.ws.rs.Produces;

import io.minio.MinioClient;

public class MinioClientProducer {

@Produces
@ApplicationScoped
@Named("minioClient")
public MinioClient produceMinioClient() {
return MinioClient.builder()
.endpoint("http://" + System.getProperty(MinioResource.PARAM_SERVER_HOST),
Integer.parseInt(System.getProperty(MinioResource.PARAM_SERVER_PORT)), false)
.credentials(MinioResource.SERVER_ACCESS_KEY, MinioResource.SERVER_SECRET_KEY)
.build();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,6 @@
*/
package org.apache.camel.quarkus.component.minio.it;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.security.GeneralSecurityException;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
Expand All @@ -34,13 +30,8 @@
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.MediaType;

import io.minio.BucketExistsArgs;
import io.minio.GetObjectArgs;
import io.minio.MakeBucketArgs;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import io.minio.Result;
import io.minio.errors.MinioException;
import io.minio.messages.Bucket;
import io.minio.messages.Item;
import org.apache.camel.ConsumerTemplate;
Expand All @@ -51,7 +42,6 @@
@Path("/minio")
@ApplicationScoped
public class MinioResource {
private static final long PART_SIZE = 50 * 1024 * 1024;

public static final String SERVER_ACCESS_KEY = "testAccessKey";
public static final String SERVER_SECRET_KEY = "testSecretKey";
Expand All @@ -64,22 +54,16 @@ public class MinioResource {
@Inject
ConsumerTemplate consumerTemplate;

@Inject
MinioClient minioClient;

@Path("/consumer")
@GET
@Produces(MediaType.TEXT_PLAIN)
public String consumer() {

String serverUrl = "http://" + System.getProperty(PARAM_SERVER_HOST) + ":" + System.getProperty(PARAM_SERVER_PORT);

final String message = consumerTemplate.receiveBody(
"minio://mycamel?moveAfterRead=true&destinationBucketName=camel-kafka-connector&autoCreateBucket=true"
+ "&accessKey=" + SERVER_ACCESS_KEY
+ "&secretKey=RAW(" + SERVER_SECRET_KEY + ")"
+ "&endpoint=" + serverUrl
+ "&secure=true",
+ "&minioClient=#minioClient",
5000, String.class);
return message;
}
Expand All @@ -94,10 +78,9 @@ public String operation(String body,
@QueryParam(MinioConstants.DESTINATION_OBJECT_NAME) String destinationObjectName,
@QueryParam(MinioConstants.DESTINATION_BUCKET_NAME) String destinationBucketName) {

String serverUrl = "http://" + System.getProperty(PARAM_SERVER_HOST) + ":" + System.getProperty(PARAM_SERVER_PORT);
String endpoint = "minio:mycamel?accessKey=" + SERVER_ACCESS_KEY
+ "&secretKey=RAW(" + SERVER_SECRET_KEY + ")"
+ "&endpoint=" + serverUrl;
+ "&minioClient=#minioClient";

MinioOperations op = (operation != "" && !"".equals(operation) ? MinioOperations.valueOf(operation) : null);

Expand Down Expand Up @@ -152,10 +135,9 @@ public String operation(String body,
public String getUsingPojo(String bucket,
@QueryParam(MinioConstants.OBJECT_NAME) String objectName) {

String serverUrl = "http://" + System.getProperty(PARAM_SERVER_HOST) + ":" + System.getProperty(PARAM_SERVER_PORT);
String endpoint = "minio:mycamel?accessKey=" + SERVER_ACCESS_KEY
+ "&secretKey=RAW(" + SERVER_SECRET_KEY + ")"
+ "&endpoint=" + serverUrl
+ "&minioClient=#minioClient"
+ "&pojoRequest=true";

GetObjectArgs.Builder body = GetObjectArgs.builder()
Expand All @@ -167,32 +149,4 @@ public String getUsingPojo(String bucket,
return producerTemplate.requestBodyAndHeaders(endpoint, body, headers, String.class);

}

@Path("/initBucket")
@POST
@Consumes(MediaType.TEXT_PLAIN)
public void initBucket(String bucketName) throws Exception {
if (!minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build())) {
minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
}
}

@Path("/putObject")
@POST
@Produces(MediaType.TEXT_PLAIN)
public void putObject(String content,
@QueryParam(MinioConstants.OBJECT_NAME) String objectName,
@QueryParam(MinioConstants.BUCKET_NAME) String bucketName) throws Exception {
try (InputStream is = new ByteArrayInputStream((content.getBytes()))) {
minioClient.putObject(
PutObjectArgs.builder()
.bucket(bucketName)
.object(objectName)
.contentType("text/xml")
.stream(is, -1, PART_SIZE)
.build());
} catch (MinioException | GeneralSecurityException | IOException e) {
throw new IllegalStateException(e);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,19 @@
*/
package org.apache.camel.quarkus.component.minio.it;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.security.GeneralSecurityException;
import java.util.Collections;
import java.util.Map;
import java.util.concurrent.TimeUnit;

import io.minio.BucketExistsArgs;
import io.minio.MakeBucketArgs;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import io.minio.errors.MinioException;
import io.quarkus.test.common.QuarkusTestResource;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.RestAssured;
Expand All @@ -39,9 +48,12 @@
@QuarkusTest
@QuarkusTestResource(MinioTestResource.class)
class MinioTest {
private static final long PART_SIZE = 50 * 1024 * 1024;

private final String BUCKET_NAME = "mycamel";

private MinioClient minioClient;

@Test
public void testConsumer() throws Exception {
initClient();
Expand Down Expand Up @@ -154,22 +166,25 @@ private void initClient() throws Exception {
}

private void initClient(String bucketName) throws Exception {
RestAssured.given()
.contentType(ContentType.TEXT)
.body(bucketName)
.post("/minio/initBucket")
.then()
.statusCode(204);
if (minioClient == null) {
minioClient = new MinioClientProducer().produceMinioClient();
}
if (!minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build())) {
minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
}
}

private void sendViaClient(String content, String objectName) {
RestAssured.given()
.contentType(ContentType.TEXT)
.queryParam(MinioConstants.OBJECT_NAME, objectName)
.queryParam(MinioConstants.BUCKET_NAME, BUCKET_NAME)
.body(content)
.post("/minio/putObject")
.then()
.statusCode(204);
try (InputStream is = new ByteArrayInputStream((content.getBytes()))) {
minioClient.putObject(
PutObjectArgs.builder()
.bucket(BUCKET_NAME)
.object(objectName)
.contentType("text/xml")
.stream(is, -1, PART_SIZE)
.build());
} catch (MinioException | GeneralSecurityException | IOException e) {
throw new IllegalStateException(e);
}
}
}

0 comments on commit 6648464

Please sign in to comment.