Refactor connect adapter API (#1290)

* [hotfix] Remove set adapters from extensions * [hotfix] Remove old resource files * [#1289] Remove data set model * [#1289] Remove data set model from ui * [#1289] Remove `adapterType` and fix tests * [#1289] Remove class `SpDataSet` * [#1289] Remove legacy code in class `AdapterDescription` * [#1289] Remove class `AdapterStreamDescription` * [#1289] Remove class `ProtocolStreamDescription` * [hotfix] Remove class `AdapterDescription` * [#1289] Very first version for new adapter interface * [#1289] Started to migrate the API * [#1289] Deactivate old protocol API * [#1289] Remove old models from the ui * [#1289] Remove old models from the ui * [#1289] Fix icons for adapters * [#1289] Fix minor bug * [#1289] Fix maven build * [#1289] WIP refactor `AdapterConfigurationBuilder` * [#1289] Fix AdapterConfigurationBuilder * [#1289] Fix AdapterConfigurationBuilder * [#1289] Guess schema works now for NewMachineDataSimulatorAdapter * [#1289] Add start adapter * [#1289] First version of new machine data simulator works * [#1289] Working on the first new parser * [#1289] First version of declare parser description * [WIP] Add adapter configuration test * [#1289] First generic adapter works now * [#1289] Add parser exception to json parser * [#1289] Add csv parser * [#1289] Change implementation for json parser * [#1289] Add json array parsers * Add runtime context to adapters (#1290) * Extract interface of (#1290) * Refactor OPC-UA adapter to new model (#1290) * Refactor S7 adapter (#1290) * Refactor Modbus adapter (#1290) * Refactor Ros adapter (#1290) * [#1289] Add geo json parser * [#1289] Add xml parser * [#1289] Add image parser * Refactor Kafka adapter (#1290) * Refactor MQTT adapter (#1290) * Refactor Nats adapter (#1290) * Refactor Pulsar adapter (#1290) * Refactor RocketMQ adapter (#1290) * Refactor TubeMQ adapter (#1290) * Refactor HTTP server adapter (#1290) * Refactor HTTP Stream adapter (#1290) * [WIP] Refactor Parser API * [WIP] Refactor Parser API * [#1289] Fix file stream e2e test * [#1289] Fix tests and csv parser for timestamps * [#1289] Fix e2e connect tests * [hotfix] Fix build * [hotfix] Deactivate Mqtt integration test * [#1289] Remove old format implementation from backend * [#1289] Remove old format implementation from backend * [#1289] Remove old format implementation from ui * [#1289] Fix kafka e2e test * Add migration script for specific adapters (#1289) * Add initial version to migrate generic adapters (#1289) * Modify format migrators (#1289) * [#1289] Migrate influxdb to new interface * [#1289] Fix ROS bridge adapter * [#1289] Rename specific-adapter-configuration.component * [#1289] Remove old data model * [#1289] Fix mqtt integration test * [#1289] Update Pulsar integration test * [#1289] Remove DebugAdapterSink which is no longer needed for integration tests * [#1289] Fix kafka adapter on restart * [#1289] Fix migrations for all parser * [hotfix] Fix image parser for adapters * [hotfix] Change StreamPipes version to fix build problem * [#1289] Move adapters * [hotfix] Remove duplicate licence headers * [hotfix] Fix conversion of pipeline element templates * Move PullAdapter to extensions-management and api modules (#1289) * Refactor structure of extensions-api, add supplier to adapter interface (#1581) * Rename AdapterInterface to StreamPipesAdapter (#1590) * [hotfix] Fix checkstyle issues * [hotfix] Fix checkstyle issue * [hotfix] Fix checkstyle issue * [hotfix] Fix checkstyle * Migrate ISS adapter and Flic adapter (#1289) * Migrate connect adapters to new model (#1289) * Properly apply configs to influx adapter (#1289) * [hotfix] Fix checkstyle * fix: minor adaptions * fix: remove data set artifacts * fix: minor fixes * [hotfix] Regenerate typescript model * Refactor data processor and sink API (#1632) (#1633) * chore: mark streampipes-wrapper-python as deprecated (#1623) * Bump types-requests in /streampipes-client-python (#1588) Bumps [types-requests](https://github.com/python/typeshed) from 2.30.0.0 to 2.31.0.0. - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-requests dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump mkdocstrings[python] in /streampipes-client-python (#1631) * chore: add missing support of NATS as messaging protocol (#1628) * Bump spring-boot.version from 3.0.6 to 3.1.0 (#1595) Bumps `spring-boot.version` from 3.0.6 to 3.1.0. Updates `spring-boot-properties-migrator` from 3.0.6 to 3.1.0 - [Release notes](https://github.com/spring-projects/spring-boot/releases) - [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0) Updates `spring-boot-starter-jersey` from 3.0.6 to 3.1.0 - [Release notes](https://github.com/spring-projects/spring-boot/releases) - [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0) Updates `spring-boot-starter-jetty` from 3.0.6 to 3.1.0 - [Release notes](https://github.com/spring-projects/spring-boot/releases) - [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0) Updates `spring-boot-starter-oauth2-client` from 3.0.6 to 3.1.0 - [Release notes](https://github.com/spring-projects/spring-boot/releases) - [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0) Updates `spring-boot-starter-undertow` from 3.0.6 to 3.1.0 - [Release notes](https://github.com/spring-projects/spring-boot/releases) - [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0) Updates `spring-boot-starter-web` from 3.0.6 to 3.1.0 - [Release notes](https://github.com/spring-projects/spring-boot/releases) - [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0) Updates `spring-boot-maven-plugin` from 3.0.6 to 3.1.0 - [Release notes](https://github.com/spring-projects/spring-boot/releases) - [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0) --- updated-dependencies: - dependency-name: org.springframework.boot:spring-boot-properties-migrator dependency-type: direct:production update-type: version-update:semver-minor - dependency-name: org.springframework.boot:spring-boot-starter-jersey dependency-type: direct:production update-type: version-update:semver-minor - dependency-name: org.springframework.boot:spring-boot-starter-jetty dependency-type: direct:production update-type: version-update:semver-minor - dependency-name: org.springframework.boot:spring-boot-starter-oauth2-client dependency-type: direct:production update-type: version-update:semver-minor - dependency-name: org.springframework.boot:spring-boot-starter-undertow dependency-type: direct:production update-type: version-update:semver-minor - dependency-name: org.springframework.boot:spring-boot-starter-web dependency-type: direct:production update-type: version-update:semver-minor - dependency-name: org.springframework.boot:spring-boot-maven-plugin dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Refactor data processor and sink API (#1632) * Cleanup pom file (#1632) * Migrate processors to API (#1632) * [hotfix] Add runtime provider, update pom * Fix bug in standalone wrapper (#1632) * Fix instance provision in compatibility layer (#1632) * [hotfix] Disable Maven cache in workflow file * [hotfix] Fix failing build by reverting Flink version * [hotfix] Add proper time selection to another cypress test (#1641) * Fix bug in aggregation flink module * Properly return output source and schema info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Tim <50115603+bossenti@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: remove Plc4xPassiveAdapter * Add implementation for schema guessing from the example file in the HTTP resolves #1637 fixes #1518 for streampipes-extensions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Dominik Riemer <dominik.riemer@gmail.com> Co-authored-by: Stefan Obermeier <st.obermeier@seeburger.de> Co-authored-by: bossenti <bossenti@posteo.de> Co-authored-by: Tim <50115603+bossenti@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
apache · Jun 2, 2023 · 752289c · 752289c
1 parent 839e10d
commit 752289c
Show file tree

Hide file tree

Showing 812 changed files with 12,134 additions and 18,290 deletions.
diff --git a/.github/workflows/pr-validation.yml b/.github/workflows/pr-validation.yml
@@ -191,4 +191,4 @@ jobs:
       - name: Build MkDocs
         working-directory: ./streampipes-client-python
         run: |
-          make doc
+          make doc
diff --git a/pom.xml b/pom.xml
@@ -160,6 +160,7 @@
 	<jsrosbridge.version>0.2.0</jsrosbridge.version>
 	<jedis.version>4.3.1</jedis.version>
 	<j2objc-annotations.version>1.3</j2objc-annotations.version>
+	<kryo.version>2.24.0</kryo.version>
 	<language-detector.version>0.6</language-detector.version>
 	<log4j-to-slf4j.version>2.8.2</log4j-to-slf4j.version>
 	<lombok.version>1.18.22</lombok.version>
@@ -230,6 +231,11 @@
 					<artifactId>logback-core</artifactId>
 					<version>${logback.version}</version>
 				</dependency>
+				<dependency>
+					<groupId>com.esotericsoftware.kryo</groupId>
+					<artifactId>kryo</artifactId>
+					<version>${kryo.version}</version>
+				</dependency>
 				<dependency>
 					<groupId>com.fasterxml.jackson.core</groupId>
 					<artifactId>jackson-annotations</artifactId>
@@ -957,12 +963,12 @@
 					<scope>test</scope>
 				</dependency>
 
-        <dependency>
-          <groupId>org.apache.flink</groupId>
-          <artifactId>flink-runtime_2.11</artifactId>
-          <version>${flink.version}</version>
-          <type>test-jar</type>
-        </dependency>
+				<dependency>
+				  <groupId>org.apache.flink</groupId>
+				  <artifactId>flink-runtime_2.11</artifactId>
+				  <version>${flink.version}</version>
+				  <type>test-jar</type>
+				</dependency>
 				<dependency>
 					<groupId>org.apache.maven</groupId>
 					<artifactId>maven-aether-provider</artifactId>
@@ -1221,11 +1227,6 @@
                 <artifactId>flink-cep_2.11</artifactId>
                 <version>${flink.version}</version>
             </dependency>
-            <dependency>
-                <groupId>org.apache.flink</groupId>
-                <artifactId>flink-cep-scala_2.11</artifactId>
-                <version>${flink.version}</version>
-            </dependency>
             <dependency>
                 <groupId>org.apache.flink</groupId>
                 <artifactId>flink-runtime_2.11</artifactId>
@@ -1409,6 +1410,7 @@
         <module>streampipes-integration-tests</module>
 		<module>streampipes-rest-extensions</module>
         <module>streampipes-extensions/streampipes-connectors-influx</module>
+        <module>streampipes-client-api</module>
     </modules>
 
     <profiles>

diff --git a/streampipes-client-api/pom.xml b/streampipes-client-api/pom.xml
@@ -0,0 +1,54 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~    http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>org.apache.streampipes</groupId>
+        <artifactId>streampipes-parent</artifactId>
+        <version>0.93.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>streampipes-client-api</artifactId>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.streampipes</groupId>
+            <artifactId>streampipes-dataformat</artifactId>
+            <version>0.93.0-SNAPSHOT</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.streampipes</groupId>
+            <artifactId>streampipes-messaging</artifactId>
+            <version>0.93.0-SNAPSHOT</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.streampipes</groupId>
+            <artifactId>streampipes-model</artifactId>
+            <version>0.93.0-SNAPSHOT</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.httpcomponents</groupId>
+            <artifactId>fluent-hc</artifactId>
+        </dependency>
+    </dependencies>
+
+</project>
diff --git a/...pache/streampipes/client/api/CRUDApi.java → ...pache/streampipes/client/api/CRUDApi.java b/...pache/streampipes/client/api/CRUDApi.java → ...pache/streampipes/client/api/CRUDApi.java
diff --git a/streampipes-client-api/src/main/java/org/apache/streampipes/client/api/IAdminApi.java b/streampipes-client-api/src/main/java/org/apache/streampipes/client/api/IAdminApi.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.client.api;
+
+import org.apache.streampipes.model.config.MessagingSettings;
+import org.apache.streampipes.model.connect.adapter.AdapterDescription;
+import org.apache.streampipes.model.function.FunctionDefinition;
+
+import java.util.List;
+
+public interface IAdminApi {
+  void registerAdapters(List<AdapterDescription> adapters);
+
+  void registerFunctions(List<FunctionDefinition> functions);
+
+  void deregisterFunction(String functionId);
+
+  MessagingSettings getMessagingSettings();
+}
diff --git a/...ampipes-client-api/src/main/java/org/apache/streampipes/client/api/ICustomRequestApi.java b/...ampipes-client-api/src/main/java/org/apache/streampipes/client/api/ICustomRequestApi.java
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.client.api;
+
+import java.util.Map;
+
+public interface ICustomRequestApi {
+  <T> void sendPost(String apiPath, T payload);
+
+  <T> T sendGet(String apiPath, Class<T> responseClass);
+
+  <T> T sendGet(String apiPath, Map<String, String> queryParameters, Class<T> responseClass);
+}
diff --git a/...pipes-client-api/src/main/java/org/apache/streampipes/client/api/IDataLakeMeasureApi.java b/...pipes-client-api/src/main/java/org/apache/streampipes/client/api/IDataLakeMeasureApi.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.client.api;
+
+import org.apache.streampipes.model.datalake.DataLakeMeasure;
+
+import java.util.List;
+import java.util.Optional;
+
+public interface IDataLakeMeasureApi extends CRUDApi<String, DataLakeMeasure> {
+  Optional<DataLakeMeasure> get(String id);
+
+  @Override
+  List<DataLakeMeasure> all();
+
+  @Override
+  void create(DataLakeMeasure element);
+
+  @Override
+  void delete(String elementId);
+
+  @Override
+  void update(DataLakeMeasure measure);
+}
diff --git a/...ampipes-client-api/src/main/java/org/apache/streampipes/client/api/IDataProcessorApi.java b/...ampipes-client-api/src/main/java/org/apache/streampipes/client/api/IDataProcessorApi.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.client.api;
+
+import org.apache.streampipes.client.api.annotation.NotYetImplemented;
+import org.apache.streampipes.client.api.constants.InputStreamIndex;
+import org.apache.streampipes.client.api.live.EventProcessor;
+import org.apache.streampipes.client.api.live.IKafkaConfig;
+import org.apache.streampipes.messaging.EventConsumer;
+import org.apache.streampipes.model.graph.DataProcessorInvocation;
+import org.apache.streampipes.model.grounding.KafkaTransportProtocol;
+
+import java.util.List;
+import java.util.Optional;
+
+public interface IDataProcessorApi extends CRUDApi<String, DataProcessorInvocation> {
+
+  @Override
+  Optional<DataProcessorInvocation> get(String s);
+
+  @Override
+  List<DataProcessorInvocation> all();
+
+  @Override
+  @NotYetImplemented
+  void create(DataProcessorInvocation element);
+
+  @Override
+  @NotYetImplemented
+  void delete(String s);
+
+  @Override
+  @NotYetImplemented
+  void update(DataProcessorInvocation element);
+
+  EventConsumer<KafkaTransportProtocol> subscribe(DataProcessorInvocation processor,
+                            EventProcessor callback);
+
+  EventConsumer<KafkaTransportProtocol> subscribe(DataProcessorInvocation processor,
+                            IKafkaConfig kafkaConfig,
+                            EventProcessor callback);
+
+  EventConsumer<KafkaTransportProtocol> subscribe(DataProcessorInvocation processor,
+                                                  InputStreamIndex index,
+                                                  EventProcessor callback);
+
+  EventConsumer<KafkaTransportProtocol> subscribe(DataProcessorInvocation processor,
+                            InputStreamIndex index,
+                            IKafkaConfig kafkaConfig,
+                            EventProcessor callback);
+}
diff --git a/streampipes-client-api/src/main/java/org/apache/streampipes/client/api/IDataSinkApi.java b/streampipes-client-api/src/main/java/org/apache/streampipes/client/api/IDataSinkApi.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.client.api;
+
+import org.apache.streampipes.client.api.annotation.NotYetImplemented;
+import org.apache.streampipes.client.api.live.EventProcessor;
+import org.apache.streampipes.client.api.live.IKafkaConfig;
+import org.apache.streampipes.messaging.EventConsumer;
+import org.apache.streampipes.model.graph.DataSinkInvocation;
+import org.apache.streampipes.model.grounding.KafkaTransportProtocol;
+
+import java.util.List;
+import java.util.Optional;
+
+public interface IDataSinkApi extends CRUDApi<String, DataSinkInvocation> {
+  @Override
+  Optional<DataSinkInvocation> get(String s);
+
+  @Override
+  List<DataSinkInvocation> all();
+
+  @Override
+  @NotYetImplemented
+  void create(DataSinkInvocation element);
+
+  @Override
+  @NotYetImplemented
+  void delete(String s);
+
+  @Override
+  void update(DataSinkInvocation element);
+
+  EventConsumer<KafkaTransportProtocol> subscribe(DataSinkInvocation sink,
+                                                  EventProcessor callback);
+
+  EventConsumer<KafkaTransportProtocol> subscribe(DataSinkInvocation sink,
+                                                  IKafkaConfig kafkaConfig,
+                                                  EventProcessor callback);
+
+  DataSinkInvocation getDataSinkForPipelineElement(String templateId, DataSinkInvocation pipelineElement);
+}