Skip to content
Permalink
Browse files
IGNITE-17049 Move ignite-spark modules to the Ignite Extension project (
#10057)

* IGNITE-17049 Move ignite-spark modules to the Ignite Extension project

* IGNITE-17049 Fix readme files

* IGNITE-17049 Fix review comments for documentation pages.
  • Loading branch information
Mmuzaf committed Jun 1, 2022
1 parent a8eaa92 commit e9d2d7666f3d0c90b383bc407771e500f8ebe463
Showing 169 changed files with 12 additions and 25,689 deletions.
@@ -153,8 +153,6 @@
<exclude>${project.groupId}:ignite-osgi</exclude>
<exclude>${project.groupId}:ignite-osgi-karaf</exclude>
<exclude>${project.groupId}:ignite-osgi-paxlogging</exclude>
<exclude>${project.groupId}:ignite-spark</exclude>
<exclude>${project.groupId}:ignite-spark-2.4</exclude>
<exclude>${project.groupId}:ignite-ssh</exclude>
<exclude>${project.groupId}:ignite-web</exclude>
<exclude>${project.groupId}:ignite-yarn</exclude>
@@ -85,7 +85,6 @@ The following modules are available:
- ignite-rest-http (for HTTP REST messages)
- ignite-schedule (for Cron-based task scheduling)
- ignite-sl4j (for SL4J logging)
- ignite-spark (for shared in-memory RDDs and faster SQL for Apache Spark)
- ignite-spring (for Spring-based configuration support)
- ignite-ssh (for starting grid nodes on remote machines)
- ignite-urideploy (for URI-based deployment)
@@ -84,23 +84,13 @@ Embedded deployment means that Apache Ignite nodes are started inside the Apache

Ignite's Spark artifact is link:http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.ignite%22[hosted in Maven Central^]. Depending on a Scala version you use, include the artifact using one of the dependencies shown below.

.Scala 2.11
.Scala 2.11+
[source, scala]
----
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-spark</artifactId>
<version>${ignite.version}</version>
</dependency>
----

.Scala 2.10
[source, scala]
----
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-spark_2.10</artifactId>
<version>${ignite.version}</version>
<artifactId>ignite-spark-ext</artifactId>
<version>2.0.0</version>
</dependency>
----

@@ -111,14 +101,7 @@ If SBT is used as a build tool for a Scala application, then Ignite's Spark arti
.Scala 2.11
[source, scala]
----
libraryDependencies += "org.apache.ignite" % "ignite-spark" % "ignite.version"
----


.Scala 2.10
[source, scala]
----
libraryDependencies += "org.apache.ignite" % "ignite-spark_2.10" % "ignite.version"
libraryDependencies += "org.apache.ignite" % "ignite-spark-ext" % "2.0.0"
----


@@ -137,7 +120,7 @@ The following shows how to fill in `spark.driver.extraClassPath` parameters:

[source, shell]
----
spark.executor.extraClassPath /opt/ignite/libs/*:/opt/ignite/libs/optional/ignite-spark/*:/opt/ignite/libs/optional/ignite-log4j/*:/opt/ignite/libs/optional/ignite-yarn/*:/opt/ignite/libs/ignite-spring/*
spark.executor.extraClassPath /opt/ignite/libs/*:/opt/ignite/libs/optional/ignite-spark-ext/*:/opt/ignite/libs/optional/ignite-log4j/*:/opt/ignite/libs/optional/ignite-yarn/*:/opt/ignite/libs/ignite-spring/*
----

=== Source Code Configuration
@@ -45,5 +45,5 @@ Ignite expands link:extensions-and-integrations/ignite-for-spark/ignite-datafram

Apache Ignite comes with two modules that support different versions of Apache Spark:

* ignite-spark — integration with Spark 2.3
* ignite-spark-2.4 — integration with Spark 2.4
* ignite-spark-ext 1.0.0 — integration with Spark 2.3
* ignite-spark-ext 2.0.0 — integration with Spark 2.4
@@ -62,15 +62,15 @@ Now that you have your cluster up and running, you can run `spark-shell` and che
[source, shell]
----
./bin/spark-shell
--packages org.apache.ignite:ignite-spark:1.8.0
--packages org.apache.ignite:ignite-spark-ext:2.0.0
--master spark://master_host:master_port
--repositories http://repo.maven.apache.org/maven2/org/apache/ignite
----
* Or by providing paths to Ignite jar file paths using `--jars` parameter
+
[source, shell]
----
./bin/spark-shell --jars path/to/ignite-core.jar,path/to/ignite-spark.jar,path/to/cache-api.jar,path/to/ignite-log4j.jar,path/to/log4j.jar --master spark://master_host:master_port
./bin/spark-shell --jars path/to/ignite-core.jar,path/to/ignite-spark-ext.jar,path/to/cache-api.jar,path/to/ignite-log4j.jar,path/to/log4j.jar --master spark://master_host:master_port
----

You should see Spark shell started up.
@@ -80,7 +80,7 @@ Note that if you are planning to use spring configuration loading, you will need
[source, shell]
----
./bin/spark-shell
--packages org.apache.ignite:ignite-spark:1.8.0,org.apache.ignite:ignite-spring:1.8.0
--packages org.apache.ignite:ignite-spark-ext:2.0.0,org.apache.ignite:ignite-spring:2.13.0
--master spark://master_host:master_port
----
--
@@ -243,8 +243,6 @@ by the Pax Logging API - the logging framework used by Apache Karaf.

|ignite-slf4j | Support for link:logging#using-slf4j[SLF4J logging framework].

|ignite-spark | This module provides an implementation of Spark RDD abstraction that enables easy access to Ignite caches.

|ignite-ssh | The Ignite SSH module provides capabilities to start Ignite nodes on remote machines via SSH.

|ignite-tensorflow | The Ignite TensorFlow Integration Module allows using TensorFlow with Ignite. In this scenario Ignite will be a datasource for any TensorFlow model training.
@@ -272,6 +270,8 @@ The following extensions are available:

|ignite-gce-ext | Ignite GCE provides Google Cloud Storage based implementations of IP finder for TCP discovery.

|ignite-spark-ext | This module provides an implementation of Spark RDD abstraction that enables easy access to Ignite caches.

|ignite-spring-data-ext | Ignite Spring Data provides an integration with Spring Data framework.

|ignite-spring-data_2.0-ext | Ignite Spring Data 2.0 provides an integration with Spring Data 2.0 framework.
@@ -13,7 +13,6 @@ The examples folder contains he following subfolders:
- `rest` - contains PHP script demonstrating how Ignite Cache can be accessed via HTTP API.
- `sql` - contains sample SQL scripts and data sets.
- `src/main/java` - contains Java examples for different Ignite modules and features.
- `src/main/scala` - contains examples demonstrating usage of API provided by Spark.
- `src/main/java-lgpl` - contains lgpl-based examples for different Ignite modules and features.


@@ -13,7 +13,6 @@ The examples folder contains he following subfolders:
- `rest` - contains PHP script demonstrating how Ignite Cache can be accessed via HTTP API.
- `sql` - contains sample SQL scripts and data sets.
- `src/main/java` - contains Java examples for different Ignite modules and features.
- `src/main/scala` - contains examples demonstrating usage of API provided by Spark.
- `src/main/java-lgpl` - contains lgpl-based examples for different Ignite modules and features.


@@ -13,7 +13,6 @@ The examples folder contains he following subfolders:
- `rest` - contains PHP script demonstrating how Ignite Cache can be accessed via HTTP API.
- `sql` - contains sample SQL scripts and data sets.
- `src/main/java` - contains Java examples for different Ignite modules and features.
- `src/main/scala` - contains examples demonstrating usage of API provided by Spark.
- `src/main/java-lgpl` - contains lgpl-based examples for different Ignite modules and features.

Starting Remote Nodes

This file was deleted.

@@ -33,7 +33,6 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<!-- will be changed by profile activation. allows to combine profiles. -->
<spark.folder>src/main/java</spark.folder>
<lgpl.folder>src/main/java</lgpl.folder>
</properties>

@@ -153,126 +152,6 @@
</dependency>
</dependencies>

<profiles>
<profile>
<id>scala</id>

<properties>
<spark.folder>src/main/spark</spark.folder>
</properties>

<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>ignite-spark</artifactId>
<version>to_be_replaced_by_ignite_version</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.0</version>
<configuration>
<jvmArgs>
<jvmArg>-Xms512m</jvmArg>
<jvmArg>-Xmx1024m</jvmArg>
</jvmArgs>
</configuration>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>

<profile>
<id>lgpl</id>

<activation>
<activeByDefault>true</activeByDefault>
</activation>

<properties>
<lgpl.folder>src/main/java-lgpl</lgpl.folder>
</properties>

<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>ignite-schedule</artifactId>
<version>to_be_replaced_by_ignite_version</version>
</dependency>
</dependencies>
</profile>

<profile>
<id>spark-2.4</id>

<properties>
<spark.folder>src/main/spark</spark.folder>
</properties>

<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>ignite-spark-2.4</artifactId>
<version>to_be_replaced_by_ignite_version</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.0</version>
<configuration>
<jvmArgs>
<jvmArg>-Xms512m</jvmArg>
<jvmArg>-Xmx1024m</jvmArg>
</jvmArgs>
</configuration>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>

<build>
<resources>
<resource>
@@ -300,7 +179,6 @@
<configuration>
<sources>
<source>${lgpl.folder}</source>
<source>${spark.folder}</source>
</sources>
</configuration>
</execution>

0 comments on commit e9d2d76

Please sign in to comment.