Merge pull request #244 from dafreels/develop

Bug fixes

metalus-aws/readme.md

@@ -20,6 +20,12 @@ Only one of the scopes is needed.
 |3.0|aws-java-sdk-s3|1.11.655|extraction,s3
 |3.0|aws-java-sdk-core|1.11.655|extraction,sdk
 |3.0|hadoop-aws|2.7.7|extraction,s3a
+|3.1|spark-streaming-kinesis-asl_2.12|3.0.0|extraction,stream
+|3.1|amazon-kinesis-client|1.12.0|extraction,stream
+|3.1|aws-java-sdk-secretsmanager|1.11.655|extraction,secretsmanager
+|3.1|aws-java-sdk-s3|1.11.655|extraction,s3
+|3.1|aws-java-sdk-core|1.11.655|extraction,sdk
+|3.1|hadoop-aws|2.7.7|extraction,s3a
 
 ## Step Classes
 * [S3Steps](docs/s3steps.md)

metalus-core/src/main/scala/com/acxiom/pipeline/PipelineStepMapper.scala

@@ -174,7 +174,7 @@ trait PipelineStepMapper {
       val (params, s) = trimmedScript.splitAt(index)
       val paramString = params.drop(1).dropRight(1).trim
       val parameterMap = if (paramString.nonEmpty) {
-        paramString.split(",").map { p =>
+        paramString.split("[,]+(?![^\\[]*])").map { p =>
           val ret = p.split("""(?<!((?<!\\)\\)):""").map(_.replaceAllLiterally("""\:""", ":").replaceAllLiterally("""\\""", "\\"))
           if(ret.length < 2){
             throw PipelineException(message = Some(s"Unable to execute script: Illegal binding format: [$p]. Expected format: <name>:<value>:<type>"),

metalus-core/src/test/scala/com/acxiom/pipeline/PipelineStepMapperTests.scala

@@ -95,7 +95,8 @@ class PipelineStepMapperTests extends FunSpec with BeforeAndAfterAll with GivenW
       "char" -> 1.toChar,
       "bigdecimal" -> BigDecimal(1),
       "string" -> "stringWithinAnOption",
-      "tripleOption" -> Some(Some(Some("stringWithinAnOption")))
+      "tripleOption" -> Some(Some(Some("stringWithinAnOption"))),
+      "pipelinesMap" -> List(Map("pipelineId" -> "30273bbb-983b-43ad-a549-6d23f3c0d143"))
     )
 
     val subPipeline = Pipeline(Some("mypipeline"), Some("My Pipeline"))
@@ -190,6 +191,10 @@ class PipelineStepMapperTests extends FunSpec with BeforeAndAfterAll with GivenW
         ("script with escaped binding",
           Parameter(value = Some(
             """(s:cChickens\:Rule\\:String)(s.toLowerCase.drop(1))"""), `type` = Some("scalascript")), "chickens:rule\\"),
+        ("script with comma in type",
+          Parameter(value = Some(
+            """(pipelines:!pipelinesMap:List[Map[String,Any]]) pipelines.exists(p => p("pipelineId").asInstanceOf[String] == "30273bbb-983b-43ad-a549-6d23f3c0d143")"""),
+            `type` = Some("scalascript")), true),
         ("Casting String global to int", Parameter(value = Some("!numericString"), `type` = Some("int")), 1),
         ("Casting String global to bigInt", Parameter(value = Some("!numericString"), `type` = Some("bigInt")), BigInt(1)),
         ("Casting String global to double", Parameter(value = Some("!numericString"), `type` = Some("double")), 1.0D),

metalus-delta/readme.md

@@ -9,7 +9,9 @@ Only one of the scopes is needed.
 |Spark|Library|Version|Scopes|
 |------|-------|-------|------|
 |2.4|delta-core_2.11|0.6.1|extraction,deltalake|
+|2.4|delta-core_2.12|0.6.1|extraction,deltalake|
 |3.0|delta-core_2.12|0.8.0|extraction,deltalake|
+|3.1|delta-core_2.12|1.0.0|extraction,deltalake|
 
 ## Step Classes
 * [DeltaLakeSteps](docs/deltalakesteps.md)

metalus-gcp/docs/gcsfilemanager.md

@@ -1,6 +1,6 @@
 [Documentation Home](../../docs/readme.md) | [GCP Home](../readme.md)
 
-# S3FileManager
+# GCSFileManager
 Provides a [FileManager](../../docs/filemanager.md) implementation that works with the GCS file system.  Full
 parameter descriptions are listed below:
 

readme.md

@@ -22,46 +22,70 @@ This project contains the core library and is the minimum requirement for any ap
 
 [Maven 2.11 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-core_2.11-spark_2.4)
 
+[Maven 2.12 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-core_2.12-spark_2.4)
+
 [Maven 2.12 Spark 3.0 library](https://search.maven.org/search?q=a:metalus-core_2.12-spark_3.0)
 
+[Maven 2.12 Spark 3.1 library](https://search.maven.org/search?q=a:metalus-core_2.12-spark_3.1)
+
 ### [Metalus Common Step Library](metalus-common/readme.md)
 This step library contains steps that are considered generic enough to be used in any project.
 
 [Maven 2.11 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-common_2.11-spark_2.4)
 
+[Maven 2.12 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-common_2.12-spark_2.4)
+
 [Maven 2.12 Spark 3.0 library](https://search.maven.org/search?q=a:metalus-common_2.12-spark_3.0)
 
+[Maven 2.12 Spark 3.1 library](https://search.maven.org/search?q=a:metalus-common_2.12-spark_3.1)
+
 ### [Metalus AWS Step Library](metalus-aws/readme.md)
 This step library contains AWS specific components. The [Kinesis](https://aws.amazon.com/kinesis/) driver provides a 
 basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.
 
 [Maven 2.11 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-aws_2.11-spark_2.4)
 
+[Maven 2.12 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-aws_2.12-spark_2.4)
+
 [Maven 2.12 Spark 3.0 library](https://search.maven.org/search?q=a:metalus-aws_2.12-spark_3.0)
 
+[Maven 2.12 Spark 3.1 library](https://search.maven.org/search?q=a:metalus-aws_2.12-spark_3.1)
+
 ### [Metalus GCP Step Library](metalus-gcp/readme.md)
 This step library contains GCP specific components. The [Pub/Sub](https://cloud.google.com/pubsub/docs/overview) driver provides a 
 basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.
 
 [Maven 2.11 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-gcp_2.11-spark_2.4)
 
+[Maven 2.12 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-gcp_2.12-spark_2.4)
+
 [Maven 2.12 Spark 3.0 library](https://search.maven.org/search?q=a:metalus-gcp_2.12-spark_3.0)
 
+[Maven 2.12 Spark 3.1 library](https://search.maven.org/search?q=a:metalus-gcp_2.12-spark_3.1)
+
 ### [Metalus Kafka Step Library](metalus-kafka/readme.md)
 This step library contains GCP specific components. The [Kafka](https://kafka.apache.org/) driver provides a 
 basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.
 
 [Maven 2.11 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-kafka_2.11-spark_2.4)
 
+[Maven 2.12 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-kafka_2.12-spark_2.4)
+
 [Maven 2.12 Spark 3.0 library](https://search.maven.org/search?q=a:metalus-kafka_2.12-spark_3.0)
 
+[Maven 2.12 Spark 3.1 library](https://search.maven.org/search?q=a:metalus-kafka_2.12-spark_3.1)
+
 ### [Metalus Mongo Step Library](metalus-mongo/readme.md)
 This step library adds support for working with Mongo.
 
 [Maven 2.11 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-mongo_2.11-spark_2.4)
 
+[Maven 2.12 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-mongo_2.12-spark_2.4)
+
 [Maven 2.12 Spark 3.0 library](https://search.maven.org/search?q=a:metalus-mongo_2.12-spark_3.0)
 
+[Maven 2.12 Spark 3.1 library](https://search.maven.org/search?q=a:metalus-mongo_2.12-spark_3.1)
+
 ### [Metalus Pipeline Examples](metalus-examples/readme.md)
 This project provides several examples to help demonstrate how to use the library.
 
@@ -70,8 +94,12 @@ This project provides utilities that help work with the project.
 
 [Maven 2.11 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-utils_2.11-spark_2.4)
 
+[Maven 2.12 Spark 2.4 library](https://search.maven.org/search?q=a:metalus-utils_2.12-spark_2.4)
+
 [Maven 2.12 Spark 3.0 library](https://search.maven.org/search?q=a:metalus-utils_2.12-spark_3.0)
 
+[Maven 2.12 Spark 3.1 library](https://search.maven.org/search?q=a:metalus-utils_2.12-spark_3.1)
+
 ### [Metalus Application](metalus-application/readme.md)
 This project provides a single jar that can be used to run the application. Additional components provide jars that can be
 added to the classpath.