Rebasing master on top of transport-api-v1

.github/workflows/ci.yml

@@ -0,0 +1,54 @@
+#
+# CI build that assembles artifacts and runs tests.
+# If validation is successful this workflow releases from the main dev branch.
+#
+# - skipping CI: add [skip ci] to the commit message
+# - skipping release: add [skip release] to the commit message
+#
+name: CI
+
+on:
+  push:
+    branches: ['master']
+    tags-ignore: [v*] # release tags are autogenerated after a successful CI, no need to run CI against them
+  pull_request:
+    branches: ['**']
+
+jobs:
+
+  build:
+    runs-on: ubuntu-latest
+    if: "! contains(toJSON(github.event.commits.*.message), '[skip ci]')"
+
+    steps:
+
+      - name: 1. Check out code
+        uses: actions/checkout@v2 # https://github.com/actions/checkout
+        with:
+          fetch-depth: '0' # https://github.com/shipkit/shipkit-changelog#fetch-depth-on-ci
+
+      - name: 2. Setup Java JDK
+        uses: actions/setup-java@v2
+        with:
+          distribution: 'adopt'
+          java-version: '8'
+
+      - name: 3. Perform build
+        run: |
+           ./gradlew build
+           ./gradlew -p transportable-udfs-examples clean build -s
+
+      - name: 4. Perform release
+        # Release job, only for pushes to the main development branch
+        if: github.event_name == 'push'
+          && github.ref == 'refs/heads/master'
+          && github.repository == 'linkedin/transport'
+          && !contains(toJSON(github.event.commits.*.message), '[skip release]')
+
+        run: ./gradlew githubRelease publishToSonatype closeAndReleaseStagingRepository
+        env:
+          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
+          SONATYPE_USER: ${{secrets.SONATYPE_USER}}
+          SONATYPE_PWD: ${{secrets.SONATYPE_PWD}}
+          PGP_KEY: ${{secrets.PGP_KEY}}
+          PGP_PWD: ${{secrets.PGP_PWD}}

README.md

@@ -4,13 +4,13 @@
 **Transport** is a framework for writing performant user-defined
 functions (UDFs) that are portable across a variety of engines
 including [Apache Spark](https://spark.apache.org/), [Apache Hive](https://hive.apache.org/), and
-[Presto](https://prestodb.io/). Transport UDFs are also
+[Trino](https://trino.io/). Transport UDFs are also
 capable of directly processing data stored in serialization formats such as
 Apache Avro. With Transport, developers only need to implement their UDF
 logic once using the Transport API. Transport then takes care of
 translating the UDF to native UDF version targeted at various engines
 or formats. Currently, Transport is capable of generating
-engine-artifacts for Spark, Hive, and Presto, and format-artifacts for
+engine-artifacts for Spark, Hive, and Trino, and format-artifacts for
 Avro. Further details on Transport can be found in this [LinkedIn Engineering blog post](https://engineering.linkedin.com/blog/2018/11/using-translatable-portable-UDFs).
 
 ## Documentation
@@ -127,7 +127,7 @@ to familiarize yourself with the API, and how to write new UDFs.
  to find out how to write UDF tests in a unified testing API, but have the framework test them on multiple platforms.
 
 * Root [`build.gradle`](transportable-udfs-examples/build.gradle) file
-to find out how to apply the `transport` plugin, which enables generating Hive, Spark, and Presto UDFs out of
+to find out how to apply the `transport` plugin, which enables generating Hive, Spark, and Trino UDFs out of
 the transportable UDFs you define once you build your project. To see that in action:
 
 Change directory to `transportable-udfs-examples`:
@@ -153,7 +153,7 @@ The results should be like:
 
 ```
 transportable-udfs-example-udfs-hive.jar
-transportable-udfs-example-udfs-presto.jar
+transportable-udfs-example-udfs-trino.jar
 transportable-udfs-example-udfs-spark.jar
 transportable-udfs-example-udfs.jar
 ```
@@ -162,13 +162,13 @@ That is it! While only one version of the UDFs is implemented, multiple jars are
 Each of those jars uses native platform APIs and data models to implement the UDFs. So from an execution engine's perspective,
 there is no data transformation needed for interoperability or portability. Only suitable classes are used for each engine.
 
-To call those jars from your SQL engine (i.e., Hive, Spark, or Presto), the standard process for deploying UDF jars is followed
+To call those jars from your SQL engine (i.e., Hive, Spark, or Trino), the standard process for deploying UDF jars is followed
 for each engine. For example, in Hive, you add the jar to the classpath using the `ADD JAR` statement,
  and register the UDF using `CREATE FUNCTION` statement.
-In Presto, the jar is deployed to the `plugin` directory. However, a small patch is required for the Presto
-engine to recognize the jar as a plugin, since the generated Presto UDFs implement the `SqlScalarFunction` API, 
-which is currently not part of Presto's SPI architecture. You can find the patch [here](transportable-udfs-documentation/transport-udfs-presto.patch) and apply it
- before deploying your UDFs jar to the Presto engine.
+In Trino, the jar is deployed to the `plugin` directory. However, a small patch is required for the Trino
+engine to recognize the jar as a plugin, since the generated Trino UDFs implement the `SqlScalarFunction` API,
+which is currently not part of Trino's SPI architecture. You can find the patch [here](docs/transport-udfs-trino.patch) and apply it
+ before deploying your UDFs jar to the Trino engine.
 
 ## Contributing
 The project is under active development and we welcome contributions of different forms:

build.gradle

@@ -14,13 +14,18 @@ buildscript {
     classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
     classpath 'org.github.ngbinh.scalastyle:gradle-scalastyle-plugin_2.11:1.0.1'
     classpath 'gradle.plugin.nl.javadude.gradle.plugins:license-gradle-plugin:0.14.0'
+    classpath "io.github.gradle-nexus:publish-plugin:1.0.0"
+    classpath "org.shipkit:shipkit-auto-version:1.1.1"
+    classpath "org.shipkit:shipkit-changelog:1.1.10"
   }
 }
 
 plugins {
-  id "org.shipkit.java" version "2.0.31"
+  id "checkstyle"
 }
 
+apply from: "gradle/shipkit.gradle"
+
 allprojects {
   group = 'com.linkedin.transport'
   apply plugin: 'idea'
@@ -48,19 +53,37 @@ subprojects {
     strictCheck true
   }
 
+  configurations {
+    all {
+      // Transport as a library should only expose slf4j-api to its consumers and should not keep any SLF4J bindings
+      // in its dependency graph
+      // Quote from http://www.slf4j.org/faq.html#maven2
+      //   "Thus, as far as your users are concerned you are exporting slf4j-api as a transitive dependency of your
+      //    library, but not any SLF4J-binding or any underlying logging system."
+      // Our dependencies (e.g. hadoop-common) do bring in slf4j-log4j12 in their dependency graph so we exclude them
+      exclude group: 'org.slf4j', module: 'slf4j-log4j12'
+    }
+  }
+
   plugins.withType(JavaPlugin) {
     project.apply(plugin: 'checkstyle')
 
     dependencies {
       testCompile 'org.testng:testng:6.11'
+      testCompile 'org.slf4j:slf4j-simple:1.7.25'
     }
 
     test {
       useTestNG()
     }
 
     checkstyle {
-      configFile = file("${rootDir}/gradle/checkstyle/linkedin-checkstyle.xml")
+      configFile = rootProject.file('gradle/checkstyle/checkstyle.xml')
+      configProperties = [
+          'configDir': rootProject.file('gradle/checkstyle'),
+          'baseDir': rootDir
+      ]
+      toolVersion '8.23'
     }
   }
 

defaultEnvironment.gradle

@@ -10,8 +10,9 @@ subprojects {
       url "https://conjars.org/repo"
     }
   }
-  project.ext.setProperty('presto-version', '319')
-  project.ext.setProperty('airlift-slice-version', '0.33')
+  project.ext.setProperty('trino-version', '352')
+  project.ext.setProperty('airlift-slice-version', '0.39')
   project.ext.setProperty('spark-group', 'org.apache.spark')
-  project.ext.setProperty('spark-version', '2.3.0')
+  project.ext.setProperty('spark2-version', '2.3.0')
+  project.ext.setProperty('spark3-version', '3.1.1')
 }

docs/release-notes.md

@@ -1,5 +1,72 @@
 <sup><sup>*Release notes were automatically generated by [Shipkit](http://shipkit.org/)*</sup></sup>
 
+#### 0.0.61
+ - 2020-11-26 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.60...v0.0.61) by [Carl Steinbach](https://github.com/cwsteinbach) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.61-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.61)
+ - Fail build if there are checkstyle violations [(#64)](https://github.com/linkedin/transport/pull/64)
+
+#### 0.0.60
+ - 2020-11-18 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.59...v0.0.60) by [Raymond](https://github.com/raymondlam12) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.60-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.60)
+ - Add Avro ENUM read support. [(#62)](https://github.com/linkedin/transport/pull/62)
+
+#### 0.0.59
+ - 2020-11-10 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.58...v0.0.59) by [curtiscwang](https://github.com/curtiscwang) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.59-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.59)
+ - Support conversion of String type to Utf8 in AvroWrapper [(#61)](https://github.com/linkedin/transport/pull/61)
+
+#### 0.0.58
+ - 2020-11-04 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.57...v0.0.58) by [Khai Tran](https://github.com/khaitranq) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.58-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.58)
+ - Support simple Avro Union schema [(#60)](https://github.com/linkedin/transport/pull/60)
+
+#### 0.0.57
+ - 2020-09-23 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.56...v0.0.57) by [Raymond](https://github.com/raymondlam12) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.57-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.57)
+ - Create a MutableMap when accessing keySet and values in SparkMap [(#58)](https://github.com/linkedin/transport/pull/58)
+
+#### 0.0.56
+ - 2020-08-17 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.55...v0.0.56) by [Shardul Mahadik](https://github.com/shardulm94) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.56-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.56)
+ - Fix test SQL generation for binary inputs [(#55)](https://github.com/linkedin/transport/pull/55)
+
+#### 0.0.55
+ - 2020-08-13 - [3 commits](https://github.com/linkedin/transport/compare/v0.0.54...v0.0.55) by [Shardul Mahadik](https://github.com/shardulm94) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.55-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.55)
+ - Bump shipkit version [(#54)](https://github.com/linkedin/transport/pull/54)
+ - Remove slf4j-log4j12 from Transport dependency graph [(#51)](https://github.com/linkedin/transport/pull/51)
+ - Hive: Struct data should not be converted to object array during StdStruct creation [(#50)](https://github.com/linkedin/transport/pull/50)
+
+#### 0.0.54
+ - 2020-08-10 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.53...v0.0.54) by [Shardul Mahadik](https://github.com/shardulm94) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.54-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.54)
+ - Publish Presto thin jar which allows consumers to control dependency graph [(#49)](https://github.com/linkedin/transport/pull/49)
+
+#### 0.0.53
+ - 2020-06-30 - [2 commits](https://github.com/linkedin/transport/compare/v0.0.52...v0.0.53) by [Shardul Mahadik](https://github.com/shardulm94) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.53-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.53)
+ - No pull requests referenced in commit messages.
+
+#### 0.0.52
+ - 2020-06-26 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.51...v0.0.52) by [John Joyce](https://github.com/jjoyce0510) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.52-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.52)
+ - No pull requests referenced in commit messages.
+
+#### 0.0.51
+ - 2020-06-23 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.50...v0.0.51) by [Khai Tran](https://github.com/khaitranq) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.51-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.51)
+ - Add support for StdFloat, StdDouble, and StdBinary [(#46)](https://github.com/linkedin/transport/pull/46)
+
+#### 0.0.50
+ - 2020-05-08 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.49...v0.0.50) by [Xingyuan Lin](https://github.com/lxynov) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.50-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.50)
+ - Upgrade to PrestoSQL 333 [(#45)](https://github.com/linkedin/transport/pull/45)
+
+#### 0.0.49
+ - 2020-04-30 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.48...v0.0.49) by [Shardul Mahadik](https://github.com/shardulm94) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.49-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.49)
+ - Presto: Make ScalarFunctionImplementation state independent of StdUdfWrapper [(#44)](https://github.com/linkedin/transport/pull/44)
+
+#### 0.0.48
+ - 2020-04-14 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.47...v0.0.48) by [Shardul Mahadik](https://github.com/shardulm94) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.48-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.48)
+ - Presto: Pass custom configuration object when using FileSystemUtils [(#43)](https://github.com/linkedin/transport/pull/43)
+
+#### 0.0.47
+ - 2020-04-01 - [2 commits](https://github.com/linkedin/transport/compare/v0.0.46...v0.0.47) by [Suren Nihalani](https://github.com/SurenNihalani) (1), [Sushant Raikar](https://github.com/HotSushi) (1) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.47-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.47)
+ - FileSystemUtils: remove an unreliable check for unit testing [(#42)](https://github.com/linkedin/transport/pull/42)
+ - Fixed Presto UDF patch's broken link [(#41)](https://github.com/linkedin/transport/pull/41)
+
+#### 0.0.46
+ - 2020-02-03 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.45...v0.0.46) by [Shardul Mahadik](https://github.com/shardulm94) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.46-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.46)
+ - Disable Jacoco for platform tests [(#37)](https://github.com/linkedin/transport/pull/37)
+
 #### 0.0.45
  - 2020-01-14 - [1 commit](https://github.com/linkedin/transport/compare/v0.0.44...v0.0.45) by [Suren Nihalani](https://github.com/SurenNihalani) - published to [![Bintray](https://img.shields.io/badge/Bintray-0.0.45-green.svg)](https://bintray.com/linkedin-transport/maven/transport/0.0.45)
  - Allow for absolute paths instead of assuming defaultFS in UDF's required files [(#33)](https://github.com/linkedin/transport/pull/33)

docs/transport-udfs-api.md

@@ -9,7 +9,8 @@ The `StdType` interface is the parent class of all type objects that
 are used to describe the schema of the data objects that can be
 manipulated by `StdUDFs`. Sub-interfaces of this interface include
 `StdIntegerType`, `StdBooleanType`, `StdLongType`, `StdStringType`,
-`StdArrayType`, `StdMapType`, `StdStructType`. Each sub-interface is
+`StdDoubleType`, `StdFloatType`, `StdBinaryType`, `StdArrayType`,
+`StdMapType`, and `StdStructType`. Each sub-interface is
 defined by methods that are specific to the corresponding type. For
 example, `StdMapType` interface is defined by the two methods shown
 below. The `keyType()` and `valueType()` methods can be used to obtain
@@ -39,9 +40,10 @@ public interface StdStructType extends StdType {
 manipulated by Transport UDFs. As a top-level interface, `StdData`
 itself does not contain any methods. A number of type-specific
 interfaces extend `StdData`, such as `StdInteger`, `StdLong`,
-`StdBoolean`, `StdString`, `StdArray`, `StdMap`, `StdStruct` to
-represent `INTEGER`, `LONG`, `BOOLEAN`, `VARCHAR`, `ARRAY`, `MAP`,
-`STRUCT` SQL types respectively. Each of those interfaces exposes
+`StdBoolean`, `StdString`, `StdDouble`, `StdFloat`, `StdBinary`,
+`StdArray`, `StdMap`, and `StdStruct` to represent `INTEGER`,
+`LONG`, `BOOLEAN`, `VARCHAR`, `DOUBLE`, `REAL`, `VARBINARY`, `ARRAY`, `MAP`,
+and `STRUCT` SQL types respectively. Each of those interfaces exposes
 operations that can manipulate that type of data. For example,
 `StdMap` interface is defined by the following methods:
 
@@ -108,6 +110,12 @@ definition:
   is StdInteger.
 * `"boolean"`: to represent SQL Boolean type. The respective Standard
   Type is StdBoolean.
+* `"double"`: to represent SQL Double type. The respective Standard
+  Type is StdDouble.
+* `"real"`: to represent SQL Real type. The respective Standard
+  Type is StdFloat.
+* `"varbinary"`: to represent SQL Binary type. The respective Standard
+  Type is StdBinary.
 * `"array(T)"`: to represent SQL Array type, with elements of type
   T. The respective Standard Type is StdArray.
 * `"map(K,V)"`: to represent SQL Map type, with keys of type K and
@@ -132,6 +140,9 @@ public interface StdFactory {
   StdLong createLong(long value);
   StdBoolean createBoolean(boolean value);
   StdString createString(String value);
+  StdDouble createDouble(double value);
+  StdFloat createFloat(float value);
+  StdBinary createBinary(ByteBuffer value);
   StdArray createArray(StdType stdType, int expectedSize);
   StdArray createArray(StdType stdType);
   StdMap createMap(StdType stdType);