Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 06: Initial drop of pravega connector #7

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/pravega-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: Java CI

on: [push]

jobs:
build:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- name: Set up JDK 1.8
uses: actions/setup-java@v1
with:
java-version: 1.8
- name: Build with Gradle
run: ./gradlew build
28 changes: 28 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
*.iml
*.ipr
*.iws
.gradle
build/
/var
/*/var/
/presto-product-tests/**/var/
test-output/
test-reports/
out/
/atlassian-ide-plugin.xml
.idea
.DS_Store
.classpath
.settings
.project
temp-testng-customsuite.xml
test-output
.externalToolBuilders
*~
benchmark_outputs
*.pyc
*.class
.checkstyle
.editorconfig
node_modules

28 changes: 15 additions & 13 deletions HEADER
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
Copyright (c) Pravega Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
/*
* Copyright (c) Pravega Authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
79 changes: 78 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,79 @@
# presto-connector
Pravega connector for Presto
Pravega Presto connector

Presto is a distributed SQL query engine for big data. Presto uses connectors to query storage from different storage sources. This repository contains the code for a connector (the Pravega Presto connector) to query storage from Pravega streams. To learn more about PrestoDB, visit https://prestodb.io

Pravega is an open source distributed storage service implementing Streams. It offers Stream as the main primitive for the foundation of reliable storage systems: a high-performance, durable, elastic, and unlimited append-only byte stream with strict ordering and consistency. To learn more about Pravega, visit https://pravega.io

See the [User Manual](https://prestodb.github.io/docs/current/) for Presto deployment instructions and end user documentation.

## Requirements

To build and run the Pravega Presto connector, you must meet the following requirements:

* Linux
* Java 8 Update 151 or higher (8u151+), 64-bit. Both Oracle JDK and OpenJDK are supported.
* Gradle 6.5.1+ (for building)
* Python 2.7+ (for running with the launcher script)
* Pravega version 0.9.0 or higher
* Pravega Schema Registry version 0.2.0 or higher

## Building Presto

Pravega Presto connector is a standard Gradle project. Simply run the following command from the project root directory:

./gradlew clean build

On the first build, Gradle will download all the dependencies from various locations of the internet and cache them in the local repository (`~/.gradle / caches `), which can take a considerable amount of time. Subsequent builds will be faster.

Pravega Presto connector has a set of unit tests that can take several minutes to run. You can run the tests using this command:

./gradlew test

## Running Presto

The plugin file that gets created during the build process is: ./build/distributions/pravega-<VERSION>.tar.gz. This file can be untar'd in the /usr/lib/presto/lib/plugins directory of a running Presto installation. Like all Presto connectors, the Pravega Presto connector uses a properties files to point to the storage provider (e.g. Pravega controller). Create a properties file similar to below, but replace the # characters with the appropriate IP address of the Pravega Controller and the Pravega Schema Registry server of your configuration.

[root@lrmk226 ~]# cat /etc/presto/catalog/pravega.properties
connector.name=pravega
pravega.controller=tcp://##.###.###.###:9090
pravega.schema-registry=http://##.###.###.###:9092

## Running Presto in your IDE

After building Presto for the first time, you can load the project into your IDE and run the server in your IDE. We recommend using [IntelliJ IDEA](http://www.jetbrains.com/idea/). Because Pravega Presto connectoris a standard Gradle project, you can import it into your IDE. In IntelliJ, choose Import Project from the Quick Start box and point it to the root of the source tree. IntelliJ will identify the *.gradle files and prompt you to confirm.

After opening the project in IntelliJ, double check that the Java SDK is properly configured for the project:

* Open the File menu and select Project Structure
* In the SDKs section, ensure that a 1.8 JDK is selected (create one if none exist)
* In the Project section, ensure the Project language level is set to 8.0 as Presto makes use of several Java 8 language features

Use the following options to create a run configuration that runs the Presto server using the Pravega Presto connector:

* Main Class: 'com.facebook.presto.server.PrestoServer'
* VM Options: '-ea -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -Xmx2G -Dconfig=etc/config.properties -Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true -Dlog.levels-file=etc/log.properties'
* Working directory: '/root/presto'
* Use classpath of module: 'pravega.main'

The working directory should be manually created and set to where the configuration properties files are located on your host.

[root@lrmk226 ~]# find /root/presto -ls
537429121 0 drwxr-xr-x 4 root root 28 Mar 10 10:40 /root/presto
537429123 0 drwxr-xr-x 4 root root 97 Mar 2 17:12 /root/presto/etc
805328026 0 drwxr-xr-x 2 root root 32 Mar 3 15:58 /root/presto/etc/catalog
816318053 4 -rw-r--r-- 1 root root 119 Mar 3 15:58 /root/presto/etc/catalog/pravega.properties
272367596 0 drwxr-xr-x 2 root root 6 Mar 2 17:04 /root/presto/etc/ecs
537429402 4 -rw-r--r-- 1 root root 854 Mar 2 17:10 /root/presto/etc/config.properties
537429389 4 -rw-r--r-- 1 root root 351 Mar 2 17:11 /root/presto/etc/jvm.config
537435775 4 -rw-r--r-- 1 root root 378 Mar 2 17:12 /root/presto/etc/log.properties
575879 0 drwxr-xr-x 3 root root 17 Mar 2 17:17 /root/presto/var
268833640 0 drwxr-xr-x 2 root root 30 Mar 10 10:42 /root/presto/var/log
272373839 12 -rw-r--r-- 1 root root 11230 Mar 10 10:42 /root/presto/var/log/http-request.log

Create the pravega.properties file as previously described.

## Schema Definitions

Currently, you must manually create schema definitions using a JSON file. In future releases, the 'CREATE TABLE' Presto command will be available.

102 changes: 102 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
/*
* Build gradle file for the Pravega connector
*/

plugins {
id 'java'
id 'distribution'
id 'checkstyle'
}

apply from: "$rootDir/gradle/checkstyle.gradle"

repositories {
mavenLocal()
maven {
url = uri('https://oss.jfrog.org/jfrog-dependencies')
}

maven {
url = uri('https://jitpack.io')
}

maven {
url = uri('https://repo.maven.apache.org/maven2')
}
}

dependencies {
compile 'com.facebook.airlift:bootstrap:0.191'
compile 'com.facebook.airlift:json:0.191'
compile 'com.facebook.airlift:log:0.191'
compile 'com.facebook.airlift:configuration:0.191'
compile 'com.google.guava:guava:26.0-jre'
compile 'com.google.inject:guice:4.2.0'
compile 'javax.validation:validation-api:1.1.0.Final'
compile 'javax.inject:javax.inject:1'
compile 'com.fasterxml.jackson.core:jackson-databind:2.10.0'
compile 'io.pravega:pravega-client:0.9.0-2705.09f82eb-SNAPSHOT'
compile 'io.pravega:pravega-common:0.9.0-2705.09f82eb-SNAPSHOT'
compile 'io.pravega:pravega-shared-protocol:0.9.0-2705.09f82eb-SNAPSHOT'
compile 'com.google.protobuf:protobuf-java:3.11.4'
compile 'com.github.everit-org.json-schema:org.everit.json.schema:1.12.1'
compile 'org.apache.avro:avro:1.8.1'
compile 'org.apache.commons:commons-lang3:3.7'
compile 'io.pravega:schemaregistry-contract:0.2.0-50.f1b6734-SNAPSHOT'
compile 'io.pravega:schemaregistry-common:0.2.0-50.f1b6734-SNAPSHOT'
compile 'io.pravega:schemaregistry-client:0.2.0-50.f1b6734-SNAPSHOT'
compile 'io.pravega:schemaregistry-serializers:0.2.0-50.f1b6734-SNAPSHOT'
compile 'io.pravega:schemaregistry-serializers-shared:0.2.0-50.f1b6734-SNAPSHOT'
compile 'io.pravega:schemaregistry-serializers-json:0.2.0-50.f1b6734-SNAPSHOT'

compile 'com.facebook.presto:presto-main:0.247'
compile 'com.facebook.presto:presto-record-decoder:0.247'
compile 'com.facebook.presto:presto-spi:0.247'
compile 'com.facebook.presto:presto-common:0.247'

runtimeOnly 'io.airlift:joda-to-java-time-bridge:3'
runtimeOnly 'com.facebook.airlift:log-manager:0.191'
runtimeOnly 'org.apache.zookeeper:zookeeper:3.5.7'
runtimeOnly 'com.101tec:zkclient:0.10'

testImplementation 'com.facebook.presto:presto-tests:0.247'
testImplementation 'org.testng:testng:7.3.0'
testImplementation 'com.facebook.airlift:testing:0.191'

compileOnly 'io.airlift:slice:0.38'
compileOnly 'io.airlift:units:1.3'
compileOnly 'com.fasterxml.jackson.core:jackson-annotations:2.10.0'
compileOnly 'org.openjdk.jol:jol-core:0.2'
}

group = 'com.facebook.presto'
version = '0.1.0'
description = 'Pravega SQL :: Pravega PrestoDB Connector'
sourceCompatibility = '1.8'

task getHomeDir {
doLast {
println gradle.gradleHomeDir
}
}

plugins.withType(DistributionPlugin) {
distTar {
compression = Compression.GZIP
extension = 'tar.gz'
}
}

distributions {
main {
contents {
distributionBaseName = 'pravega'
from jar
from(project.configurations.runtime)
}
}
}

test {
useTestNG()
}
Loading