Use docker-compose to ease product tests execution

Thanks to that developer can easily set up docker test environment on
which he could run product tests.

presto-product-tests/README.md

@@ -2,14 +2,10 @@
 
 ## Test configuration
 
-Test assume that you have hadoop and presto cluster running. For setting up development environment we
-recommend presto-docker-devenv project. This presto-docker-devenv contains tools for building and running
-docker images to be used on local developer machine for easy setup of presto instance together with dependencies
-like hadoop. You can also start presto on your machine locally from IDE (please check main presto README).
+Test assume that you have hadoop and Presto cluster running. 
 
 To make it work you need either:
- - define environment variables for hadoop ${HADOOP_MASTER} and presto ${PRESTO_MASTER} with their IP numbers,
-and ${PRESTO_PRODUCT_TESTS_ROOT} with path to the presto product test directory
+ - define environment variables for hadoop ${HADOOP_MASTER} and Presto ${PRESTO_MASTER} with their IP numbers
  - create test-configuration-local.yaml with following (example content):
 
 ```
@@ -26,5 +22,104 @@ databases:
 Product tests are not run by default. To start them use run following command:
 
 ```
-java -jar target/presto-product-tests-*-executable.jar --config-local file://`pwd`/tempto-configuration-local.yaml
+java -jar target/presto-product-tests-*-executable.jar --config-local tempto-configuration-local.yaml
 ```
+
+## Running tests with Using preconfigured docker based clusters
+
+In case you do not have an access to hadoop cluster or do not want to spent your time on configuration, this section
+describes how to run product tests with preconfigured docker based clusters.
+
+### Prerequisites
+
+* docker >= 1.10
+
+[https://docs.docker.com/installation/#installation](https://docs.docker.com/installation/#installation)
+
+For linux users:
+```
+wget -qO- https://get.docker.com/ | sh
+```
+
+For Mac OS X you need to install docker-machine.
+
+[https://docs.docker.com/engine/installation/mac/](https://docs.docker.com/engine/installation/mac/).
+
+* docker-compose >= 1.6
+
+[https://docs.docker.com/compose/install/](https://docs.docker.com/compose/install/).
+
+```
+pip install docker-compose
+```
+
+> Note that if you are using Mac OS X and installed docker-toolbox to have docker, 
+> then docker-compose should be already installed on your system.
+
+### Running product tests
+
+Below manual describe how to set up and teardown docker clusters needed to run product tests.
+It also covers actual product tests execution with usage of these clusters.
+
+> Note that if you using Mac OS X you may need to configure your environment to make docker commands working. 
+> Additionally, please make sure that your docker machine is able to allocate at least 4GB memory.
+
+```
+docker-machine create -d virtualbox --virtualbox-memory 4096 <machine>
+eval $(docker env <machine>)
+```
+
+#### Configuration profiles
+
+Configuration profiles are stored in `presto-product-tests/etc` diretory. There are two such profiles:
+
+**distributed** - consists of single node (pseudo-distributed) hadoop cluster and multiple node presto cluster
+ with one coordinator node and at least one worker node. You can select number of workers by below command:
+
+    cd presto-product-tests/etc/distributed
+    docker-compose scale presto-worker=<N>
+
+**singlenode** - consists of single node (pseudo-distributed) hadoop cluster and one node presto cluster
+
+#### Execution
+
+1. Build Presto project (to have presto-server and product tests compiled).
+
+    ```
+    ./mvnw install
+    ```
+
+2. Create and start Presto cluster
+
+    ```
+    cd presto-product-tests/etc/<profile>
+    docker-compose up -d
+    ```
+
+3. Wait for Presto to be ready.
+
+    To see if all the components are ready you can use below command.
+
+    ```
+    select count(node_id) from system.runtime.nodes where state = 'active';
+    ```
+
+4. Run product tests
+
+    ```
+    cd presto-product-tests/etc/<profile>
+    java -jar ../target/presto-product-tests-<version>-executable.jar 
+    ```
+
+    > Note that some tests may run queries too big to fit into docker resource constraints.
+    > To exclude these tests from execution you use below switch to run product tests command.
+    > `-x big_query,quarantine`
+
+    You can run product tests from your IDE, all you need to set is to set build directory to ```presto-product-tests/etc```.
+
+5. Stop clusters
+
+    ```
+    cd presto-product-tests/etc/<profile>
+    docker-compose stop
+    ```

presto-product-tests/etc/distributed/docker-compose.yml

@@ -0,0 +1,36 @@
+version: '2'
+services:
+  hadoop-master:
+    hostname: hadoop-master
+    image: 'teradatalabs/cdh5-hive'
+    ports:
+      - '8020:8020'
+      - '8088:8088'
+      - '9083:9083'
+      - '10000:10000'
+      - '50070:50070'
+      - '50075:50075'
+
+  presto-master:
+    hostname: presto-master
+    build: 'presto-dev-env'
+    image: 'teradatalabs/presto-dev-env-distributed'
+    ports: 
+      - '8080:8080'
+    depends_on:
+      - 'hadoop-master'
+    command: /presto-launcher-wrapper.sh master run
+    volumes:
+      - ../../../presto-server/target/:/presto-server/target:ro
+    env_file:
+        - ../../target/classes/presto.env
+
+  presto-worker:
+    image: 'teradatalabs/presto-dev-env-distributed'
+    depends_on:
+      - 'presto-master'
+    command: /presto-launcher-wrapper.sh worker run
+    volumes_from: 
+      - presto-master
+    env_file:
+        - ../../target/classes/presto.env

presto-product-tests/etc/distributed/presto-dev-env/Dockerfile

@@ -0,0 +1,7 @@
+FROM teradatalabs/centos6-java8-oracle
+MAINTAINER Grzegorz Kokosiński <grzegorz.kokosinski@teradata.com>
+
+COPY /presto-launcher-wrapper.sh /
+COPY catalog /etc/catalog
+COPY master /etc/master
+COPY worker /etc/worker

presto-product-tests/etc/distributed/presto-dev-env/catalog/blackhole.properties

@@ -0,0 +1 @@
+connector.name=blackhole

presto-product-tests/etc/distributed/presto-dev-env/catalog/hive.properties

@@ -0,0 +1,5 @@
+connector.name=hive-cdh5
+hive.metastore.uri=thrift://hadoop-master:9083
+hive.allow-drop-table=true
+hive.allow-rename-table=true
+hive.metastore-cache-ttl=0s

presto-product-tests/etc/distributed/presto-dev-env/catalog/jmx.properties

@@ -0,0 +1 @@
+connector.name=jmx

presto-product-tests/etc/distributed/presto-dev-env/catalog/tpch.properties

@@ -0,0 +1,3 @@
+connector.name=tpch
+tpch.splits-per-node=4
+

presto-product-tests/etc/distributed/presto-dev-env/master/config.properties

@@ -0,0 +1,11 @@
+node.id=will-be-overwritten
+node.environment=test
+
+coordinator=true
+node-scheduler.include-coordinator=false
+http-server.http.port=8080
+query.max-memory=1GB
+query.max-memory-per-node=512MB
+discovery-server.enabled=true
+discovery.uri=http://presto-master:8080
+plugin.config-dir=/etc/catalog

presto-product-tests/etc/distributed/presto-dev-env/master/jvm.config

@@ -0,0 +1,14 @@
+-server
+-Xmx2G
+-XX:+UseG1GC
+-XX:+UseGCOverheadLimit
+-XX:+ExplicitGCInvokesConcurrent
+-XX:+CMSClassUnloadingEnabled
+-XX:+AggressiveOpts
+-XX:+HeapDumpOnOutOfMemoryError
+-XX:OnOutOfMemoryError=kill -9 %p
+-XX:ReservedCodeCacheSize=150M
+-DHADOOP_USER_NAME=hive
+-Duser.timezone=UTC
+-Xdebug 
+-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005

presto-product-tests/etc/distributed/presto-dev-env/master/log.properties

@@ -0,0 +1,5 @@
+com.facebook.presto=INFO
+com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory=WARN
+com.ning.http.client=DEBUG
+com.facebook.presto.server.PluginManager=DEBUG
+io.airlift.discovery.client=INFO

presto-product-tests/etc/distributed/presto-dev-env/presto-launcher-wrapper.sh

@@ -0,0 +1,17 @@
+#!/bin/bash -xe
+
+CONFIG=$1
+
+if [[ "$CONFIG" != "master" && "$CONFIG" != "worker" ]]; then
+   echo "Usage: launcher-wrapper <master|worker> <launcher args>"
+   exit 1
+fi
+
+shift 1
+presto-server/target/presto-server-$PRESTO_VERSION/bin/launcher \
+  -Dnode.id=$HOSTNAME \
+  --config=/etc/$CONFIG/config.properties \
+  --jvm-config=/etc/$CONFIG/jvm.config \
+  --log-levels-file=/etc/$CONFIG/log.properties \
+  --data-dir=/var/presto \
+  $* 

presto-product-tests/etc/distributed/presto-dev-env/worker/config.properties

@@ -0,0 +1,10 @@
+node.id=will-be-overwritten
+node.environment=test
+
+coordinator=false
+http-server.http.port=8081
+query.max-memory=1GB
+query.max-memory-per-node=512MB
+discovery-server.enabled=false
+discovery.uri=http://presto-master:8080
+plugin.config-dir=/etc/catalog

presto-product-tests/etc/distributed/presto-dev-env/worker/jvm.config

@@ -0,0 +1,14 @@
+-server
+-Xmx2G
+-XX:+UseG1GC
+-XX:+UseGCOverheadLimit
+-XX:+ExplicitGCInvokesConcurrent
+-XX:+CMSClassUnloadingEnabled
+-XX:+AggressiveOpts
+-XX:+HeapDumpOnOutOfMemoryError
+-XX:OnOutOfMemoryError=kill -9 %p
+-XX:ReservedCodeCacheSize=150M
+-DHADOOP_USER_NAME=hive
+-Duser.timezone=UTC
+-Xdebug 
+-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005

presto-product-tests/etc/distributed/presto-dev-env/worker/log.properties

@@ -0,0 +1,5 @@
+com.facebook.presto=INFO
+com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory=WARN
+com.ning.http.client=DEBUG
+com.facebook.presto.server.PluginManager=DEBUG
+io.airlift.discovery.client=INFO

presto-product-tests/etc/distributed/tempto-configuration-local.yaml

@@ -0,0 +1,7 @@
+hosts:
+  hadoop-master: ${DOCKER_MACHINE_HOST} 
+databases:
+  hive:
+    host: ${DOCKER_MACHINE_HOST}
+  presto:
+    host: ${DOCKER_MACHINE_HOST}

presto-product-tests/etc/singlenode/docker-compose.yml

@@ -0,0 +1,26 @@
+version: '2'
+services:
+  hadoop-master:
+    hostname: hadoop-master
+    image: 'teradatalabs/cdh5-hive'
+    ports:
+      - '8020:8020'
+      - '8088:8088'
+      - '9083:9083'
+      - '10000:10000'
+      - '50070:50070'
+      - '50075:50075'
+
+  presto-master:
+    hostname: presto-master
+    build: 'presto-dev-env'
+    image: 'teradatalabs/presto-dev-env-singlenode'
+    ports: 
+      - '8080:8080'
+    depends_on:
+      - 'hadoop-master'
+    command: /presto-launcher-wrapper.sh run
+    volumes:
+      - ../../../presto-server/target/:/presto-server/target:ro
+    env_file:
+      - ../../target/classes/presto.env

presto-product-tests/etc/singlenode/presto-dev-env/Dockerfile

@@ -0,0 +1,6 @@
+FROM teradatalabs/centos6-java8-oracle
+MAINTAINER Grzegorz Kokosiński <grzegorz.kokosinski@teradata.com>
+
+COPY /presto-launcher-wrapper.sh /
+COPY catalog /etc/catalog
+COPY master /etc/master

presto-product-tests/etc/singlenode/presto-dev-env/catalog/blackhole.properties

@@ -0,0 +1 @@
+connector.name=blackhole

presto-product-tests/etc/singlenode/presto-dev-env/catalog/hive.properties

@@ -0,0 +1,5 @@
+connector.name=hive-cdh5
+hive.metastore.uri=thrift://hadoop-master:9083
+hive.allow-drop-table=true
+hive.allow-rename-table=true
+hive.metastore-cache-ttl=0s

presto-product-tests/etc/singlenode/presto-dev-env/catalog/jmx.properties

@@ -0,0 +1 @@
+connector.name=jmx

presto-product-tests/etc/singlenode/presto-dev-env/catalog/tpch.properties

@@ -0,0 +1,3 @@
+connector.name=tpch
+tpch.splits-per-node=4
+

presto-product-tests/etc/singlenode/presto-dev-env/master/config.properties

@@ -0,0 +1,11 @@
+node.id=will-be-overwritten
+node.environment=test
+
+coordinator=true
+node-scheduler.include-coordinator=true
+http-server.http.port=8080
+query.max-memory=1GB
+query.max-memory-per-node=512MB
+discovery-server.enabled=true
+discovery.uri=http://presto-master:8080
+plugin.config-dir=/etc/catalog

presto-product-tests/etc/singlenode/presto-dev-env/master/jvm.config

@@ -0,0 +1,14 @@
+-server
+-Xmx2G
+-XX:+UseG1GC
+-XX:+UseGCOverheadLimit
+-XX:+ExplicitGCInvokesConcurrent
+-XX:+CMSClassUnloadingEnabled
+-XX:+AggressiveOpts
+-XX:+HeapDumpOnOutOfMemoryError
+-XX:OnOutOfMemoryError=kill -9 %p
+-XX:ReservedCodeCacheSize=150M
+-DHADOOP_USER_NAME=hive
+-Duser.timezone=UTC
+-Xdebug 
+-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005

presto-product-tests/etc/singlenode/presto-dev-env/master/log.properties

@@ -0,0 +1,5 @@
+com.facebook.presto=INFO
+com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory=WARN
+com.ning.http.client=DEBUG
+com.facebook.presto.server.PluginManager=DEBUG
+io.airlift.discovery.client=INFO

presto-product-tests/etc/singlenode/presto-dev-env/presto-launcher-wrapper.sh

@@ -0,0 +1,9 @@
+#!/bin/bash -xe
+
+presto-server/target/presto-server-$PRESTO_VERSION/bin/launcher \
+  -Dnode.id=$HOSTNAME \
+  --config=/etc/master/config.properties \
+  --jvm-config=/etc/master/jvm.config \
+  --log-levels-file=/etc/master/log.properties \
+  --data-dir=/var/presto \
+  $* 

presto-product-tests/etc/singlenode/tempto-configuration-local.yaml

@@ -0,0 +1,7 @@
+hosts:
+  hadoop-master: ${DOCKER_MACHINE_HOST} 
+databases:
+  hive:
+    host: ${DOCKER_MACHINE_HOST}
+  presto:
+    host: ${DOCKER_MACHINE_HOST}

presto-product-tests/pom.xml

@@ -81,6 +81,12 @@
     </dependencies>
 
     <build>
+        <resources>
+            <resource>
+                <directory>${project.basedir}/src/main/resources</directory>
+                <filtering>true</filtering>
+            </resource>
+        </resources>
         <plugins>
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>