Skip to content
Permalink
Browse files
Refactored remaining test suites to use data preparation phases inste…
…ad of scripts

Also updated README.md and CONTRIBUTING.md to reflect the changes
  • Loading branch information
agozhiy committed Apr 22, 2022
1 parent 5adf73d commit fd0c90353a74027b5c4cf5eb91315f12a8bc9292
Showing 47 changed files with 1,013 additions and 439 deletions.
@@ -85,6 +85,18 @@ A test definition file is a JSON file that defines one or a set of tests within
}
</code></pre>

The data described in "datasources" section is prepared in several stages. The stages are defined by "mode" field and executed in the following order:
- **rm** - remove data specified in "dest" field
- **cp** - copy data from "src" in local file system to "dest" in the target fs
- **mkdir** - make directory specified in "dest"
- **gen** - execute command specified in "src". In case of a file with .ddl extension it will be send to Drill through JDBC
- **post_rm** - the same as "rm" but executed later
- **post_cp** - the same as "cp" but executed later
- **dfs_cp** - copy data between dfs locations
- **ddl** - executes specified ddl file through JDBC

Note that local paths are based on DRILL_TEST_DATA_DIR property (by default it is framework/resources), and all dfs paths are based on DRILL_TESTDATA property (by default it is /drill/testdata)

### Special handling for Drill version in regex based baseline verification

The Drill version in a regex based baseline file can be substituted with "{DRILL_VERSION}".
@@ -36,11 +36,25 @@ bin/build_framework -Pdownload

If you've already downloaded the datasets previously, you can simply skip the download.

If you're going to run tests in docker, you'll need to build the docker image first:

```
mvn clean install docker:build -DskipTests
```

## Execute Tests

In the root directory of your repository, execute the following command to run tests:

`bin/run_tests -s <suites> -g <groups> -t <Timeout> -x <Exclude> -n <Concurrency> -d`
```
bin/run_tests -s <suites> -g <groups> -t <Timeout> -x <Exclude> -n <Concurrency> -d`
```

Alternatively, you can run tests in docker:

```
mvn docker:start -Dtest.args="-s <suites> -g <groups> -t <Timeout> -x <Exclude> -n <Concurrency> -d"
```

Example:
<pre><code> bin/run_tests -s Functional/aggregates,Functional/joins -g functional -x hbase -t 180 -n 20 -d
@@ -57,7 +71,8 @@ Example:
-n concurrency (optional)
Here, '20' queries can execute concurrently
-x exclude dependencies (optional)
Here, any 'hbase' test suites within the specified directory are excluded
Here, any test suites within the specified directory that have specified dependencies are excluded.
In case of "-x all" all tests suites that have at least one dependency are excluded
-h help (optional)
Use this option to provide the usage of the command, which includes additional options
</code></pre>
@@ -1,5 +1,5 @@
#!/bin/bash
source conf/drillTestConfig.properties
DRILL_TEST_DATA_DIR=$1

set -x
set -e
@@ -39,21 +39,4 @@ if [ ! -d ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/json

fi

if ! $(hadoop fs -test -d ${DRILL_TESTDATA}/schema_change_empty_batch/json)
then

echo "Copying to hadoop"

hadoop fs -mkdir -p ${DRILL_TESTDATA}/schema_change_empty_batch/json

hadoop fs -put ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/json/part ${DRILL_TESTDATA}/schema_change_empty_batch/json/

hadoop fs -put ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/json/partsupp ${DRILL_TESTDATA}/schema_change_empty_batch/json/

hadoop fs -put ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/json/empty ${DRILL_TESTDATA}/schema_change_empty_batch/json/

hadoop fs -put ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/json/part_small ${DRILL_TESTDATA}/schema_change_empty_batch/json/

fi

set +x
@@ -1,5 +1,5 @@
#!/bin/bash
source conf/drillTestConfig.properties
DRILL_TEST_DATA_DIR=$1

set -x
set -e
@@ -35,24 +35,7 @@ if [ ! -d ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/psv/

touch ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/psv/partsupp/partsuppa{f..h}.tbl

touch ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/psv/empty/empty{a..d}.tbl

fi

if ! $(hadoop fs -test -d ${DRILL_TESTDATA}/schema_change_empty_batch/psv)
then

echo "Copying to hadoop"

hadoop fs -mkdir -p ${DRILL_TESTDATA}/schema_change_empty_batch/psv

hadoop fs -put ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/psv/part ${DRILL_TESTDATA}/schema_change_empty_batch/psv/

hadoop fs -put ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/psv/partsupp ${DRILL_TESTDATA}/schema_change_empty_batch/psv/

hadoop fs -put ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/psv/empty ${DRILL_TESTDATA}/schema_change_empty_batch/psv/

hadoop fs -put ${DRILL_TEST_DATA_DIR}/Datasources/schema_change_empty_batch/data/psv/json_field ${DRILL_TESTDATA}/schema_change_empty_batch/psv/
touch ${DRILL_TEST_DATA_DIR}/Datasources/spsv/json_field/empty_json_field.tblchema_change_empty_batch/data/psv/empty/empty{a..d}.tbl

fi

@@ -1,8 +1,7 @@
#!/bin/bash
source conf/drillTestConfig.properties
DRILL_TEST_DATA_DIR=$1

test_dir=${DRILL_TEST_DATA_DIR}/Datasources/table_function
dfs_location=${DRILL_TESTDATA}/table_function

untar_data () {
local location=$1
@@ -12,37 +11,21 @@ untar_data () {
tar xzvf $location/$tar_name -C $location ./$file_name
}

copy_to_dfs () {
local location=$1
local file_name=$2
local file=$location/$file_name
local destination=$3

hadoop fs -test -f $destination/$file_name
if [ $? -eq 0 ]
then
hadoop fs -rm -r $destination/$file_name
fi
hadoop fs -put $file $destination/
}

prepare_dataset () {
local location=$1
local file_name=$2
local file=$test_dir/$file_name
local tar_name=$3
local destination=$4

# Reusing of existing file if exists
if [ ! -f $file ]
then
untar_data $location $tar_name $file_name
copy_to_dfs $location $file_name $destination
fi
}

base_file_name="DRILL-5166_NPE_with_table_function"
tar_name="DRILL-5166_data.tar.gz"
prepare_dataset $test_dir ${base_file_name}_group_select_1.csv $tar_name $dfs_location
prepare_dataset $test_dir ${base_file_name}_group_select_2.csv $tar_name $dfs_location
prepare_dataset $test_dir ${base_file_name}_large_file.csv $tar_name $dfs_location
prepare_dataset $test_dir ${base_file_name}_group_select_1.csv $tar_name
prepare_dataset $test_dir ${base_file_name}_group_select_2.csv $tar_name
prepare_dataset $test_dir ${base_file_name}_large_file.csv $tar_name
@@ -1,4 +1,5 @@
set `store.parquet.reader.int96_as_timestamp`=true;
use dfs.drillTestDir;
analyze table `table_stats/alltypes_with_nulls` compute statistics;
analyze table `table_stats/date_intervals` compute statistics;
analyze table `table_stats/impala_parquet` compute statistics;
@@ -19,7 +19,7 @@
],
"datasources": [
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "complex/parquet/complex.json"
},
@@ -28,7 +28,7 @@
"dest": "partition_pruning/hive"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/existing_partition_pruning"
},
@@ -28,7 +28,7 @@
"dest": "partition_pruning/hive"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/existing_partition_pruning"
},
@@ -28,7 +28,7 @@
"dest": "partition_pruning/hive"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/existing_partition_pruning"
},
@@ -28,7 +28,7 @@
"dest": "partition_pruning/hive"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/existing_partition_pruning"
},
@@ -28,7 +28,7 @@
"dest": "partition_pruning/hive"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/existing_partition_pruning"
},
@@ -28,7 +28,7 @@
"dest": "partition_pruning/hive"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/existing_partition_pruning"
},
@@ -33,12 +33,12 @@
"dest": "/drill/testdata/drill-3947"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/existing_partition_pruning"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/tpch_single_partition1"
},
@@ -33,12 +33,12 @@
"dest": "/drill/testdata/drill-3947"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/existing_partition_pruning"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/tpch_single_partition1"
},
@@ -18,7 +18,7 @@
],
"datasources": [
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/tpch_multiple_partitions"
},
@@ -18,7 +18,7 @@
],
"datasources": [
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/tpch_multiple_partitions"
},
@@ -58,7 +58,7 @@
"dest": "Tpch0.01/parquet/orders/orders.parquet"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/tpch_single_partition"
},
@@ -18,7 +18,7 @@
],
"datasources": [
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/tpch_single_partition1"
},
@@ -18,7 +18,7 @@
],
"datasources": [
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_flatten"
},
@@ -18,7 +18,7 @@
],
"datasources": [
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_flatten"
},
@@ -18,7 +18,7 @@
],
"datasources": [
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas/parquet"
},
@@ -18,7 +18,7 @@
],
"datasources": [
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "filter/pushdown/varchar_decimal"
},
@@ -18,7 +18,7 @@
],
"datasources": [
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "filter/pushdown/varchar_decimal"
},
@@ -21,7 +21,7 @@
"dest": "subqueries"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "subqueries/hive1_parquet_part"
},
@@ -21,7 +21,7 @@
"dest": "subqueries"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "subqueries/hive1_parquet_part"
},
@@ -28,7 +28,7 @@
"dest": "partition_pruning/hive"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/existing_partition_pruning"
},
@@ -28,7 +28,7 @@
"dest": "partition_pruning/hive"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "ctas_auto_partition/existing_partition_pruning"
},
@@ -21,12 +21,12 @@
"dest": "subqueries"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "subqueries/optional_type_v.view.drill"
},
{
"mode": "del",
"mode": "rm",
"src": "",
"dest": "subqueries/required_type_v.view.drill"
},

0 comments on commit fd0c903

Please sign in to comment.