Skip to content
Permalink
Browse files
BIGTOP-2227. smoke-tests should be a part of the top-level project, n…
…ot separated
  • Loading branch information
Konstantin Boudnik committed Jan 12, 2016
1 parent 6c9829e commit 5d63481d533e79f89f2be56dede58c7c2bb29fc5
Show file tree
Hide file tree
Showing 12 changed files with 73 additions and 92 deletions.
@@ -97,10 +97,7 @@ For specific questions it's always a good idea to ping the mailing list at dev-s
For Users: Running the smoke tests.
-----------------------------------

The simplest way to test bigtop is to:

* Step 1: cd bigtop-tests/smoke-tests/
* Step 2: Follow the instructions in the smoke-tests/README file.
The simplest way to test bigtop is described in bigtop-tests/smoke-tests/README file

For integration (API level) testing with maven, read on.

@@ -28,15 +28,13 @@ fi
echo -e "\n===== START TO RUN SMOKE TESTS: $SMOKE_TESTS =====\n"

export HADOOP_CONF_DIR=/etc/hadoop/conf/
export BIGTOP_HOME=/bigtop-home/
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/
export HIVE_HOME=/usr/lib/hive/
export PIG_HOME=/usr/lib/pig/
export FLUME_HOME=/usr/lib/flume/
export SQOOP_HOME=/usr/lib/sqoop/
export HIVE_CONF_DIR=/etc/hive/conf/
export MAHOUT_HOME="/usr/lib/mahout"
export ITEST="1.0.0"

su -s /bin/bash $HCFS_USER -c '/usr/bin/hadoop fs -mkdir /user/vagrant /user/root'
su -s /bin/bash $HCFS_USER -c 'hadoop fs -chmod 777 /user/vagrant'
@@ -47,4 +45,8 @@ if [ -f /etc/debian_version ] ; then
else
yum install -y pig hive flume mahout sqoop
fi
cd /bigtop-home && ./gradlew -b bigtop-tests/smoke-tests/build.gradle clean test -Dsmoke.tests=$SMOKE_TESTS --info
ALL_SMOKE_TASKS=""
for s in `echo $SMOKE_TESTS | sed -e 's#,# #g'`; do
ALL_SMOKE_TASKS="$ALL_SMOKE_TASKS bigtop-tests:smoke-tests:$s:test"
done
cd /bigtop-home && ./gradlew clean $ALL_SMOKE_TASKS -Psmoke.tests --info
@@ -18,7 +18,6 @@ So overall, this framework makes testing with apache bigtop easy as well as fun!
You need to export several environment variables. For example,
```
export HADOOP_CONF_DIR=/etc/hadoop/conf/
export BIGTOP_HOME=/bigtop-home/
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/
export HIVE_HOME=/usr/lib/hive/
export PIG_HOME=/usr/lib/pig/
@@ -28,19 +27,33 @@ export JAVA_HOME="/usr/lib/jvm/java-openjdk/"
export MAHOUT_HOME="/usr/lib/mahout"
export KITE_HOME="/usr/lib/kite"
export SPARK_HOME="/usr/lib/spark"
export ITEST="1.0.0"
```

Then, simply invoke the tests including the smoke.tests you want to run, as described below.
Then, simply invoke the tests you want to run, as described below.

Each directory is a gradle "subproject" mapping to an ecosystem component.
If you specify, for example, "flume", then the flume test runs. To test hive as well,
you would specify "flume,hive", and so on. To run the tests from the top-level
Invoking test task of flume subproject will execute flume tests. To test hive as well,
you would add the invocation of hive:test, and so on. To run the tests from the top-level
Bigtop folder, follow the example below:
```
./gradlew -b bigtop-tests/smoke-tests/build.gradle clean test -Dsmoke.tests=flume,hive --info
./gradlew bigtop-tests:smoke-tests:flume:test bigtop-tests:smoke-tests:hive:test -Psmoke.tests --info
```
You also can run smoke tests from bigtop-tests/smoke-tests directory,
or from any subproject of it. E.g.
```
cd bigtop-tests/smoke-tests
../../gradlew hive:test -Psmoke.tests
```
or
```
cd bigtop-tests/smoke-tests/hive
../../../gradlew test -Psmoke.tests
```
Instead of using top-level Gradle wrapper one can also use system-wide
Gradle, if installed and available.

Setting up project property smoke.tests is important, cause this is
required to to enable test tasks for smoke projects.
The --info option is a nice feature to have here: You can easily debug your tests this way,
just make sure you have log4j.properties in your conf/ directory locally.

@@ -15,14 +15,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
allprojects {
apply plugin: 'groovy'

repositories {
mavenCentral()
}
}

subprojects {
/**
* Utility function for tests to use to confirm EVN Variables.
@@ -32,21 +24,20 @@ subprojects {
def value = System.getenv("${it}")
if (value == null || value == "null")
throw new GradleException("undeclared env variable: ${it}")
else
println("ENV VARIABLE: ${it} = $value");
}
}

/**
* Two important environment variables.
* java_home should be declared in all cases.
* itest can be greater than or equal to = 0.7.0, 0.8.0
*/
checkEnv(["JAVA_HOME","ITEST","BIGTOP_HOME"])
checkEnv(["JAVA_HOME"])

ext.itestVersion = System.getenv("ITEST")
ext.groovyVersion = '1.8.0'
ext.hadoopVersion = '2.6.0'
// itest needs be greater than or equal to = 1.0.0
ext.itestVersion = '1.0.0' // Might need to be able to read an input for alternate version?
ext.BIGTOP_HOME = rootDir

dependencies {
//needed to avoid groovy not on classpath error.
@@ -81,9 +72,8 @@ subprojects {
return !keep_this_test;
}

println("Now testing...");
test {

test << {
println("Now testing...");
//todo, add back in 'basic' after BIGTOP-1392 .
testLogging {
events "passed", "skipped", "failed"
@@ -26,8 +26,8 @@ def tests_to_include() {
sourceSets {
test {
groovy {
srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/hive/",
"${System.env.BIGTOP_HOME}/bigtop-tests/smoke-tests/hive/"]
srcDirs = ["${BIGTOP_HOME}/bigtop-tests/test-artifacts/hive/",
"${BIGTOP_HOME}/bigtop-tests/smoke-tests/hive/"]
exclude { FileTreeElement elem -> (doExclude(elem.getName())) }
}
}
@@ -39,7 +39,5 @@ test {
}

test.doFirst {
// BIGTOP_HOME is necessary here, since we use it to read in from
// test-artifacts.
checkEnv(["BIGTOP_HOME", "HIVE_HOME", "HIVE_CONF_DIR", "BIGTOP_HOME"])
checkEnv(["HIVE_HOME", "HIVE_CONF_DIR"])
}
@@ -24,11 +24,11 @@ def tests_to_include() {
sourceSets {
test {
groovy {
srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/smoke-tests/ignite-hadoop/"]
srcDirs = ["${BIGTOP_HOME}/bigtop-tests/smoke-tests/ignite-hadoop/"]
}
}
}

test.doFirst {
checkEnv(["BIGTOP_HOME", "HADOOP_MAPRED_HOME"])
checkEnv(["HADOOP_MAPRED_HOME"])
}
@@ -24,12 +24,12 @@ def tests_to_include() {
sourceSets {
test {
groovy {
srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/mahout/"]
srcDirs = ["${BIGTOP_HOME}/bigtop-tests/test-artifacts/mahout/"]
exclude { FileTreeElement elem -> (doExclude(elem.getName())) }
}
}
}

test.doFirst {
checkEnv(["BIGTOP_HOME", "MAHOUT_HOME"])
checkEnv(["MAHOUT_HOME"])
}
@@ -25,7 +25,7 @@ sourceSets {
test {
groovy {
srcDirs = [
"${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/mapreduce"
"${BIGTOP_HOME}/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/mapreduce"
]
exclude {
FileTreeElement elem -> (doExclude(elem.getName()))
@@ -26,7 +26,7 @@ def tests_to_include() {
sourceSets {
test {
groovy {
srcDirs = ["./"]
srcDirs = [projectDir]
exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/**'
exclude { FileTreeElement elem -> (doExclude(elem.getName())) }
}

This file was deleted.

@@ -172,6 +172,24 @@ project(':itest-common') {
}
}

/**
* checkSmokeTestProjects() configures smoke-tests:<component>:test task only
* in if the project has smoke.tests properties set.
* This is done to avoid running cluster smoke tests during the normal life-cycle
* of the project development
*/
def checkSmokeTestProjects = {
FileTree fTree = fileTree(dir: 'bigtop-tests/smoke-tests', include: '*/build.gradle')
fTree.each() { smokeProject ->
def parent = smokeProject.getParentFile().name
project (":bigtop-tests:smoke-tests:$parent") {
test {
onlyIf {project.hasProperty('smoke.tests')}
}
}
}
}

def TESTARTIFACTS_GROUP = 'test artifacts'
def DEVENV_GROUP = 'development tools'
def DEPLOY_GROUP = 'deployment'
@@ -389,7 +407,8 @@ task "bigtop-slaves"(dependsOn: 'gen-gradle-home', type:Exec,
commandLine command
}

project.afterEvaluate{
project.afterEvaluate {
checkSmokeTestProjects()
artifactToInstall(dependsOn: [installTopLevel, installCommon, installConf, installiTest])
}

@@ -18,5 +18,17 @@
rootProject.name = 'bigtop'

include 'bigtop-test-framework', 'bigtop-tests:smoke-tests'

project(":bigtop-test-framework").name = 'itest-common'

/**
* The following code will does include all subprojects of smoke-tests
* so they are lined-up properly in the structure of the Bigtop project.
* The inclusion is done dynamically, so when new smoke test projects
* there's no extra effort to get it running.
*/
FileTree fTree = fileTree(dir: 'bigtop-tests/smoke-tests', include: '*/build.gradle')
fTree.each() { buildFile ->
def parent = buildFile.getParentFile().name
include("bigtop-tests:smoke-tests:$parent")
}

0 comments on commit 5d63481

Please sign in to comment.