Cascading Compatibility Test Suite
This project will test the binary compatibility of Cascading with differing releases of an underlying platform.
Cascading 3.1 is currently built and published against Apache Hadoop 1.2.x and Apache Hadoop 2.6.0. So is fully tested with those Apache releases.
This suite allows other distributions to be verified as being compatible with the current Cascading release as available through Maven and through other tools apart of the Cascading ecosystem.
To see the current results, visit the Cascading Compatibility page.
See the FAQ, included in this project, for more information.
Cascading is built against a target set of dependencies and made available through the Conjars.org Maven repository.
In the case of Apache Hadoop, and likely any future platforms, there are multiple stable releases, many of which are API backwards compatible, but not necessary binary backwards compatible.
A recompile of Cascading across each release should not be necessary. This test suite is intended to help identify which platform release is not compatible with a current Cascading stable release.
Running Compatibility Tests
To execute all tests across all platforms, install Gradle 1.0 and execute:
> gradle test
Testing a different distribution
Any non-Apache Haddoop distribution may fork this project and add a sub-project specific to the vendor distribution. You may want to send the test results to the Cascading project, so that your distribution will be listed on the Compatibility page.
To add a sub-project, copy the example project to a new directory named after the new distribution. For example,
> cp -R example-1.0 vendor-1.0
If your distribution is based on Hadoop2, use the
example-hadoop2-1.0 directory. In case you are testing a distro,
which supports Apache Tez, you can use the
distribution.properties file with all information about your distribution. Please make sure to fill in all
fields, if something does not apply to your distribution, leave it empty. The meaning of each property is explained
# the internal name of the distribution, used as the key distribution.name=vendor # the version of the distribution you are testing distribution.version=1.0.0 # the name used for displaying on http://cascading.org/support/compatibility distribution.displayname=HadoopVendor # the homepage of your distribution distribution.url=http://vendor.example.com # dowload link of your distribution distribution.downloadurl=http://vendor.example.com/1.0/download # release notes of the version at hand distribution.releaseurl=http://vendor.example.com/1.0/releasenotes.html # maven repository of the distribution distribution.mavenrepo=http://maven.example.com/repository # does the distribution ship with the cascading SDK? distribution.includessdk=yes # URL to the (commercial) support offering of the distribution distribution.supporturl # The version of yarn/hdfs supported by the distribution. Leave out for distros w/o YARN or HDFS distribution.yarnversion=2.6 # Name of the platform to be displayed on the compat page distribution.platformdisplayname=Apache Tez # version of the platform distribution.platformversion=0.6 # url of the platform distribution.platformurl=http://tez.apache.org
Afterwards add your distribution to the
settings.gradle to include the new directory
> echo "include 'vendor-1.0'" >> settings.gradle
Then make the necessary changes in the
vendor-1.0/build.gradle file. For example, pointing to a different
Maven repository, and updating the Gradle Maven dependencies.
The line loading the
hadoop-settings.gradle settings should not be changed.
When updated, to run the tests, call:
> gradle :vendor-1.0:test -i
If your distribution is not deployed to a maven repository, please see the FAQ on how to run the tests without one.
If the tests have to run on a remote cluster, change the test settings at the bottom of your
If you distribution supports both hadoop1 and hadoop2, you have to add 2 sub-projects to the test setup; one for each set of hadoop APIs.
Sending cascading compatibility results
The build contains a task to send test results back to the cascading project. If you want to send the results of your
distribution upstream use the
uploadResults tasks like this:
> gradle :vendor-1.0:test uploadResults -i
If you have multiple versions of your distribution and you want to upload results for all of them (which you should), add them to the same gradle run:
> gradle :vendor-1.0:test :vendor-2.0:test :vendor-42:test uploadResults -i
Please note, that the build will fail on the first test failure, if the
uploadResults task is enabled. This is
intentional to prevent reporting failed tests.
Please also note, that in order to guarantee the correctness of the compatibility page, the Cascading project can not list distributions, which did not follow this process.
Running a single test
> gradle -Dtest.single=CoGroupFieldedPipesPlatformTest :apache-0.20.2:test -i
-i allows you to see the tests running and is quite useful when calling just the
Creating an IntelliJ module file
This will create an
.iml file for each sub-project:
> gradle ideaModule