[BIGTOP-2961] Add Livy packages

[akitanaka]

Adding Apache Livy (incubating) package to BigTop. The Livy version is 0.6.0-incubating.

[evans-ye]

Tested on

• CentOS 7: https://ci.bigtop.apache.org/view/Packages/job/Manual-build-deploy-smoke-test/13/console
• Ubuntu 16.04: https://ci.bigtop.apache.org/view/Packages/job/Manual-build-deploy-smoke-test/15/console

From the error log it seems that packaging livy require higher version of setuptool:
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)

Would you like to fire up another patch that fix the build env requirement as well?
We handle the env stuffs in Bigtop Toolchain: https://github.com/apache/bigtop/tree/master/bigtop_toolchain

Let me know what you think :)

[oflebbe]

That might be solvable for centos7 by running

yum install python2-pip
python -m pip install --upgrade pip setuptools wheel

(if livy is using python2)

It would make sense to include this into bigtop/slave creation for bigtop 1.5.0

@evans-ye : Could you possibly try to include this two commands into the test-compile script ?

[evans-ye]

I tried something like this:
python -m pip install --upgrade pip setuptools wheel

Is able to pass the setuptool version check, however hit another problem:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-api verify) on project livy-python-api: Command execution failed.: Cannot run program "flake8" (in directory "/bigtop/build/livy/rpm/BUILD/livy-0.6.0/python-api"): error=2, No such file or directory -> [Help 1]

Did not dig into it yet. Anyone have a clue?

[evans-ye]

@akitanaka Awesome! I saw your patch. Would you please separate the commit 350daf8 out of this PR?
You can create another JIRA and create a new PR containing 350daf8. That way we have a clean view on the commits and also I can try to build images to test out livy.

[akitanaka]

Hello, I created BIGTOP-3214 and following PR to update setuptools version and install flake8. (#530) Please let me know this works on your env. Thank you.