Adding YARNContainerFactory. This allows OpenWhisk to run actions on Apache Hadoop clusters.

[SamHjelmfelt]

Thousands of organizations have Apache Hadoop clusters today. By implementing a YARNContainerFactory, OpenWhisk can run actions on Hadoop clusters. This will lower the barrier to adoption and expand the potential use cases for OpenWhisk.

Description

The YARNContainerFactory uses the Apache Hadoop Services REST API to create a single YARN service with a component for each action type. Both simple authentication and Kerberos/SPNEGO are supported. This was tested with Apache Hadoop 3.1.1.
https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html
The implemention is based on the MesosContainerFactory.

This was first implemented using Akka HTTP, but was rewritten to use the Apache HTTP client in order to suport SPNEGO.

There is a MockYARNRM in the tests directory. This mock RM simulates the YARN Resource Manager REST API and is used for the YARNContainerFactory tests.

Related issue and scope

• I opened an issue to propose and discuss this change (#????)

My changes affect the following components

• API
• Controller
• Message Bus (e.g., Kafka)
• Loadbalancer
• Invoker
• Intrinsic actions (e.g., sequences, conductors)
• Data stores (e.g., CouchDB)
• Tests
• Deployment
• CLI
• General tooling
• Documentation

Types of changes

• Bug fix (generally a non-breaking change which closes an issue).
• Enhancement or new feature (adds new functionality).
• Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

• I signed an Apache CLA.
• I reviewed the style guides and followed the recommendations (Travis CI will check :).
• I added tests to cover my changes.
• My changes require further changes to the documentation.
  The website should be updated as well
• I updated the documentation where necessary.

[rabbah]

🤯 this is excellent!

[SamHjelmfelt]

For anyone wanting to test this out, here is a quick guide for enabling Docker on YARN:
https://community.hortonworks.com/articles/226331/dockerized-yarn-services-quickstart.html

It was tested with HDP 3.0.1 from Hortonworks. Available here:
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/index.html

The steps are similar with a base Apache Hadoop installation (version 3.1.1 or higher).

[rabbah]

This looks generally straightforward to follow but there are some things to consider in the use of futures with synchronized and blocking operations. I'm curious to try this and will find some time to do that.

[SamHjelmfelt]

@rabbah Thanks for the feedback. I refactored the solution to use an Actor instead of synchronized blocks.

The TravisCI build errored due to:
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

[rabbah]

Nice! Thanks for improving the PR. I will go through the changes and restart Travis for you.

[codecov-io]

Codecov Report

Merging #4129 into master will decrease coverage by 5.04%.
The diff coverage is 79.91%.

@@            Coverage Diff            @@
##           master   #4129      +/-   ##
=========================================
- Coverage   85.75%   80.7%   -5.05%     
=========================================
  Files         163     168       +5     
  Lines        7594    7838     +244     
  Branches      502     525      +23     
=========================================
- Hits         6512    6326     -186     
- Misses       1082    1512     +430

Impacted Files	Coverage Δ
.../scala/org/apache/openwhisk/core/WhiskConfig.scala	94.57% <100%> (+0.04%)	⬆️
.../org/apache/openwhisk/core/yarn/YARNRESTUtil.scala	57.4% <57.4%> (ø)
...cala/org/apache/openwhisk/core/yarn/YARNTask.scala	70% <70%> (ø)
...pache/openwhisk/core/yarn/YARNComponentActor.scala	85.29% <85.29%> (ø)
...che/openwhisk/core/yarn/YARNContainerFactory.scala	87.5% <87.5%> (ø)
...e/openwhisk/core/yarn/YARNContainerInfoActor.scala	87.75% <87.75%> (ø)
...core/database/cosmosdb/RxObservableImplicits.scala	0% <0%> (-100%)	⬇️
...core/database/cosmosdb/CosmosDBArtifactStore.scala	0% <0%> (-95.18%)	⬇️
...sk/core/database/cosmosdb/CosmosDBViewMapper.scala	0% <0%> (-92.6%)	⬇️
...whisk/core/database/cosmosdb/CosmosDBSupport.scala	0% <0%> (-84.62%)	⬇️
... and 10 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6cc6ab4...69532fb. Read the comment docs.

[SamHjelmfelt]

The test coverage is missing some of the error handling and all the Kerberos/SPNEGO code. Is it possible to test Kerberos/SPNEGO with TravisCI?

[rabbah]

I don't know @SamHjelmfelt - not something I've tried. We do have a Jenkins job we can try once it's fully operational.

[SamHjelmfelt]

Here is a YARN sandbox that will simplify testing. It is a ~5GB docker image that contains YARN pre-installed and pre-configured. Just run this container and configure the invoker for a YARN RM at localhost:8088. Just use yarnquickstart-sample-hotfix.ini instead of yarnquickstart-sample.ini.
https://community.hortonworks.com/articles/232540/docker-on-yarn-sandbox.html.

Additionally, the following project has this pull request pre-configured with a single command quickstart. The relevant YARN configurations are commented out in the docker-whisk-controller.env file.
https://github.com/SamHjelmfelt/OpenWhisk-YarnDeployment.

[rabbah]

sorry @SamHjelmfelt i'm behind on this but i'm going through and regaining velocity on the prs so i'm hoping to get to it sooner rather than later.

[SamHjelmfelt]

Rebasing

[rabbah]

Thanks for reviewing this code with me and showing me a demo. Minor nits but LGTM generally.

[SamHjelmfelt]

@rabbah I believe I addressed everything you requested, but I am happy to make further improvements.

[rabbah]

@SamHjelmfelt this is a very neat addition!