[FLINK-1295][FLINK-883] Allow to deploy 'job only' YARN cluster. Add …

…tests to YARN

- users can now also deploy Flink on YARN for executing a single job.
- The flink-yarn project has been moved out of the flink-addons module
- the MiniYARNCluster is used for testing Flink on YARN
- There is now a (undocumented) Java interface Flink's YARN client, allowing users to manually control the Yarn session.
- ALL ports used by Flink when running on YARN are automatically determined. In the past users reported problems with blocked ports (YARN is telling the client the RPC address of the application master)
- The checks before deployment have been improved to give better error messages if the user is requesting too many resources for a YARN session

docs/cli.md

@@ -33,7 +33,8 @@ started from the same installation directory.
 
 A prerequisite to using the command line interface is that the Flink
 master (JobManager) has been started (via `<flink-home>/bin/start-
-local.sh` or `<flink-home>/bin/start-cluster.sh`).
+local.sh` or `<flink-home>/bin/start-cluster.sh`) or that a YARN
+environment is available.
 
 The command line can be used to
 
@@ -64,6 +65,11 @@ The command line can be used to
                                ./examples/flink-java-examples-{{ site.FLINK_VERSION_SHORT }}-WordCount.jar \
                                -file:///home/user/hamlet.txt file:///home/user/wordcount_out
 
+-   Run example program using a [per-job YARN cluster](yarn_setup.html#run-a-single-flink-job-on-hadoop-yarn) with 2 TaskManagers:
+
+        ./bin/flink run -m yarn-cluster -yn 2 \
+                               ./examples/flink-java-examples-{{ site.FLINK_VERSION_STABLE }}-WordCount.jar \
+                               -file:///home/user/hamlet.txt file:///home/user/wordcount_out
 
 -   Display the expected arguments for the WordCount example program:
 

docs/config.md

@@ -304,8 +304,7 @@ to avoid duplicate port allocations when running multiple YARN sessions in paral
 
 So if `yarn.am.rpc.port` is configured to `10245` and the session's application ID is `application_1406629969999_0002`, then the actual port being used is 10245 + 2 = 10247
 
-- `yarn.am.rpc.port`: The port that is being opened by the Application Master (AM) to 
-let the YARN client connect for an RPC serice. (DEFAULT: Port 10245)
+- `yarn.heap-cutoff-ratio`: Percentage of heap space to remove from containers started by YARN.
 
 
 ## Background

docs/yarn_setup.md

@@ -36,14 +36,14 @@ cd flink-yarn-{{ site.FLINK_VERSION_SHORT }}/
 
 Specify the `-s` flag for the number of processing slots per Task Manager. We recommend to set the number of slots to the number of processors per machine.
 
-## Introducing YARN
+## Apache Flink on Hadoop YARN using a YARN Session
 
 Apache [Hadoop YARN](http://hadoop.apache.org/) is a cluster resource management framework. It allows to run various distributed applications on top of a cluster. Flink runs on YARN next to other applications. Users do not have to setup or install anything if there is already a YARN setup.
 
 **Requirements**
 
 - Apache Hadoop 2.2
-- HDFS (Hadoop Distributed File System)
+- HDFS (Hadoop Distributed File System) (or another distributed file system supported by Hadoop)
 
 If you have troubles using the Flink YARN client, have a look in the [FAQ section](faq.html).
 
@@ -80,16 +80,14 @@ This command will show you the following overview:
 ~~~bash
 Usage:
    Required
-     -n,--container <arg>   Number of Yarn container to allocate (=Number of Task Managers)
+     -n,--container <arg>   Number of YARN container to allocate (=Number of Task Managers)
    Optional
-     -D <arg>                       Dynamic Properties
+     -D <arg>                        Dynamic properties
      -jm,--jobManagerMemory <arg>    Memory for JobManager Container [in MB]
      -q,--query                      Display available YARN resources (memory, cores)
      -qu,--queue <arg>               Specify YARN queue.
      -s,--slots <arg>                Number of slots per TaskManager
      -tm,--taskManagerMemory <arg>   Memory per TaskManager Container [in MB]
-     -tmc,--taskManagerCores <arg>   Virtual CPU cores per TaskManager
-     -v,--verbose                    Verbose debug mode
 ~~~
 
 Please note that the Client requires the `HADOOP_HOME` (or `YARN_CONF_DIR` or `HADOOP_CONF_DIR`) environment variable to be set to read the YARN and HDFS configuration.
@@ -118,7 +116,7 @@ The client has to remain open to keep the deployment running. We suggest to use
 4. Use `screen -r` to resume again.
 
 
-## Submit Job to Flink
+### Submit Job to Flink
 
 Use the following command to submit a Flink program to the YARN cluster:
 
@@ -173,6 +171,24 @@ You can check the number of TaskManagers in the JobManager web interface. The ad
 If the TaskManagers do not show up after a minute, you should investigate the issue using the log files.
 
 
+## Run a single Flink job on Hadoop YARN
+
+The documentation above describes how to start a Flink cluster within a Hadoop YARN environment.
+It is also possible to launch Flink within YARN only for executing a single job.
+
+To deploy a job to a per-job YARN cluster, set the master name to `yarn-cluster`.
+Please note that the client then expects the `-n` value to be set (number of TaskManagers).
+
+***Example:***
+
+~~~bash
+./bin/flink run -m yarn-cluster -yn 2 ./examples/flink-java-examples-{{site.FLINK_VERSION_STABLE }}-WordCount.jar 
+~~~
+
+The command line options of the YARN session are also available with the `./bin/flink` tool. They are prefixed with a `y` or `yarn` (for the long argument options).
+
+
+
 ## Debugging a failed YARN session
 
 There are many reasons why a Flink YARN session deployment can fail. A misconfigured Hadoop setup (HDFS permissions, YARN configuration), version incompatibilities (running Flink with vanilla Hadoop dependencies on Cloudera Hadoop) or other errors.
@@ -203,7 +219,7 @@ It allows to access log files for running YARN applications and shows diagnostic
 Users using Hadoop distributions from companies like Hortonworks, Cloudera or MapR might have to build Flink against their specific versions of Hadoop (HDFS) and YARN. Please read the [build instructions](building.html) for more details.
 
 
-## Background
+## Background / Internals
 
 This section briefly describes how Flink and YARN interact.