Skip to content

Commit

Permalink
[hotfix] Include Hadoop version into EnvironmentInformation
Browse files Browse the repository at this point in the history
  • Loading branch information
rmetzger committed Aug 30, 2015
1 parent a18994a commit 554b77b
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 4 deletions.
14 changes: 10 additions & 4 deletions docs/setup/yarn_setup.md
Expand Up @@ -57,7 +57,7 @@ Apache [Hadoop YARN](http://hadoop.apache.org/) is a cluster resource management

**Requirements**

- Apache Hadoop 2.2
- at least Apache Hadoop 2.2
- HDFS (Hadoop Distributed File System) (or another distributed file system supported by Hadoop)

If you have troubles using the Flink YARN client, have a look in the [FAQ section]({{ site.baseurl }}/faq.html).
Expand All @@ -79,7 +79,8 @@ tar xvzf flink-{{ site.version }}-bin-hadoop2.tgz
cd flink-{{site.version }}/
~~~

If you want to build the YARN .tgz file from sources, follow the [build instructions](building.html). You can find the result of the build in `flink-dist/target/flink-{{ site.version }}-bin/flink-{{ site.version }}/` (*Note: The version might be different for you* ).
If you want to build the YARN .tgz file from sources, follow the [build instructions](building.html).
You can find the result of the build in `flink-dist/target/flink-{{ site.version }}-bin/flink-{{ site.version }}/` (*Note: The version might be different for you* ).


#### Start a Session
Expand Down Expand Up @@ -208,9 +209,14 @@ Please note that the client then expects the `-yn` value to be set (number of Ta
./bin/flink run -m yarn-cluster -yn 2 ./examples/flink-java-examples-{{site.version }}-WordCount.jar
~~~

The command line options of the YARN session are also available with the `./bin/flink` tool. They are prefixed with a `y` or `yarn` (for the long argument options).
The command line options of the YARN session are also available with the `./bin/flink` tool.
They are prefixed with a `y` or `yarn` (for the long argument options).

Note: You can use a different configuration directory per job by setting the environment variable `FLINK_CONF_DIR`. To use this copy the `conf` directory from the Flink distribution and modify, for example, the logging settings on a per-job basis.
Note: You can use a different configuration directory per job by setting the environment variable `FLINK_CONF_DIR`.
To use this copy the `conf` directory from the Flink distribution and modify, for example, the logging settings on a per-job basis.

Note: It is possible to combine `-m yarn-cluster` with a detached YARN submission (`-yd`) to "fire and forget" a Flink job
to the YARN cluster. In this case, your application will not get any accumulator results or exceptions from the ExecutionEnvironment.execute() call!


## Recovery behavior of Flink on YARN
Expand Down
Expand Up @@ -26,6 +26,7 @@
import java.util.List;
import java.util.Properties;

import org.apache.hadoop.util.VersionInfo;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.security.UserGroupInformation;
Expand Down Expand Up @@ -280,6 +281,7 @@ public static void logEnvironmentInfo(Logger log, String componentName, String[]
log.info(" JVM: " + jvmVersion);
log.info(" Maximum heap size: " + maxHeapMegabytes + " MiBytes");
log.info(" JAVA_HOME: " + (javaHome == null ? "(not set)" : javaHome));
log.info(" Hadoop version: " + VersionInfo.getVersion());

if (options.length == 0) {
log.info(" JVM Options: (none)");
Expand Down

3 comments on commit 554b77b

@StephanEwen
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some points there were thoughts about a "hadoop-free" version. How would this play together?

@rmetzger
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, flink-runtime has a dependency on Hadoop, so I can assume its always available.
Even for a binary Flink release without build in Hadoop dependencies, we would assume Hadoop to be present (from the classpath).
For a Flink release without any Hadoop, we can either remove this again or use some reflection / fake hadoop class magic (added via maven) if needed.
But for now, I would like to have this in the code base because it helps debugging user issues.

@StephanEwen
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good addition. In the future (Hadoop not present), we may have to go for reflection, true.

Please sign in to comment.