Skip to content

Remove YARN Integration #1027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 3, 2017
Merged

Remove YARN Integration #1027

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
53185c2
Removed YARN integration from master
jbaiera Jul 26, 2017
ce5ecca
Documentation updates for the removal of ESYarn
jbaiera Jul 27, 2017
e5aa979
clarify that the project was a beta.
jbaiera Jul 27, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -915,7 +915,7 @@ configure(rootProject) {
afterEvaluate {
classDirectories = files(classDirectories.files.collect {
// We're excluding Spark since the Scala bytecode trips Jacoco over and the plugin can't import yet filters
fileTree(dir: it, excludes: [ "org/elasticsearch/hadoop/yarn/**", "org/apache/hadoop/hive/**", "org/elasticsearch/spark/**", "org/elasticsearch/plugin/hadoop/**"])
fileTree(dir: it, excludes: [ "org/apache/hadoop/hive/**", "org/elasticsearch/spark/**", "org/elasticsearch/plugin/hadoop/**"])
})
}
}
Expand Down
2 changes: 0 additions & 2 deletions docs/src/reference/asciidoc/core/intro.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ While the official name of the project is {ehtm} throughout the documentation th
include::intro/typos.adoc[]

TIP: If you are looking for {es} HDFS Snapshot/Restore plugin (a separate project), please refer to its https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs[home page].

TIP: If you are looking for {es} on YARN (a separate project), please refer to its dedicated <<es-yarn,section>>.
--

[[doc-sections]]
Expand Down
18 changes: 13 additions & 5 deletions docs/src/reference/asciidoc/index.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,18 +27,26 @@

{ehtm} is an `umbrella' project consisting of three similar, yet independent sub-projects with their own, dedicated, section in the documentation:

{eh} proper:: interact with {es} from within a Hadoop environment. If you are using {mr}, Hive, Pig, {sp}, {st}, or Cascading, this project is for you. For feature requests or bugs, please open an issue in the https://github.com/elastic/elasticsearch-hadoop/issues[Elasticsearch-Hadoop repository].
{eh} proper::
Interact with {es} from within a Hadoop environment. If you are using {mr}, Hive, Pig, {sp}, {st}, or Cascading, this
project is for you. For feature requests or bugs, please open an issue in the
https://github.com/elastic/elasticsearch-hadoop/issues[Elasticsearch-Hadoop repository].

repository-hdfs:: use HDFS as a back-end repository for doing snapshot/restore from/to {es}. For more information, refer to its https://www.elastic.co/guide/en/elasticsearch/plugins/master/repository-hdfs.html[home page]. For feature requests or bugs, please open an issue in the https://github.com/elastic/elasticsearch/issues[Elasticsearch repository] with the ":Plugin Repository HDFS" tag.
repository-hdfs::
Use HDFS as a back-end repository for doing snapshot/restore from/to {es}. For more information, refer to its
https://www.elastic.co/guide/en/elasticsearch/plugins/master/repository-hdfs.html[home page]. For feature requests or
bugs, please open an issue in the https://github.com/elastic/elasticsearch/issues[Elasticsearch repository] with the
":Plugin Repository HDFS" tag.

{ey}:: run {es} on top of YARN - see <<es-yarn>>. This project is in beta.
{ey} [DISCONTINUED]::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dedemorton the es on yarn project had been deprecated and will be removed from the codebase in 6.0. discontinued sounded perfect to me, but thought you might know better. :)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@acchen97 TBH, I would expect to see a discussion of this feature's removal in the release notes rather than the main documentation. I say this because content has a tendency to live longer than we expect. If you put in the release notes, you'll know that it goes away after 6.0.

Curious if @debadair has an opinion on this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dedemorton thanks for the input. The rationale behind having it persistent here is the fact that the project has been around for a couple years at this point, so we'd like to keep this discontinuation visible likely through the 6.x releases before removing it completely in 7.0.

Does that sound reasonable?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@acchen97 I don't have a strong opinion either way since I don't own this doc. But I think your point could be made for a lot of features that we remove from the docs over time. Docs can get pretty bloated if we keep stuff like this around, and people forget to remove it. If we are good about flagging deprecation early on and reminding users that the feature is going away, statements like this are unnecessary, IMO. While useful to users in the short term, these statements represent obsolete content that we need to maintain over time. IMO, it's better to train users to look in the release notes. I've been a tech writer for 20+ years so I've seen a lot of stuff like this die on the vine over the years. Anyhow, just something to keep in mind for the future.

Run {es} on top of YARN. Due to the limitations in YARN regarding long running services, we have decided to discontinue
our development and support of the {ey} integration. If you are searching for documentation on {ey}, you can find the
documentation for the last released beta version https://www.elastic.co/guide/en/elasticsearch/hadoop/5.6/es-yarn.html[here].


Thus, while all projects fall under the Hadoop umbrella, each is covering a certain aspect of it so please be sure to read the appropriate documentation. For general questions around any of these projects, the https://discuss.elastic.co/c/elasticsearch-and-hadoop[Elastic Discuss forum] is a great place to interact with other users in the community.

include::core/index.adoc[]

include::yarn/index.adoc[]

include::appendix/index.adoc[]

47 changes: 0 additions & 47 deletions docs/src/reference/asciidoc/yarn/download.adoc
View file
Open in desktop

This file was deleted.

Binary file removed docs/src/reference/asciidoc/yarn/img/yarn-app-list.png
View file
Open in desktop
Binary file not shown.
Binary file removed docs/src/reference/asciidoc/yarn/img/yarn-container-list.png
View file
Open in desktop
Binary file not shown.
27 changes: 0 additions & 27 deletions docs/src/reference/asciidoc/yarn/index.adoc
View file
Open in desktop

This file was deleted.

69 changes: 0 additions & 69 deletions docs/src/reference/asciidoc/yarn/requirements.adoc
View file
Open in desktop

This file was deleted.

67 changes: 0 additions & 67 deletions docs/src/reference/asciidoc/yarn/setup.adoc
View file
Open in desktop

This file was deleted.

Loading