[BEAM-2839, BEAM-2838] Add MapReduce runner to Beam asf-site. #313

peihe · 2017-09-07T07:50:31Z

No description provided.

asfgit · 2017-09-07T08:01:07Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/670/

Jenkins built the site at commit id 7b6a866 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

asfgit · 2017-09-07T08:19:51Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/671/

Jenkins built the site at commit id 6a3304c with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

asfgit · 2017-09-07T08:43:52Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/672/

Jenkins built the site at commit id 7b6a866 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

asfgit · 2017-09-08T07:30:40Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/681/

Jenkins built the site at commit id 04a5d2d with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

asfgit · 2017-09-11T09:58:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/689/

Jenkins built the site at commit id 444c04d with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

peihe · 2017-09-11T11:05:43Z

R: @kennknowles

asfgit · 2017-09-11T11:15:51Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/690/

Jenkins built the site at commit id 5fead25 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

kennknowles · 2017-09-11T16:13:51Z

R: @melap

On the overview page, we just list the runners that are on master.

kennknowles · 2017-09-11T16:15:06Z

src/_data/capability-matrix.yml

@@ -11,6 +11,8 @@ columns:
    name: Apache Apex
  - class: gearpump
    name: Apache Gearpump
+  - class: mapreduce
+    name: MapReduce


Apache Hadoop MapReduce

melap

made some minor edit suggestions below.

one general comment along the same lines as @kennknowles -- are we adding runners not in master to the main menu pulldown? I'm not sure. I ask as header.html will need a line item for this in the runners section, otherwise I don't think people can get to it. Or alternately, an additional column in the table on https://beam.apache.org/contribute/work-in-progress/ for links to the feature branch runner's pages?

melap · 2017-09-11T23:56:44Z

src/documentation/runners/mapreduce.md

+
+The Apache Hadoop MapReduce runner currently supports Apache Hadoop 2.8.1 version.
+
+You can add a dependency on the latest version of the Apache Hadoop MapReduce runner by adding to your pom.xml the following:


by adding the following to your pom.xml:

melap · 2017-09-11T23:59:17Z

src/documentation/runners/mapreduce.md

+## Apache Hadoop MapReduce Runner prerequisites and setup
+You need to have an Apache Hadoop environment with either [Single Node Setup](https://hadoop.apache.org/docs/r1.2.1/single_node_setup.html) or [Cluster Setup](https://hadoop.apache.org/docs/r1.2.1/cluster_setup.html)
+
+The Apache Hadoop MapReduce runner currently supports Apache Hadoop 2.8.1 version.


Apache Hadoop version 2.8.1.

melap · 2017-09-12T00:00:11Z

src/documentation/runners/mapreduce.md

+The Apache Hadoop MapReduce runner currently supports Apache Hadoop 2.8.1 version.
+
+You can add a dependency on the latest version of the Apache Hadoop MapReduce runner by adding to your pom.xml the following:
+```java


I'd recommend making this a regular block with just ``` ... sometimes the language toggling can be strange and it might be blank if the user chose python SDK on another page.

melap · 2017-09-12T00:00:35Z

src/documentation/runners/mapreduce.md

+```
+
+## Deploying Apache Hadoop MapReduce with your application
+To execute in a local hadoop environment, use this command:


capitalize Hadoop (multiple places on the page)

melap · 2017-09-12T00:01:09Z

src/documentation/runners/mapreduce.md

+      --fileOutputDir=<directory for intermediate outputs>"
+```
+
+To execute in a hadoop cluster, you need to package your program along will all dependencies in a so-called fat jar.


can remove "you need to"
will -> with
remove "so-called"

melap · 2017-09-12T00:05:04Z

src/documentation/runners/mapreduce.md

+<tr>
+  <td><code>runner</code></td>
+  <td>The pipeline runner to use. This option allows you to determine the pipeline runner at runtime.</td>
+  <td>Set to <code>MapReduceRunner</code> to run using the Apache Hadoop MapReduce.</td>


remove "the" after using

melap · 2017-09-12T00:05:25Z

src/documentation/runners/mapreduce.md

+</tr>
+<tr>
+  <td><code>fileOutputDir</code></td>
+  <td>The directory for files output.</td>


files output -> output files

melap · 2017-09-12T00:09:42Z

src/documentation/runners/mapreduce.md

+
+To execute in a hadoop cluster, you need to package your program along will all dependencies in a so-called fat jar.
+
+If you follow along the [Beam Quickstart]({{ site.baseurl }}/get-started/quickstart/) this is the command that you can run:


this quickstart link is a redirect to the Java page.. perhaps link directly to the Java version and text something like this?

If you are using through the [Beam Java SDK Quickstart]({{ site.baseurl }}/get-started/quickstart-java/), you can run this command:

Done with a minor change to "following through"

melap · 2017-09-12T00:20:04Z

src/get-started/beam-overview.md

@@ -36,6 +36,8 @@ Beam currently supports Runners that work with the following distributed process
         alt="Apache Flink">
 * Apache Gearpump (incubating) <img src="{{ site.baseurl }}/images/logos/runners/gearpump.png"
         alt="Apache Gearpump">
+* Apache Hadoop MapReduce <img src="{{ site.baseurl }}/images/logos/runners/mapreduce.png"


this image is very big compared to the others on the page... may need to specify dimensions or use a smaller image. more a comment for later, as from Kenn's comment, it sounds like this runner shoudn't be listed here yet.

peihe

PTAL

I have to update header.html in a follow-up PR. Otherwise, it will complain about dead link.

peihe · 2017-09-12T02:18:33Z

src/_data/capability-matrix.yml

@@ -11,6 +11,8 @@ columns:
    name: Apache Apex
  - class: gearpump
    name: Apache Gearpump
+  - class: mapreduce
+    name: MapReduce


peihe · 2017-09-12T02:19:07Z

src/documentation/runners/mapreduce.md

+## Apache Hadoop MapReduce Runner prerequisites and setup
+You need to have an Apache Hadoop environment with either [Single Node Setup](https://hadoop.apache.org/docs/r1.2.1/single_node_setup.html) or [Cluster Setup](https://hadoop.apache.org/docs/r1.2.1/cluster_setup.html)
+
+The Apache Hadoop MapReduce runner currently supports Apache Hadoop 2.8.1 version.


peihe · 2017-09-12T02:20:10Z

src/documentation/runners/mapreduce.md

+
+The Apache Hadoop MapReduce runner currently supports Apache Hadoop 2.8.1 version.
+
+You can add a dependency on the latest version of the Apache Hadoop MapReduce runner by adding to your pom.xml the following:


peihe · 2017-09-12T02:20:16Z

src/documentation/runners/mapreduce.md

+The Apache Hadoop MapReduce runner currently supports Apache Hadoop 2.8.1 version.
+
+You can add a dependency on the latest version of the Apache Hadoop MapReduce runner by adding to your pom.xml the following:
+```java


peihe · 2017-09-12T02:20:37Z

src/documentation/runners/mapreduce.md

+```
+
+## Deploying Apache Hadoop MapReduce with your application
+To execute in a local hadoop environment, use this command:


peihe · 2017-09-12T02:21:32Z

src/documentation/runners/mapreduce.md

+      --fileOutputDir=<directory for intermediate outputs>"
+```
+
+To execute in a hadoop cluster, you need to package your program along will all dependencies in a so-called fat jar.


peihe · 2017-09-12T02:22:57Z

src/documentation/runners/mapreduce.md

+<tr>
+  <td><code>runner</code></td>
+  <td>The pipeline runner to use. This option allows you to determine the pipeline runner at runtime.</td>
+  <td>Set to <code>MapReduceRunner</code> to run using the Apache Hadoop MapReduce.</td>


peihe · 2017-09-12T02:24:16Z

src/documentation/runners/mapreduce.md

+
+To execute in a hadoop cluster, you need to package your program along will all dependencies in a so-called fat jar.
+
+If you follow along the [Beam Quickstart]({{ site.baseurl }}/get-started/quickstart/) this is the command that you can run:


Done with a minor change to "following through"

peihe · 2017-09-12T02:24:38Z

src/documentation/runners/mapreduce.md

+</tr>
+<tr>
+  <td><code>fileOutputDir</code></td>
+  <td>The directory for files output.</td>


peihe · 2017-09-12T02:25:15Z

src/get-started/beam-overview.md

@@ -36,6 +36,8 @@ Beam currently supports Runners that work with the following distributed process
         alt="Apache Flink">
 * Apache Gearpump (incubating) <img src="{{ site.baseurl }}/images/logos/runners/gearpump.png"
         alt="Apache Gearpump">
+* Apache Hadoop MapReduce <img src="{{ site.baseurl }}/images/logos/runners/mapreduce.png"


asfgit · 2017-09-12T02:39:01Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/693/

Jenkins built the site at commit id f95c84b with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

kennknowles · 2017-09-12T02:48:27Z

src/_data/capability-matrix.yml

+          - class: mapreduce
+            l1: 'Yes'
+            l2: fully supported
+            l3: ''


Normally, I'd want one sentence about the MapReduce feature used to implement them, but I think with the upcoming revamp it is not worth filling it out. We should have this page revised before the runner is on master.

kennknowles · 2017-09-12T02:49:57Z

src/_data/capability-matrix.yml

@@ -505,6 +575,10 @@ categories:
            l1: 'No'
            l2: ''
            l3: ''
+          - class: mapreduce
+            l1: 'Yes'


I think the most useful thing for users here is to say "No" and indicate that it is a batch-only runner.

kennknowles · 2017-09-12T02:50:45Z

src/_data/capability-matrix.yml

@@ -532,6 +606,10 @@ categories:
            l1: 'Yes'
            l2: fully supported
            l3: ''
+          - class: mapreduce
+            l1: 'Yes'


Once you say 'No' to triggering, you can leave these blank or just say 'No' without the explanation. It is a nice explanation, though, so you could also leave it.

kennknowles · 2017-09-12T02:51:38Z

src/_data/capability-matrix.yml

@@ -668,6 +762,10 @@ categories:
            l1: 'Yes'
            l2: fully supported
            l3: ''
+          - class: mapreduce
+            l1: 'Yes'


This stuff is another area where it doesn't really make sense for a batch-only runner to have any answer.

kennknowles · 2017-09-12T02:52:03Z

src/documentation/runners/mapreduce.md

+layout: default
+title: "Apache Hadoop MapReduce Runner"
+permalink: /documentation/runners/mapreduce/
+redirect_from: /learn/runners/mapreduce/


Shouldn't need this, since there's no past URL.

kennknowles · 2017-09-12T02:52:55Z

src/documentation/runners/mapreduce.md

+---
+layout: default
+title: "Apache Hadoop MapReduce Runner"
+permalink: /documentation/runners/mapreduce/


I think since we will not link from the overview and will not link from the header, you should follow this up by adding a link from the work-in-progress page.

peihe

PTAL

peihe · 2017-09-13T02:20:17Z

src/_data/capability-matrix.yml

+          - class: mapreduce
+            l1: 'Yes'
+            l2: fully supported
+            l3: ''


peihe · 2017-09-13T02:21:58Z

src/_data/capability-matrix.yml

@@ -505,6 +575,10 @@ categories:
            l1: 'No'
            l2: ''
            l3: ''
+          - class: mapreduce
+            l1: 'Yes'


peihe · 2017-09-13T02:24:35Z

src/_data/capability-matrix.yml

@@ -668,6 +762,10 @@ categories:
            l1: 'Yes'
            l2: fully supported
            l3: ''
+          - class: mapreduce
+            l1: 'Yes'


peihe · 2017-09-13T02:24:42Z

src/_data/capability-matrix.yml

@@ -532,6 +606,10 @@ categories:
            l1: 'Yes'
            l2: fully supported
            l3: ''
+          - class: mapreduce
+            l1: 'Yes'


peihe · 2017-09-13T02:26:18Z

src/documentation/runners/mapreduce.md

+---
+layout: default
+title: "Apache Hadoop MapReduce Runner"
+permalink: /documentation/runners/mapreduce/


peihe · 2017-09-13T02:26:23Z

src/documentation/runners/mapreduce.md

+layout: default
+title: "Apache Hadoop MapReduce Runner"
+permalink: /documentation/runners/mapreduce/
+redirect_from: /learn/runners/mapreduce/


asfgit · 2017-09-13T02:38:44Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/695/

Jenkins built the site at commit id f2ae6e6 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

kennknowles · 2017-09-13T02:46:52Z

Looks mostly good. When I expand details on the capability matrix, the formatting is a bit weird. There are some boxes that start with a :

peihe · 2017-09-13T06:05:27Z

PTAL

Added "No" back, and it should fix the ':' issue

asfgit · 2017-09-13T06:15:01Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/696/

Jenkins built the site at commit id fa0c7a1 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

melap · 2017-09-14T20:25:37Z

LGTM from a text perspective

melap · 2017-09-14T20:39:26Z

Actually, one thing -- does the capability matrix need to note somewhere that this runner is in development and isn't part of the release yet?

peihe · 2017-09-18T02:54:15Z

We had gearpump in the capability matrix before merging it to master.

PTAL @kennknowles

kennknowles · 2017-09-19T03:42:44Z

It would be nice to also call it out on the capability matrix page, but I don't think that is so critical. Mostly the header and overview are where we should stick to released runners. We need a major revamp there anyhow, and that will give a good opportunity to add verbiage about in-progress runners. Let's get this added to the site so that anyone dropping in might see it.

kennknowles · 2017-09-19T03:42:49Z

@asfgit merge

peihe force-pushed the asf-site-jstorm branch from 6a3304c to 7b6a866 Compare September 7, 2017 08:32

peihe changed the title ~~[BEAM-2666] Add JStorm runner to Beam asf-site.~~ [Ignore] Add runner to Beam asf-site. Sep 8, 2017

peihe force-pushed the asf-site-jstorm branch from 7b6a866 to 04a5d2d Compare September 8, 2017 07:18

peihe changed the title ~~[Ignore] Add runner to Beam asf-site.~~ WIP [BEAM-2839, BEAM-2838] Add MapReduce runner to Beam asf-site. Sep 8, 2017

peihe force-pushed the asf-site-jstorm branch from 04a5d2d to 444c04d Compare September 11, 2017 09:48

[BEAM-2839, BEAM-2838] Add MapReduce runner to Beam asf-site.

5fead25

peihe force-pushed the asf-site-jstorm branch from 444c04d to 5fead25 Compare September 11, 2017 11:02

peihe changed the title ~~WIP [BEAM-2839, BEAM-2838] Add MapReduce runner to Beam asf-site.~~ [BEAM-2839, BEAM-2838] Add MapReduce runner to Beam asf-site. Sep 11, 2017

kennknowles reviewed Sep 11, 2017

View reviewed changes

melap reviewed Sep 12, 2017

View reviewed changes

fixup: address comments.

f95c84b

peihe commented Sep 12, 2017

View reviewed changes

kennknowles requested changes Sep 12, 2017

View reviewed changes

fixup: address comments.

f2ae6e6

peihe commented Sep 13, 2017

View reviewed changes

fixup: address comments.

fa0c7a1

kennknowles approved these changes Sep 19, 2017

View reviewed changes

asfgit closed this in 9bc6678 Sep 19, 2017

melap mentioned this pull request Sep 29, 2017

Manually add missed files to content dir #329

Closed

robertwb pushed a commit to robertwb/incubator-beam that referenced this pull request Jun 5, 2018

This closes apache/beam-site#313

1e562ee

robertwb pushed a commit to robertwb/incubator-beam that referenced this pull request Jun 5, 2018

This closes apache/beam-site#313

2dcddba

melap pushed a commit to apache/beam that referenced this pull request Jun 20, 2018

This closes apache/beam-site#313

bb3edb8


		The Apache Hadoop MapReduce runner currently supports Apache Hadoop 2.8.1 version.

		You can add a dependency on the latest version of the Apache Hadoop MapReduce runner by adding to your pom.xml the following:


		To execute in a hadoop cluster, you need to package your program along will all dependencies in a so-called fat jar.

		If you follow along the [Beam Quickstart]({{ site.baseurl }}/get-started/quickstart/) this is the command that you can run:

[BEAM-2839, BEAM-2838] Add MapReduce runner to Beam asf-site. #313

[BEAM-2839, BEAM-2838] Add MapReduce runner to Beam asf-site. #313

Conversation

peihe commented Sep 7, 2017

asfgit commented Sep 7, 2017

asfgit commented Sep 7, 2017

asfgit commented Sep 7, 2017

asfgit commented Sep 8, 2017

asfgit commented Sep 11, 2017

peihe commented Sep 11, 2017

asfgit commented Sep 11, 2017

kennknowles commented Sep 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

melap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peihe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfgit commented Sep 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peihe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfgit commented Sep 13, 2017

kennknowles commented Sep 13, 2017

peihe commented Sep 13, 2017

asfgit commented Sep 13, 2017

melap commented Sep 14, 2017

melap commented Sep 14, 2017

peihe commented Sep 18, 2017

kennknowles commented Sep 19, 2017

kennknowles commented Sep 19, 2017