[BEAM-825] Fill in the documentation/runners/apex portion of the website #98

sandeepdeshmukh · 2016-12-01T14:03:50Z

@tweise : Could you please review.

asfbot · 2016-12-01T14:05:19Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/126/
--none--

asfbot · 2016-12-01T14:07:46Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/169/

Jenkins built the site at commit id d8ecc4e with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

asfbot · 2016-12-01T14:42:31Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/127/
--none--

asfbot · 2016-12-01T14:43:16Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/170/
--none--

asfbot · 2016-12-01T14:48:20Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/128/
--none--

asfbot · 2016-12-01T14:51:09Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/171/

Jenkins built the site at commit id e47749d with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

sandeepdeshmukh · 2016-12-01T15:15:04Z

Failure on external links but all the links are valid and working fine. Not sure of the root cause.

davorbonaci

Thanks @sandeepdeshmukh, this is a great start!

Left a few minor comments, but the high level point of being somewhat vendor-specific should be discussed. What do you think?

CC: @francesperry
CC: @jbonofre

davorbonaci · 2016-12-02T02:44:43Z

src/documentation/runners/apex.md

@@ -2,8 +2,113 @@
 layout: default
 title: "Apache Apex Runner"
 permalink: /documentation/runners/apex/
+redirect_from: /learn/runners/apex/


Not necessary, I'd remove.

davorbonaci · 2016-12-02T02:47:25Z

src/documentation/runners/apex.md

 ---
 # Using the Apache Apex Runner

-This page is under construction ([BEAM-825](https://issues.apache.org/jira/browse/BEAM-825)).
+Apex‐Runner is a Runner for Apache Beam which executes Beam pipelines with Apache Apex as underlying engine. The runner has broad support for the Beam model and supports streaming and batch pipelines.


The Apex Runner executes Apache Beam pipelines using [Apache Apex](link) as an underlying engine.

davorbonaci · 2016-12-02T02:48:08Z

src/documentation/runners/apex.md

 ---
 # Using the Apache Apex Runner

-This page is under construction ([BEAM-825](https://issues.apache.org/jira/browse/BEAM-825)).
+Apex‐Runner is a Runner for Apache Beam which executes Beam pipelines with Apache Apex as underlying engine. The runner has broad support for the Beam model and supports streaming and batch pipelines.


The second sentence could link to the capability matrix.

davorbonaci · 2016-12-02T02:49:24Z

src/documentation/runners/apex.md


+[Apache Apex](http://apex.apache.org/) is a stream processing platform and framework for low-latency, high-throughput and fault-tolerant analytics applications on Apache Hadoop. Apex is Java based and also provides its own API for application development (native compositional and declarative Java API, SQL) with a comprehensive [operator library](https://github.com/apache/apex-malhar). Apex has a unified streaming architecture and can be used for real-time and batch processing. With its stateful stream processing architecture Apex can support all of the concepts in the Beam model (event time, triggers, watermarks etc.).


I'd remove "Apex is Java based and also provides its own API for application development (native compositional and declarative Java API, SQL) with a comprehensive operator library." -- not applicable in this context.

davorbonaci · 2016-12-02T02:50:48Z

src/documentation/runners/apex.md

+
+
+
+## Apex-Runner prerequisites and setup


Would it be possible to make this vendor neutral?

Not the UI portion. It relies on the DataTorrent community edition.

We need a hadoop cluster with YARN setup. So, trying make it easy for people to use it. Hence thought of using ready made Sandbox.

To monitor details of the application via UI, user can use community edition.

Do share your thoughts on how to proceed.

Acknowledged. Let me think about it and try to figure out whether there are any policies or precedents here.

Also, CC: @francesperry and @jbonofre.

asfbot · 2016-12-02T13:00:42Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/131/
--none--

asfbot · 2016-12-02T13:03:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/174/

Jenkins built the site at commit id ea2ba63 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

tweise · 2016-12-06T19:00:53Z

@davorbonaci the YARN cluster is the users choice. Apex runs on all distros. The DataTorrent sandbox is just one option. YARN via dataproc could be another. The UI can be installed on either. @kennknowles may have further thoughts on this.

davorbonaci · 2016-12-06T19:57:43Z

After some thinking, I think Beam shouldn't be giving a preference to any Apex distributions or services. A Dataproc runner could do it, a DataTorrent runner could do it as well, but a generic Apache Apex runner should not. There's another level of indirection here and, when referring to the Apex runner, we should be Apex vendor-independent. Similarly, Flink runner should be referring to Apache Flink only.

davorbonaci · 2016-12-06T20:00:23Z

src/documentation/runners/apex.md

+
+## Apex-Runner prerequisites and setup
+
+To use Apache Apex Runner for Beam, you need to have a Hadoop cluster with YARN setup. You can setup your own Hadoop cluster or simply download DataTorrent Sandbox which comes with Hadoop and Apex pre-installed.


You may set up your own Hadoop cluster, or choose any vendor-specific distribution that includes Hadoop and Apex pre-installed. Please see the [distribution information on the Apache Apex website](some link).

+1. This new wording gives users what they need to make an informed decision. Maybe a bit less telegraphic. I think it is OK to give generic encouragement to first try it with a distro and only later go a DIY route.

You can set up your own Hadoop+YARN cluster or use an existing cluster if you have one available. Otherwise, to get started quickly and try out Beam it may be easiest to use a vendor distribution that includes Hadoop and Apex pre-installed. Please see the [distribution information on the Apache Apex website](some link) to learn of options.

davorbonaci · 2016-12-06T20:01:42Z

src/documentation/runners/apex.md

+hadoop/sbin/stop-yarn.sh && hadoop/sbin/start-yarn.sh
+```
+
+#### 2. Download code and compile


we should use releases, not running from head

davorbonaci · 2016-12-06T20:02:43Z

src/documentation/runners/apex.md

+
+## Montoring progress of your job
+
+If you are using DataTorrent RTS Sandbox then you can monitor progress of the application on http://localhost:9090/ using your browser. The console provides you updates on the application progress, general statistics like number of operators, number of containers - planned and allcated, allocated memory to the application etc.


Depending on your installation, you may be able to monitor the progress of your job on the Hadoop cluster. Please see the relevant information on the Apache Apex website. (add specific links)

tweise · 2016-12-16T01:29:20Z

@sandeepdeshmukh any update on incorporating the feedback?

sandeepdeshmukh · 2016-12-16T14:01:20Z

I am working on it. Trying to run the app on YARN and checking things around.
Would update the PR next week.

asfbot · 2016-12-27T08:17:35Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/147/
--none--

asfbot · 2016-12-27T08:19:46Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/190/

Jenkins built the site at commit id d80cd54 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

sandeepdeshmukh · 2016-12-27T08:32:16Z

Updated the PR with review comments and is currently based on Apex 3.5.0 and Beam 0.4-Incubating releases.

davorbonaci

Thanks @sandeepdeshmukh; left a few comments.

davorbonaci · 2016-12-29T23:34:09Z

src/documentation/runners/apex.md

+
+Wordcount example to run on Apex-Runner
+```
+git clone https://github.com/tweise/apex-samples.git


Can we use Beam examples instead, similarly to other runners?

Apex runner is does not work as-is on the Beam examples and hence we would be maintaining this example till Apex runner works out of the box for Beam examples.

We should mention why this is currently needed here as well as in the README in https://github.com/tweise/apex-samples/tree/master/beam-apex-wordcount

Reasons include the ability to specify the resource constraints (BEAM-980) etc.

davorbonaci · 2016-12-29T23:34:40Z

src/documentation/runners/apex.md

+
+Download some data for processing and put it on HDFS
+```
+curl http://www.gutenberg.org/cache/epub/1128/pg1128.txt > /tmp/kinglear.txt


wouldn't this be automatically handled by Beam's WordCount?

The default is to use gs:// but we are using HDFS based input. So added steps explicitly for this.

davorbonaci · 2016-12-29T23:35:22Z

src/documentation/runners/apex.md

+Depending on your installation, you may be able to monitor the progress of your job on the Hadoop cluster. Alternately, you have folloing optoins: 
+
+* YARN : Using YARN web UI generally running on 8088 on the node running resource manager
+* Apex cli: [Using apex cli to get running application information](http://apex.apache.org/docs/apex/apex_cli/#apex-cli-commands)


console.jpg is not currently used? drop?

asfbot · 2017-01-03T14:04:36Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/171/
--none--

asfbot · 2017-01-03T14:07:47Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/215/

Jenkins built the site at commit id 85fbcae with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

asfbot · 2017-01-13T11:58:52Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/186/
--none--

asfbot · 2017-01-13T11:59:44Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/230/
--none--

asfbot · 2017-02-01T04:28:38Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/250/

Jenkins built the site at commit id a6b435e with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

sandeepdeshmukh · 2017-02-01T06:26:22Z

@tweise : I have updated the PR with your suggestions.
@davorbonaci : Plz review.

dhalperi · 2017-02-21T16:30:08Z

R: @tweise @davorbonaci

tweise · 2017-02-22T05:42:49Z

src/documentation/runners/apex.md

@@ -5,5 +5,72 @@ permalink: /documentation/runners/apex/
 ---
 # Using the Apache Apex Runner

-This page is under construction ([BEAM-825](https://issues.apache.org/jira/browse/BEAM-825)).
+The Apex Runner executes Apache Beam pipelines using [Apache Apex](http://apex.apache.org/) as an underlying engine. The runner has broad support for the [Beam model and supports streaming and batch pipelines]({{ site.baseurl }}/documentation/runners/capability-matrix/).


The runner does not have good support for batch pipelines currently, please remove this.

tweise · 2017-02-22T05:43:29Z

src/documentation/runners/apex.md

+
+[Apache Apex](http://apex.apache.org/) is a stream processing platform and framework for low-latency, high-throughput and fault-tolerant analytics applications on Apache Hadoop. Apex has a unified streaming architecture and can be used for real-time and batch processing. With its stateful stream processing architecture, Apex can support all of the concepts in the Beam model (event time, triggers, watermarks etc.).
+
+ApexRunner does not implement launch on YARN yet, hence we move the Beam pipeline code into Apex StreamingApplication to translate it into the Apex DAG and then launch the application using apex CLI. Later, instead of implementing Apex StreamingApplication, the main method will call the runner. The current process is an intermedia step as of 0.4.0.


There is support for launch on YARN now.

tweise · 2017-02-22T05:46:50Z

With https://issues.apache.org/jira/browse/BEAM-980 in master it should be possible to come up with a configuration that will allow to run the existing example as is and without the Apex packaging workaround?

sandeepdeshmukh · 2017-02-23T06:29:26Z

I would take care of these comments in a day or two.
Thanks.

davorbonaci · 2017-02-28T00:31:51Z

I still have a problem with usage of the private repositories in our instructions, outside Apache's control -- we should fix this. Otherwise, no worries here.

sandeepdeshmukh · 2017-02-28T04:36:45Z

@tweise : How do you recommend to address @davorbonaci 's comment.

tweise · 2017-02-28T04:39:39Z

See earlier comment, there should be no need for this workaround, example should work as is with YARN launcher in master.

asfbot · 2017-03-03T13:39:12Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/279/
--none--

asfbot · 2017-03-03T13:41:09Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/321/

Jenkins built the site at commit id 468d2fd with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

davorbonaci

Thanks @sandeepdeshmukh. I like the content. This is nice.

I left just a few grammar/style/typo comments that I think we can resolve quickly and get this done! Once again, thanks.

davorbonaci · 2017-03-05T06:28:00Z

src/documentation/runners/apex.md

+
+[Apache Apex](http://apex.apache.org/) is a stream processing platform and framework for low-latency, high-throughput and fault-tolerant analytics applications on Apache Hadoop. Apex has a unified streaming architecture and can be used for real-time and batch processing. With its stateful stream processing architecture, Apex can support all of the concepts in the Beam model (event time, triggers, watermarks etc.).
+
+## Apex-Runner prerequisites and setup


Please remove dash between Apex and Runner.

davorbonaci · 2017-03-05T06:28:38Z

src/documentation/runners/apex.md

@@ -5,5 +5,46 @@ permalink: /documentation/runners/apex/
 ---
 # Using the Apache Apex Runner

-This page is under construction ([BEAM-825](https://issues.apache.org/jira/browse/BEAM-825)).
+The Apex Runner executes Apache Beam pipelines using [Apache Apex](http://apex.apache.org/) as an underlying engine. The runner has broad support for the [Beam model and supports streaming and batch pipelines]({{ site.baseurl }}/documentation/runners/capability-matrix/).


Grammar: a broad support

davorbonaci · 2017-03-05T06:29:04Z

src/documentation/runners/apex.md

+
+## Apex-Runner prerequisites and setup
+
+You may set up your own Hadoop cluster,  and [setup Apache Apex on top of it](http://apex.apache.org/docs/apex/apex_development_setup/) or choose any vendor-specific distribution that includes Hadoop and Apex pre-installed. Please see the [distribution information on the Apache Apex website](http://apex.apache.org/downloads.html).


Double space before and.

davorbonaci · 2017-03-05T06:29:22Z

src/documentation/runners/apex.md

+
+You may set up your own Hadoop cluster,  and [setup Apache Apex on top of it](http://apex.apache.org/docs/apex/apex_development_setup/) or choose any vendor-specific distribution that includes Hadoop and Apex pre-installed. Please see the [distribution information on the Apache Apex website](http://apex.apache.org/downloads.html).
+
+## Running wordcount using Apex-Runner


wordcount --> the WordCount example
Apex-Runner --> Apex Runner

davorbonaci · 2017-03-05T06:30:14Z

src/documentation/runners/apex.md

+
+## Running wordcount using Apex-Runner
+
+Download some data for processing and put it on HDFS


Upload input data to a location in HDFS

Also, end with a period ('.').

davorbonaci · 2017-03-05T06:31:35Z

src/documentation/runners/apex.md

+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="--inputFile=/tmp/input/ --output=/tmp/output/ --runner=ApexRunner --embeddedExecution=false --configFile=beam-runners-apex.properties" -Papex-runner
+```
+
+This will launch an Apex application.


move this comment into above, e.g.:

Run the WordCount example as an Apex application.

davorbonaci · 2017-03-05T06:32:50Z

src/documentation/runners/apex.md

+
+## Checking output
+
+The sample program which is processing small amount of data would finish quickly. You can check contents on /tmp/output/ on HDFS


The first sentence should be obvious. I'd just remove it.

Second sentence: end with a period ('.').

Also, no need for the /tmp/output in the prose -- it is obvious from the command below. You can just say: "Check the output of the pipeline from an HDFS location."

davorbonaci · 2017-03-05T06:34:20Z

src/documentation/runners/apex.md

+
+## Montoring progress of your job
+
+Depending on your installation, you may be able to monitor the progress of your job on the Hadoop cluster. Alternately, you have folloing optoins:


Typo: alternatively
Typo: following
Typo: options

davorbonaci · 2017-03-05T06:34:38Z

src/documentation/runners/apex.md

+
+Depending on your installation, you may be able to monitor the progress of your job on the Hadoop cluster. Alternately, you have folloing optoins:
+
+* YARN : Using YARN web UI generally running on 8088 on the node running resource manager


End both sentences with a period.

davorbonaci · 2017-03-05T06:35:18Z

src/documentation/runners/apex.md

+Depending on your installation, you may be able to monitor the progress of your job on the Hadoop cluster. Alternately, you have folloing optoins:
+
+* YARN : Using YARN web UI generally running on 8088 on the node running resource manager
+* Apex cli: [Using apex cli to get running application information](http://apex.apache.org/docs/apex/apex_cli/#apex-cli-commands)


Apex cli -> Apex command-line interface

apex -> Apex
cli -> CLI

tweise · 2017-03-05T15:33:44Z

src/documentation/runners/apex.md

+
+Run the wordcount example
+```
+mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="--inputFile=/tmp/input/ --output=/tmp/output/ --runner=ApexRunner --embeddedExecution=false --configFile=beam-runners-apex.properties" -Papex-runner


Where do the properties come from?

One version, with limited documentation: https://github.com/apache/beam/blob/master/runners/apex/src/test/resources/beam-runners-apex.properties

Would be better with a link to the list of properties in a comment

The possible attributes are here: https://github.com/apache/apex-core/blob/master/api/src/main/java/com/datatorrent/api/Context.java

I think we should merge this PR and then add examples for the properties in a separate PR. The instructions are already useful and we validated that the YARN launcher works today.

davorbonaci · 2017-03-14T21:13:47Z

@sandeepdeshmukh, any update here perhaps?

davorbonaci · 2017-03-20T21:57:31Z

@sandeepdeshmukh, any update here perhaps?

sandeepdeshmukh · 2017-03-23T13:34:28Z

I am extremely sorry for the delay. I am on a long vacation with no compute access and hence requesting @chinmaykolhatkar to please take this up.

tweise · 2017-04-14T20:02:46Z

@davorbonaci @dhalperi as mentioned above, I would suggest to merge and open follow-up JIRA to clarify the properties usage.

tweise · 2017-04-14T20:19:01Z

@dhalperi the issue I run into when trying to run the example:

[WARNING] 
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Unable to find any files matching StaticValueProvider{value=file:///tmp/input/}
	at org.apache.beam.sdks.java.core.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:518)
	at org.apache.beam.sdk.io.TextIO$Read$Bound.expand(TextIO.java:257)
	at org.apache.beam.sdk.io.TextIO$Read$Bound.expand(TextIO.java:174)
	at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:475)
	at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:405)
	at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:58)
	at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:175)
	at org.apache.beam.examples.WordCount.main(WordCount.java:179)

The command line was:

mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="--inputFile=file:///tmp/input/ --output=file:///tmp/output/ --runner=ApexRunner --embeddedExecution=false --configFile=~/beam-runners-apex.properties" -Papex-runner

I tried it without the file scheme also, same result.

tweise · 2017-04-14T20:21:46Z

The Apex CLI binaries are at:

https://github.com/atrato/apex-cli-package/releases

@dhalperi can probably add some notes about troubleshooting of the pipeline when it runs on YARN.

dhalperi · 2017-04-20T16:47:10Z

@tweise -- sounds great. Can you take over, fixup as desired, and merge? beam-site instructions are in the Beam contribution guide

closes apache#98

closes apache/beam-site#98

davorbonaci requested changes Dec 2, 2016

View reviewed changes

sandeepdeshmukh force-pushed the apex-runner-doc branch from e47749d to ea2ba63 Compare December 2, 2016 12:58

tweise mentioned this pull request Dec 6, 2016

[BEAM-830] ApexRunner launch on YARN cluster. apache/beam#1517

Merged

4 tasks

davorbonaci reviewed Dec 6, 2016

View reviewed changes

sandeepdeshmukh force-pushed the apex-runner-doc branch from ea2ba63 to d80cd54 Compare December 27, 2016 08:16

davorbonaci requested changes Dec 29, 2016

View reviewed changes

sandeepdeshmukh force-pushed the apex-runner-doc branch from d80cd54 to 85fbcae Compare January 3, 2017 14:03

sandeepdeshmukh force-pushed the apex-runner-doc branch from 85fbcae to f126242 Compare January 13, 2017 11:57

tweise reviewed Feb 22, 2017

View reviewed changes

[BEAM-825] Fill in the documentation/runners/apex portion of the website

468d2fd

sandeepdeshmukh force-pushed the apex-runner-doc branch from a6b435e to 468d2fd Compare March 3, 2017 13:37

davorbonaci requested changes Mar 5, 2017

View reviewed changes

tweise reviewed Mar 5, 2017

View reviewed changes

tweise added a commit to tweise/incubator-beam-site that referenced this pull request Apr 21, 2017

Fix Apex runner instructions (pending review comments and other changes)

b681fa2

closes apache#98

tweise mentioned this pull request Apr 21, 2017

[BEAM-825] Fill in the documentation/runners/apex portion of the website #219

Closed

tweise added a commit to tweise/incubator-beam-site that referenced this pull request Apr 21, 2017

Fix Apex runner instructions (pending review comments and other changes)

239e000

closes apache#98

tweise added a commit to tweise/incubator-beam-site that referenced this pull request Apr 21, 2017

Fix Apex runner instructions (pending review comments and other changes)

7e9aa08

closes apache#98

asfgit closed this in 0da810c Apr 21, 2017

robertwb pushed a commit to robertwb/incubator-beam that referenced this pull request Jun 5, 2018

Fix Apex runner instructions (pending review comments and other changes)

2038935

closes apache/beam-site#98

melap pushed a commit to apache/beam that referenced this pull request Jun 20, 2018

Fix Apex runner instructions (pending review comments and other changes)

5385e57

closes apache/beam-site#98


		[Apache Apex](http://apex.apache.org/) is a stream processing platform and framework for low-latency, high-throughput and fault-tolerant analytics applications on Apache Hadoop. Apex is Java based and also provides its own API for application development (native compositional and declarative Java API, SQL) with a comprehensive [operator library](https://github.com/apache/apex-malhar). Apex has a unified streaming architecture and can be used for real-time and batch processing. With its stateful stream processing architecture Apex can support all of the concepts in the Beam model (event time, triggers, watermarks etc.).


		## Apex-Runner prerequisites and setup

		To use Apache Apex Runner for Beam, you need to have a Hadoop cluster with YARN setup. You can setup your own Hadoop cluster or simply download DataTorrent Sandbox which comes with Hadoop and Apex pre-installed.


		## Montoring progress of your job

		If you are using DataTorrent RTS Sandbox then you can monitor progress of the application on http://localhost:9090/ using your browser. The console provides you updates on the application progress, general statistics like number of operators, number of containers - planned and allcated, allocated memory to the application etc.


		[Apache Apex](http://apex.apache.org/) is a stream processing platform and framework for low-latency, high-throughput and fault-tolerant analytics applications on Apache Hadoop. Apex has a unified streaming architecture and can be used for real-time and batch processing. With its stateful stream processing architecture, Apex can support all of the concepts in the Beam model (event time, triggers, watermarks etc.).

		ApexRunner does not implement launch on YARN yet, hence we move the Beam pipeline code into Apex StreamingApplication to translate it into the Apex DAG and then launch the application using apex CLI. Later, instead of implementing Apex StreamingApplication, the main method will call the runner. The current process is an intermedia step as of 0.4.0.


		## Apex-Runner prerequisites and setup

		You may set up your own Hadoop cluster, and [setup Apache Apex on top of it](http://apex.apache.org/docs/apex/apex_development_setup/) or choose any vendor-specific distribution that includes Hadoop and Apex pre-installed. Please see the [distribution information on the Apache Apex website](http://apex.apache.org/downloads.html).


		You may set up your own Hadoop cluster, and [setup Apache Apex on top of it](http://apex.apache.org/docs/apex/apex_development_setup/) or choose any vendor-specific distribution that includes Hadoop and Apex pre-installed. Please see the [distribution information on the Apache Apex website](http://apex.apache.org/downloads.html).

		## Running wordcount using Apex-Runner


		## Running wordcount using Apex-Runner

		Download some data for processing and put it on HDFS


		## Checking output

		The sample program which is processing small amount of data would finish quickly. You can check contents on /tmp/output/ on HDFS


		Depending on your installation, you may be able to monitor the progress of your job on the Hadoop cluster. Alternately, you have folloing optoins:

		* YARN : Using YARN web UI generally running on 8088 on the node running resource manager

[BEAM-825] Fill in the documentation/runners/apex portion of the website #98

[BEAM-825] Fill in the documentation/runners/apex portion of the website #98

Conversation

sandeepdeshmukh commented Dec 1, 2016

asfbot commented Dec 1, 2016

asfbot commented Dec 1, 2016

asfbot commented Dec 1, 2016

asfbot commented Dec 1, 2016

asfbot commented Dec 1, 2016

asfbot commented Dec 1, 2016

sandeepdeshmukh commented Dec 1, 2016

davorbonaci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandeepdeshmukh Dec 2, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfbot commented Dec 2, 2016

asfbot commented Dec 2, 2016

tweise commented Dec 6, 2016

davorbonaci commented Dec 6, 2016

davorbonaci Dec 6, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tweise commented Dec 16, 2016

sandeepdeshmukh commented Dec 16, 2016

asfbot commented Dec 27, 2016

asfbot commented Dec 27, 2016

sandeepdeshmukh commented Dec 27, 2016

davorbonaci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfbot commented Jan 3, 2017

asfbot commented Jan 3, 2017

asfbot commented Jan 13, 2017

asfbot commented Jan 13, 2017

asfbot commented Feb 1, 2017

sandeepdeshmukh commented Feb 1, 2017

dhalperi commented Feb 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tweise commented Feb 22, 2017

sandeepdeshmukh commented Feb 23, 2017

davorbonaci commented Feb 28, 2017

sandeepdeshmukh commented Feb 28, 2017

tweise commented Feb 28, 2017

asfbot commented Mar 3, 2017

asfbot commented Mar 3, 2017

davorbonaci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davorbonaci commented Mar 14, 2017

sandeepdeshmukh Dec 2, 2016 •

edited

davorbonaci Dec 6, 2016 •

edited

sandeepdeshmukh commented Mar 23, 2017 •

edited