[BEAM-505] Fill in the documentation/runners/direct portion of the website #76

melap · 2016-11-11T18:48:10Z

…bsite

asfbot · 2016-11-11T18:48:44Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/93/
--none--

asfbot · 2016-11-11T18:49:38Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/48/

davorbonaci · 2016-11-11T21:39:42Z

retest this please

asfbot · 2016-11-11T21:40:47Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/50/

asfbot · 2016-11-11T21:42:58Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/95/

Jenkins built the site at commit id 38e0f07 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

davorbonaci

Beautiful! Just a few minor comments ;-)

davorbonaci · 2016-11-13T18:42:32Z

src/documentation/runners/direct.md

+* enforcing immutability of elements
+* enforcing encodability of elements
+* elements are processed in an arbitrary order at all points
+* serialization of user Fns (DoFn, CombineFn, etc.)


Perhaps Fns -> functions?
Perhaps code formatting for DoFn, CombineFn?

davorbonaci · 2016-11-13T18:44:08Z

src/documentation/runners/direct.md

+Using the direct runner for testing and development helps ensure that pipelines are robust across different Beam runners. In addition, debugging failed runs can be a non-trivial task when a pipeline executes on a remote cluster. Instead, it is often faster and simpler to perform local unit testing on your pipeline code. Unit testing your pipeline locally also allows you to use your preferred local debugging tools.
+
+Here are some resources with information about how to test your pipelines.
+* [Testing Unbounded Pipelines in Apache Beam](https://beam.incubator.apache.org/blog/2016/10/20/test-stream.html) talks about the use of Java classes [PAssert]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/testing/PAssert.html) and [TestStream]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/testing/TestStream.html) to test your pipelines.


Please don't hardcode the full URL -- it is fragile and guaranteed to break when we go TLP. Please link to baseurl/blog....

davorbonaci · 2016-11-13T18:44:24Z

src/documentation/runners/direct.md

+
+Here are some resources with information about how to test your pipelines.
+* [Testing Unbounded Pipelines in Apache Beam](https://beam.incubator.apache.org/blog/2016/10/20/test-stream.html) talks about the use of Java classes [PAssert]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/testing/PAssert.html) and [TestStream]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/testing/TestStream.html) to test your pipelines.
+* The [Apache Beam WordCount Example](http://beam.incubator.apache.org/get-started/wordcount-example/) contains an example of logging and testing a pipeline with `PAssert`.


Same here, no full URLs.

davorbonaci · 2016-11-13T18:45:46Z

src/documentation/runners/direct.md

+For example, if you are using Maven for development and want to use the SDK for Java with [DirectRunner]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/runners/direct/DirectRunner.html), add the following dependency to your `pom.xml` file:
+
+```
+   <dependency>


Can you abstract the text to be SDK-independent and use language toggles for the code snippet?

Yeah, it's a bit troublesome. If I put the text that specifies it's for Maven/Java outside of the language toggle, it looks strange if the user toggles python... but it also looks strange when I put the text inside the Java toggle because it forces everything inside to be code formatted, along with syntax highlighting. I will see how it looks with the Maven/Java specific text in a comment inside the Java toggle. Other ideas welcome!

davorbonaci · 2016-11-13T18:46:54Z

src/documentation/runners/direct.md

+   </dependency>
+```
+
+## Pipeline options for the direct runner


I think there are mentions of "Direct Runner", "direct runner" and "Direct runner". I think it would be good to be consistent.

I'll change everything to Direct Runner to be consistent with other runners.

davorbonaci · 2016-11-13T18:47:29Z

src/documentation/runners/direct.md

+
+## Pipeline options for the direct runner
+
+When executing your pipeline from the command-line, set `runner` to `direct` or `directrunner`. The default values for the pipeline options are generally sufficient.


for the other pipeline options are generally sufficient.

No need to mention directrunner, perhaps leave just direct.

davorbonaci · 2016-11-13T18:48:33Z

src/documentation/runners/direct.md

+
+## Additional information and caveats
+
+Local execution is limited by the memory available in your local environment. It is highly recommended that you run your pipeline with data sets small enough to fit in local memory. You can create a small in-memory data set using a Create transform, or you can use a Read transform to work with small local or remote files.


Perhaps code formatting for Create and Read along with Javadoc/Py links to them?

asfbot · 2016-11-14T22:46:52Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/69/
--none--

asfbot · 2016-11-14T22:49:45Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/113/

Jenkins built the site at commit id 8157e4e with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

davorbonaci

LGTM. Good to go!

davorbonaci · 2016-11-14T23:34:45Z

src/documentation/runners/direct.md

+
+Here are some resources with information about how to test your pipelines.
+* [Testing Unbounded Pipelines in Apache Beam]({{ site.baseurl }}/blog/2016/10/20/test-stream.html) talks about the use of Java classes [`PAssert`]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/testing/PAssert.html) and [`TestStream`]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/testing/TestStream.html) to test your pipelines.
+* The [Apache Beam WordCount Example]({{ site.baseurl }}/get-started/wordcount-example/) contains an example of logging and testing a pipeline with Java [`PAssert`]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/testing/PAssert.html).


this will probably be language independent. I'd remove word "Java" before PAssert.

davorbonaci · 2016-11-14T23:35:53Z

src/documentation/runners/direct.md

+```java
+// If you use Maven and the SDK for Java, add the following dependency to your pom.xml file:
+
+<dependency>


FYI: unfortunately this is trying to apply Java formatting to an XML snippet. Sad, but orthogonal to this PR.

davorbonaci · 2016-11-14T23:37:59Z

src/documentation/runners/direct.md

+You must specify your dependency on the Direct Runner.
+
+```java
+// If you use Maven and the SDK for Java, add the following dependency to your pom.xml file:


I'd remove this comment: // isn't a comment here. It's XML, so we'd need something like <!--.

davorbonaci · 2016-11-15T00:34:10Z

LGTM. Merging.

[BEAM-505] Fill in the documentation/runners/direct portion of the we…

38e0f07

…bsite

This was referenced Nov 13, 2016

[BEAM-899] Add Flink Instructions to quickstart.md #79

Merged

[BEAM-845] Update Apex runner info after merge to master. #78

Merged

davorbonaci requested changes Nov 13, 2016

View reviewed changes

davorbonaci mentioned this pull request Nov 13, 2016

[BEAM-508] Fill in the documentation/runners/dataflow portion of the website #77

Closed

Addressing code review comments, round 1

8157e4e

davorbonaci approved these changes Nov 14, 2016

View reviewed changes

davorbonaci reviewed Nov 14, 2016

View reviewed changes

Addressing code review comments, round 2

da44de8

asfgit closed this in a82a0f3 Nov 15, 2016

robertwb pushed a commit to robertwb/incubator-beam that referenced this pull request Jun 5, 2018

This closes apache/beam-site#76

c198967

melap pushed a commit to apache/beam that referenced this pull request Jun 20, 2018

This closes apache/beam-site#76

5ddc062

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-505] Fill in the documentation/runners/direct portion of the website #76

[BEAM-505] Fill in the documentation/runners/direct portion of the website #76

melap commented Nov 11, 2016

asfbot commented Nov 11, 2016

asfbot commented Nov 11, 2016

davorbonaci commented Nov 11, 2016

asfbot commented Nov 11, 2016

asfbot commented Nov 11, 2016

davorbonaci left a comment

davorbonaci Nov 13, 2016

melap Nov 14, 2016

davorbonaci Nov 13, 2016

melap Nov 14, 2016

davorbonaci Nov 13, 2016

melap Nov 14, 2016

davorbonaci Nov 13, 2016

melap Nov 14, 2016

davorbonaci Nov 13, 2016

melap Nov 14, 2016

davorbonaci Nov 13, 2016 •

edited

melap Nov 14, 2016

davorbonaci Nov 13, 2016

melap Nov 14, 2016

asfbot commented Nov 14, 2016

asfbot commented Nov 14, 2016

davorbonaci left a comment

davorbonaci Nov 14, 2016

melap Nov 15, 2016

davorbonaci Nov 14, 2016

davorbonaci Nov 14, 2016

melap Nov 15, 2016

davorbonaci commented Nov 15, 2016


		## Pipeline options for the direct runner

		When executing your pipeline from the command-line, set `runner` to `direct` or `directrunner`. The default values for the pipeline options are generally sufficient.


		## Additional information and caveats

		Local execution is limited by the memory available in your local environment. It is highly recommended that you run your pipeline with data sets small enough to fit in local memory. You can create a small in-memory data set using a Create transform, or you can use a Read transform to work with small local or remote files.

[BEAM-505] Fill in the documentation/runners/direct portion of the website #76

[BEAM-505] Fill in the documentation/runners/direct portion of the website #76

Conversation

melap commented Nov 11, 2016

asfbot commented Nov 11, 2016

asfbot commented Nov 11, 2016

davorbonaci commented Nov 11, 2016

asfbot commented Nov 11, 2016

asfbot commented Nov 11, 2016

davorbonaci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davorbonaci Nov 13, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfbot commented Nov 14, 2016

asfbot commented Nov 14, 2016

davorbonaci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davorbonaci commented Nov 15, 2016

davorbonaci Nov 13, 2016 •

edited