[BEAM-1665] Add Pipeline I/O section to website - outline + move some existing txt #173

ssisk · 2017-03-09T02:10:28Z

This PR has a fair number of TODOs in the website content, but I have PRs queued up with more content from the IO authoring guide and none of the TODOs are for content already on the website.

It has very little actual new content - mostly, it created structure and moves existing content around.

I did not to go with a single page for all this content b/c both java and python have enough unique content that they deserve their own separate sections (ie, just tabs on the code isn't enough), and the "click to the next page" model currently implemented allows the user to pick java vs python, but then after reading those pages, the next page for both points at the same place - the users mostly follow the same path, but for java vs python specific content, they will diverge then converge again.
I moved the "list of built-in I/O" content over to it's own separate page since it'd be nice to have more content there - e.g. capabilities matrix, and it felt special enough to pull out of the programming guide.
We decided not to put all of this content in the contribute section of the site since the expectation is we don't think all users will contribute their IO transforms, so we want most of the docs to just be about writing an IO transforms, and they lay out the expectations in the contribute part of the IO section.

R @melap

asfbot · 2017-03-09T02:13:52Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/285/
--none--

asfbot · 2017-03-09T02:19:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/327/

Jenkins built the site at commit id b0e0862 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

melap

Sorry for the delay, thanks for setting up the skeleton for a new I/O section! I've added some comments.

melap · 2017-03-14T05:13:12Z

src/documentation/io/built-in.md

+---
+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+


this page is missing a title

done, thanks!

melap · 2017-03-14T05:16:55Z

src/documentation/io/built-in.md

+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+
+This is the set of I/O transforms currently available in Beam.


hmm, what about something like "This table contains the currently available I/O transforms."

melap · 2017-03-14T05:28:45Z

src/documentation/io/built-in.md

+
+This is the set of I/O transforms currently available in Beam.
+
+Consult the [Programming Guide I/O section]({{site.baseurl }}/documentation/programming-guide#io) for general usage instructions and the javadoc/pydoc for the particular I/O transforms.


I initially read it quickly as the instructions and javadoc/pydoc are both available in the programming guide. perhaps add a comma and "see" --> "usage instructions, and see the javadoc ..."

melap · 2017-03-14T05:31:21Z

src/documentation/io/contributing.md

+
+# Contributing I/O Transforms to Beam
+
+* If you are planning to contribute your I/O transform to the beam community, you'll be going through the normal beam contribution life cycle - see the Apache Beam Contribution Guide for more details.


in general we've been trying to always have first mention of Beam be prefixed with Apache. maybe add Apache to the H1 title?

beam should be capitalized (x2)

link to the contribution guide?

done. I ended up from the "to Beam" part, but I moved the Apache to the first instance of beam in the doc

melap · 2017-03-14T05:40:58Z

src/documentation/io/testing.md

+
+# Next steps
+
+If you've got a well tested I/O transform, why not contribute it to Beam? Read all about it: 


"If you have a"

melap · 2017-03-14T06:12:12Z

src/documentation/programming-guide.md

-</tr>
-</table>
+### Beam-provided I/O Transforms
+There is a page devoted to [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/).


I'm wondering if it might be good to specify what's roughly on the page, so users can easily decide if they want/need to jump there or not. though, this would need to be updated again later, after/if more is added to the page. and admittedly, the sentence below is a bit redundant right now while it's just a list of the provided transforms.

See the Beam-provided I/O Transforms page for a list of the currently available I/O transforms.

yeah, I struggled with the wording here, this feels better.

done.

melap · 2017-03-14T06:12:27Z

src/documentation/sdks/java.md

-
-
+## Pipeline I/O
+There is a page devoted to [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/).


same as above

melap · 2017-03-14T06:30:52Z

src/documentation/io/authoring-overview.md

+title: "Authoring I/O - Overview"
+permalink: /documentation/io/authoring-overview/
+---
+


nit: a couple extra blank lines

melap · 2017-03-14T06:34:26Z

src/documentation/io/authoring-overview.md

+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+
+# Authoring I/O - Overview


the two titles (metadata at top and this H1) are missing the "Transforms", "Authoring I/O Transforms"

melap · 2017-03-14T06:56:30Z

src/documentation/io/authoring-overview.md

+
+# Authoring I/O - Overview
+
+_A guide for those who need to connect to a data store not covered by a [Built-in I/O Transforms]({{site.baseurl }}/documentation/io/built-in/)_


slight tweaks, maybe:
A guide for users who need to connect to a data store that isn't supported by the [Built-in I/O Transforms].

…ntent * I did not to go with a single page for all this content b/c both java and python have enough unique content that they deserve their own separate sections (ie, just tabs on the code isn't enough), and the "click to the next page" model currently implemented allows the user to pick java vs python, but then after reading those pages, the next page for both points at the same place - the users mostly follow the same path, but for java vs python specific content, they will diverge then converge again. * I moved the "list of built-in I/O" content over to it's own separate page since it'd be nice to have more content there - e.g. capabilities matrix, and it felt special enough to pull out of the programming guide. * We decided not to put all of this content in the contribute section of the site since the expectation is we don't think all users will contribute their IO transforms, so we want most of the docs to just be about writing an IO transforms, and they lay out the expectations in the contribute part of the IO section.

asfbot · 2017-03-14T23:17:22Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/300/
--none--

ssisk

okay! thanks so much for all your comments.

ssisk · 2017-03-14T22:01:24Z

src/documentation/io/built-in.md

+---
+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+


done, thanks!

ssisk · 2017-03-14T22:01:51Z

src/documentation/io/authoring-overview.md

+title: "Authoring I/O - Overview"
+permalink: /documentation/io/authoring-overview/
+---
+


ssisk · 2017-03-14T22:02:38Z

src/documentation/io/authoring-overview.md

+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+
+# Authoring I/O - Overview


ssisk · 2017-03-14T22:03:22Z

src/documentation/io/authoring-overview.md

+
+# Authoring I/O - Overview
+
+_A guide for those who need to connect to a data store not covered by a [Built-in I/O Transforms]({{site.baseurl }}/documentation/io/built-in/)_


ssisk · 2017-03-14T22:04:31Z

src/documentation/io/built-in.md

+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+
+This is the set of I/O transforms currently available in Beam.


ssisk · 2017-03-14T22:05:24Z

src/documentation/io/built-in.md

+
+This is the set of I/O transforms currently available in Beam.
+
+Consult the [Programming Guide I/O section]({{site.baseurl }}/documentation/programming-guide#io) for general usage instructions and the javadoc/pydoc for the particular I/O transforms.


ssisk · 2017-03-14T22:08:35Z

src/documentation/io/contributing.md

+
+# Contributing I/O Transforms to Beam
+
+* If you are planning to contribute your I/O transform to the beam community, you'll be going through the normal beam contribution life cycle - see the Apache Beam Contribution Guide for more details.


done. I ended up from the "to Beam" part, but I moved the Apache to the first instance of beam in the doc

ssisk · 2017-03-14T22:08:59Z

src/documentation/io/testing.md

+
+# Next steps
+
+If you've got a well tested I/O transform, why not contribute it to Beam? Read all about it: 


ssisk · 2017-03-14T22:12:28Z

src/documentation/programming-guide.md

-</tr>
-</table>
+### Beam-provided I/O Transforms
+There is a page devoted to [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/).


yeah, I struggled with the wording here, this feels better.

done.

ssisk · 2017-03-14T22:14:42Z

src/documentation/sdks/java.md

-
-
+## Pipeline I/O
+There is a page devoted to [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/).


asfbot · 2017-03-14T23:22:17Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/342/

Jenkins built the site at commit id a6f367a with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

melap · 2017-03-14T23:35:02Z

Thanks, LGTM as a skeleton starting point. Looking forward to seeing it full of good stuff.

ssisk · 2017-03-15T16:05:06Z

Thanks for the review Melissa!
cc @davorbonaci @francesperry for merge (and review if you'd like)

davorbonaci · 2017-03-15T20:32:46Z

I'm worried about publishing this as-is. We could do it, but users will just see a ton of TODOs, probably resulting in worse experience than without this content.

@ssisk, can we perhaps merge it, but comment it out, so users don't see so many TODOs?

ssisk · 2017-03-15T21:27:25Z

sure, I'll take a pass and see what I can comment out so that it doesn't present a ton of TODOs to the users.

That'll probably look like: initially the TOC page will be there and the "using" section will be visible, but not the authoring section.

asfbot · 2017-03-17T00:53:57Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/308/
--none--

ssisk · 2017-03-17T00:56:31Z

okay! I've:

Updated the TOC so that it only displays one "this is a work in progress" note for the authoring content, and users won't see the authoring content yet/the other set of TODOs.
Looked around to see if there's any pointers directly into the "authoring transforms" content, and changed the one reference (in the programming guide) to point at the IO TOC instead of directly at the content.

PTAL

asfbot · 2017-03-17T01:00:29Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/350/

Jenkins built the site at commit id 1f4d996 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

…ting content

asfbot · 2017-03-18T00:18:44Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/314/
--none--

asfbot · 2017-03-18T00:31:54Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/356/

Jenkins built the site at commit id be4463f with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

…ting content

asfbot · 2017-03-18T00:47:57Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Test/315/
--none--

asfbot · 2017-03-18T01:02:55Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Website_Stage/357/

Jenkins built the site at commit id 76bf199 with Jekyll and staged it here. Happy reviewing.

Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again.

davorbonaci

LGTM.

Test breakage seems unrelated GitHub outage. Merging.

melap reviewed Mar 14, 2017

View reviewed changes

ssisk force-pushed the io-guide branch from b0e0862 to a6f367a Compare March 14, 2017 23:15

ssisk commented Mar 14, 2017

View reviewed changes

fixup! Add Pipeline I/O section to website - outline + move some exis…

be4463f

…ting content

ssisk force-pushed the io-guide branch from 1f4d996 to be4463f Compare March 18, 2017 00:17

fixup! Add Pipeline I/O section to website - outline + move some exis…

76bf199

…ting content

davorbonaci approved these changes Mar 18, 2017

View reviewed changes

asfgit closed this in b574876 Mar 18, 2017

ssisk deleted the io-guide branch March 28, 2017 23:13

robertwb pushed a commit to robertwb/incubator-beam that referenced this pull request Jun 5, 2018

This closes apache/beam-site#173

acd615c

robertwb pushed a commit to robertwb/incubator-beam that referenced this pull request Jun 5, 2018

This closes apache/beam-site#173

9193fe8

melap pushed a commit to apache/beam that referenced this pull request Jun 20, 2018

This closes apache/beam-site#173

119c5d5

		---

		[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)


		[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)

		This is the set of I/O transforms currently available in Beam.


		This is the set of I/O transforms currently available in Beam.

		Consult the [Programming Guide I/O section]({{site.baseurl }}/documentation/programming-guide#io) for general usage instructions and the javadoc/pydoc for the particular I/O transforms.


		# Contributing I/O Transforms to Beam

		* If you are planning to contribute your I/O transform to the beam community, you'll be going through the normal beam contribution life cycle - see the Apache Beam Contribution Guide for more details.


		# Next steps

		If you've got a well tested I/O transform, why not contribute it to Beam? Read all about it:



		## Pipeline I/O
		There is a page devoted to [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/).


		[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)

		# Authoring I/O - Overview


		# Authoring I/O - Overview

		_A guide for those who need to connect to a data store not covered by a [Built-in I/O Transforms]({{site.baseurl }}/documentation/io/built-in/)_

[BEAM-1665] Add Pipeline I/O section to website - outline + move some existing txt #173

[BEAM-1665] Add Pipeline I/O section to website - outline + move some existing txt #173

Conversation

ssisk commented Mar 9, 2017

asfbot commented Mar 9, 2017

asfbot commented Mar 9, 2017

melap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfbot commented Mar 14, 2017

ssisk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfbot commented Mar 14, 2017

melap commented Mar 14, 2017

ssisk commented Mar 15, 2017

davorbonaci commented Mar 15, 2017

ssisk commented Mar 15, 2017

asfbot commented Mar 17, 2017

ssisk commented Mar 17, 2017

asfbot commented Mar 17, 2017

asfbot commented Mar 18, 2017

asfbot commented Mar 18, 2017

asfbot commented Mar 18, 2017

asfbot commented Mar 18, 2017

davorbonaci left a comment

Choose a reason for hiding this comment