[BEAM-2083] Go SDK prototype #4200
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Separation of graph and beam-level operations. Fine control over composites. A lot of freedom in surrounding helper code. * DoFn classification and runtime shims, which we will rely heavily on for concretely-typed DoFns. * Runs word count.
* Instead of java’s option registry, we pass pipeline-time options directly to the relevant transforms. * Removes the hack of using closures in textio to capture the filename.
* Improved some of the classification code. * Moved “count” to be generic.
* Avoids the explicit error propagation.
* The local and real Dataflow runners use the old client, so we must confirm to the v1b3 schema. * Successfully creates non-functional WordCap jobs on Dataflow: https://screenshot.googleplex.com/Lc8rhX4xcN5
* Intended to work against new job type, which is still underway. * Added a bit of display data.
* Go makes it harder to register runners implicitly, because runners ultimately need to be a dependency of the main program.
* We might want to consider dynamically loaded runner plugins, but for now all built-in runners are simply always included.
* Added fake size estimate for input sources.
* All examples are accepted by Dataflow now and render in the UX. * Much simpler examples than pingpong (unedited) break the UX. I suspect that composites must be a linear sequence of steps and are parsed like a stack.
* Also make it retry, so that we’re not quite as sensitive to startup order on the worker.
* Fix translation for ParDo sinks.
* Runs Wordcap on Dataflow (dev) * Updated fn_api photo.
* Add Filter with universal types
Apply - removed - it was only used by aggregators to simplify using the combine function outside of a parallel context. We don't have general purpose aggregators in Beam 1.0 so it's not needed. Default - removed - only use case is to help Java understand what data types are. We get this from reflection machinery, so unneeded. Compact - kept - borrowed from existing documentation to clarify the use case. Also, compact is only valid for keyed combine functions, so updated the docs to reflect that.
Clarify what the required methods are, how the signatures need uniformity, and the failure modes for violations of those requirements.
Change-Id: Iea1686b6309df97ca04776f50dff0fc6084e3473 Reviewed-on: https://team-review.git.corp.google.com/135842 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Bill Neubauer <wcn@google.com>
Change-Id: I17cd481d5856f43a7c9052fed34d0a3127ee8205 Reviewed-on: https://team-review.git.corp.google.com/136501 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Henning Rohde <herohde@google.com>
Change-Id: I77a1928559213f8412d152081c6126d16fd020d9 Reviewed-on: https://team-review.git.corp.google.com/137589 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Bill Neubauer <wcn@google.com>
Change-Id: I6846aaf711236773fb15af3baf856605820882ba Reviewed-on: https://team-review.git.corp.google.com/137368 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Bill Neubauer <wcn@google.com>
Change-Id: I98431303c4b7f15c0eb0ba2b0cad23dd9cdfed94 Reviewed-on: https://team-review.git.corp.google.com/137085 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Bill Neubauer <wcn@google.com>
Change-Id: I596f41a024701316e26adc7ea8ba190db9413323 Reviewed-on: https://team-review.git.corp.google.com/137262 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Bill Neubauer <wcn@google.com>
Change-Id: I3d696a56df2dd3aea5a8f9135788f85ed9e04622 Reviewed-on: https://team-review.git.corp.google.com/144971 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Burke <rebo@google.com>
Change-Id: I6867172b94a33cf946820e967e8a37095c2bcbbc Reviewed-on: https://team-review.git.corp.google.com/145307 Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Verified-Style: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Burke <rebo@google.com>
Fixing a typo and adding some background. Change-Id: Iff6c7fb5526ff0503e0bbebe37896a3650071c1f Reviewed-on: https://team-review.git.corp.google.com/145479 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Henning Rohde <herohde@google.com>
Change-Id: I9bc6098347e7944cc4822a4db42b4895ee76d9f4 Reviewed-on: https://team-review.git.corp.google.com/143709 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Henning Rohde <herohde@google.com> Reviewed-by: Robert Burke <rebo@google.com>
Change-Id: I42db154aae9727a53e167328949a10c36209e33b Reviewed-on: https://team-review.git.corp.google.com/146732 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Bill Neubauer <wcn@google.com> Reviewed-by: Robert Burke <rebo@google.com>
Change-Id: I64a7a5c1da3897ce99c2f6d948ffb0d40ddd842e Reviewed-on: https://team-review.git.corp.google.com/155587 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Bill Neubauer <wcn@google.com>
Change-Id: I50bfba445093fe161393a99a8bae9fdd7640e72e Reviewed-on: https://team-review.git.corp.google.com/155588 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Bill Neubauer <wcn@google.com>
Change-Id: If48838f996adc41bd646acbe85bf01b8e486940e Reviewed-on: https://team-review.git.corp.google.com/155669 Reviewed-by: Bill Neubauer <wcn@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Verified-Style: kokoro <noreply+kokoro@google.com>
Change-Id: I73e2f83a621c08243a577dca36af6462e4eb2056 Reviewed-on: https://team-review.git.corp.google.com/155987 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Burke <rebo@google.com>
Change-Id: I6b4bbfb088f1d2752a509c92915b1a86ae09e61a Reviewed-on: https://team-review.git.corp.google.com/159087 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Henning Rohde <herohde@google.com> Reviewed-by: Robert Burke <rebo@google.com>
Change-Id: I034e421203c35092e6823a753c24aef2c23d1b53 Reviewed-on: https://team-review.git.corp.google.com/156009 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Burke <rebo@google.com> Reviewed-by: Bill Neubauer <wcn@google.com>
Change-Id: I8d74ee96445e39048f05948e31d53a2b58df8721 Reviewed-on: https://team-review.git.corp.google.com/155788 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Henning Rohde <herohde@google.com>
Change-Id: I82facf56b933a692cb0c1fff8c8c2d4c0cb2cc99 Reviewed-on: https://team-review.git.corp.google.com/160787 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Burke <rebo@google.com>
Change-Id: Ieff7bccec815d54aac6d4f0f57f59647a8ee2585 Reviewed-on: https://team-review.git.corp.google.com/161492 Reviewed-by: Robert Burke <rebo@google.com> Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com>
* Extend proto unmarshaller to understand CoderRef leafs to accomodate Dataflow. * Remove old complex coder translation in harness Change-Id: I5fa8728004550f999fc37764baa2b8c8a78357f5 Reviewed-on: https://team-review.git.corp.google.com/167228 Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Burke <rebo@google.com>
Change-Id: Icee5dfdc628bd31b1c14f14a9946ac8825d56bd4 Reviewed-on: https://team-review.git.corp.google.com/157291 Reviewed-by: Bill Neubauer <wcn@google.com> Verified-Style: kokoro <noreply+kokoro@google.com> Verified-BuildAndTest: kokoro <noreply+kokoro@google.com>
|
retest this please |
|
Test failures are not related. Great to see it green already. |
|
R: @kennknowles (for the merge into go-sdk) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prototype of a Go SDK based on the design described here:
https://s.apache.org/beam-go-sdk-design-rfc
Work in progress. Note that PR is to the go-sdk branch.