Do not merge: scratch work 2 by charlesccychen · Pull Request #3646 · apache/beam

charlesccychen · 2017-07-26T18:01:48Z

Follow this checklist to help us incorporate your contribution quickly and easily:

Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
Each commit in the pull request should have a meaningful subject line and body.
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

…ectRunner.

…prepare to expose DirectUnmergedState API.

Fixes earliest hold to be < instead of <=.

…side inputs.

PubSubIO for direct runner.

charlesccychen

R: @cosmoskitten

cosmoskitten

Please fix before approval.

At sdks/python/apache_beam/pipeline.py:139:

    self.transforms_stack = [AppliedPTransform(None, None, '', None)]

Thanks!

At sdks/python/apache_beam/pipeline.py:160:

  def run(self, test_runner_api=True):

I don't like this name.

cosmoskitten · 2017-07-26T18:16:19Z

@@ -161,7 +161,7 @@ def run(self, test_runner_api=True):
    """Runs the pipeline. Returns whatever our runner returns after running."""

    # When possible, invoke a round trip through the runner API.


cosmoskitten

LGTM

charlesccychen

At sdks/python/apache_beam/pipeline.py:160:

  def run(self, test_runner_api=True):

cosmoskitten wrote:
I don't like this name.

Done.

charlesccychen · 2017-07-26T18:17:58Z

@@ -161,7 +161,7 @@ def run(self, test_runner_api=True):
    """Runs the pipeline. Returns whatever our runner returns after running."""

    # When possible, invoke a round trip through the runner API.


cosmoskitten wrote:
Please fix

Done.

charlesccychen

Please fix.

At sdks/python/apache_beam/pvalue.py:103:

class PCollection(PValue):

Please fix PCollection.

charlesccychen · 2017-07-28T19:53:09Z

-    return _Bundle(output_pcollection, self._stacked)
+    return _Bundle(output_pcollection, None, stacked=self._stacked)
+
+  def create_keyed_bundle(self, output_pcollection, key):


Please fix create_keyed_bundle.

cosmoskitten

Please fix.

charlesccychen

R: @aaltay

aaltay

Thanks! this looks good.

At sdks/python/apache_beam/pvalue.py:103:

class PCollection(PValue):

charlesccychen wrote:
Please fix PCollection.

Acknowledged.

aaltay · 2017-07-28T20:52:38Z

-    return _Bundle(output_pcollection, self._stacked)
+    return _Bundle(output_pcollection, None, stacked=self._stacked)
+
+  def create_keyed_bundle(self, output_pcollection, key):


charlesccychen wrote:
Please fix create_keyed_bundle.

Done.

aaltay

charlesccychen wrote:
Please fix.

Done.

charlesccychen and others added 22 commits June 6, 2017 14:40

streaming test

3383c26

Working TestStream evaluator

038f3f3

Slightly better watermark propagation, debugging print cleanup.

fc661f6

Factor out GroupAlsoByWindows so that it can be overridden by the Dir…

053c7ae

…ectRunner.

Don't buffer elements in GroupByKeyOnly.

c2b3f32

Skeleton implementation of GroupAlsoByKeyEvaluator.

8854c53

Mostly working streaming GroupAlsoByWindow.

8be01a9

Introduce keyed bundles, KeyedWorkItems.

bf38fe7

Rename old non-keyed state to legacy_state, introduce KeyedWorkItem, …

ae0d80d

…prepare to expose DirectUnmergedState API.

Fix rebase to head after Beam 2.0 release.

97de556

Clarify transform_keyed_states.

341fa92

Support timer firings.

1e83d5e

Reintroduce hacks from head.

256309f

Add debugging output.

25ce9f1

Don't hold input watermark on pending elements.

7be85c6

Finish moving earliest hold extraction to evaluators, dbg output++

1df07a5

Batch WordCount works in streaming mode!!

e52ba42

Fixes earliest hold to be < instead of <=.

Working batch and streaming mode execution, except for certain empty …

e5bb716

…side inputs.

PubSubIO for direct runner.

91873cc

Merge pull request #1 from robertwb/streaming

e05778f

PubSubIO for direct runner.

Use keyed holds in watermark manager

12a9517

Fix dropped messages in PubSubRead.

b90a2e4

charlesccychen commented Jul 26, 2017

View reviewed changes

cosmoskitten reviewed Jul 26, 2017

View reviewed changes

cosmoskitten approved these changes Jul 26, 2017

View reviewed changes

charlesccychen commented Jul 26, 2017

View reviewed changes

charlesccychen commented Jul 28, 2017

View reviewed changes

cosmoskitten suggested changes Jul 28, 2017

View reviewed changes

charlesccychen commented Jul 28, 2017

View reviewed changes

aaltay approved these changes Jul 28, 2017

View reviewed changes

charlesccychen closed this Jul 28, 2017

		@@ -161,7 +161,7 @@ def run(self, test_runner_api=True):
		"""Runs the pipeline. Returns whatever our runner returns after running."""

		# When possible, invoke a round trip through the runner API.

Conversation

charlesccychen commented Jul 26, 2017

Uh oh!

charlesccychen left a comment

Choose a reason for hiding this comment

Uh oh!

cosmoskitten left a comment

Choose a reason for hiding this comment

Uh oh!

cosmoskitten Jul 26, 2017

Choose a reason for hiding this comment

Uh oh!

cosmoskitten left a comment

Choose a reason for hiding this comment

Uh oh!

charlesccychen left a comment

Choose a reason for hiding this comment

Uh oh!

charlesccychen Jul 26, 2017

Choose a reason for hiding this comment

Uh oh!

charlesccychen left a comment

Choose a reason for hiding this comment

Uh oh!

charlesccychen Jul 28, 2017

Choose a reason for hiding this comment

Uh oh!

cosmoskitten left a comment

Choose a reason for hiding this comment

Uh oh!

charlesccychen left a comment

Choose a reason for hiding this comment

Uh oh!

aaltay left a comment

Choose a reason for hiding this comment

Uh oh!

aaltay Jul 28, 2017

Choose a reason for hiding this comment

Uh oh!

aaltay left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants