[BEAM-53] Java-only Pubsub sink for streaming. #171

mshields822 · 2016-04-12T20:17:19Z

No description provided.

mshields822 · 2016-04-12T20:17:38Z

mshields822 · 2016-04-12T20:21:20Z

Note the progression were on is:
pubsub-grpc DONE
pubsub-sink (this)
pubsub (for Source)
pubsub-runner (rework PubsubIO and google runners)
nexmark (target)

tgroh · 2016-04-13T17:31:15Z

sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/io/PubsubUnboundedSink.java

+  // ================================================================================
+
+  /**
+   * Number of cores available for publishing.


What is a core in this context? My initial instinct is across machines, but it seems to be being used as the sharding factor.

This is now 'numShards' and needs to be chosen to balance num records batch with pubsub latency. Note that a random long indeed shards but results in most bundles having a single element which kills apiary/grpc quota.

I currently (ie in a pending branch) hard code this in PubsubIO. It is comparable to the heuristics we have baked in for calculating the initial splits for an UnboundedReader.

kennknowles · 2016-04-14T02:16:03Z

Better to have the owner of I/O review this. Also just want to call out here that the follow-ups suggested on #120 should probably go first, but I'll leave that up to Dan.

R: @dhalperi -@kennknowles

mshields822 · 2016-04-14T03:32:48Z

ACK working on a pubsub-apiary follow up. Thanks!

On Wed, Apr 13, 2016 at 7:16 PM, Kenn Knowles notifications@github.com
wrote:

Better to have the owner of I/O review this. Also just want to call out
here that the follow-ups suggested on #120
#120 should probably go
first, but I'll leave that up to Dan.

R: @dhalperi https://github.com/dhalperi -@kennknowles
https://github.com/kennknowles

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#171 (comment)

tgroh · 2016-04-15T01:18:48Z

sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/io/PubsubUnboundedSink.java

+      elementCounter.addValue(1L);
+      byte[] elementBytes = CoderUtils.encodeToByteArray(elementCoder, c.element());
+      long timestampMsSinceEpoch = c.timestamp().getMillis();
+      c.output(KV.of(ThreadLocalRandom.current().nextInt(numCores * SCALE_OUT),


Using a random int will allow most systems to choose the sharding to an even more arbitrary degree, and then you can remove numCores

See comment above.

Backport apache#30

dhalperi · 2016-04-25T15:11:15Z

Looking into this -- looks like you'll have to rebase. I see this code is pre-name-change.

mshields822 · 2016-05-04T17:49:04Z

PTAL
Note this is not wired into anything yet - I plan to hook this and the upcoming PubsubUnboundedSource into PubsubIO in a stand-alone PR.

kennknowles · 2016-05-06T03:31:55Z

sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubUnboundedSink.java

+  /**
+   * Coder for conveying outgoing messages between internal stages.
+   */
+  private static final Coder<PubsubClient.OutgoingMessage> CODER = new


Noting that this is a case we support quite poorly today: A library author offers a sink, requiring a user to convert to a particular datatype Foo. The library author writes a coder for Foo and would like the user to get this benefit automatically.

Not necessarily directly applicable here, but putting it out there.

Note this is entirely internal to the Sink / PubsubClient interface. I could put the coder in PubsubClient but since it has no other PCollection/Coder/etc dependencies it felt better leaving it outside.

kennknowles · 2016-05-06T03:33:02Z

Added a couple initial comments to share the review load.

dhalperi · 2016-05-07T01:26:06Z

sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubUnboundedSink.java

+      elementCounter.addValue(1L);
+      byte[] elementBytes = CoderUtils.encodeToByteArray(elementCoder, c.element());
+      long timestampMsSinceEpoch = c.timestamp().getMillis();
+      c.output(KV.of(ThreadLocalRandom.current().nextInt(numShards),


it is preferable (in terms of serialization overhead, which I understand is important for streaming) to make these static inner classes and then pass in constants in constructors. But up to you.

Yeah, good catch.

mshields822 · 2016-05-10T19:03:36Z

PTAL.
Went back and made the factories in PubsubXClient static classes too just to be consistent.

dhalperi · 2016-05-11T02:17:13Z

sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubUnboundedSink.java

+     * BLOCKING
+     * Send {@code messages} as a batch to Pubsub.
+     */
+    private void publishBatch(List<PubsubClient.OutgoingMessage> messages, int bytes)


I don't much care, but you write this as both OutgoingMessage and PubsubClient.OutgoingMessage in this file. Probably could pick one and stick with it.

dhalperi · 2016-05-11T02:24:28Z

Generally looks good to me. Let's sync tomorrow over any remaining comments.

mshields822 · 2016-05-11T22:18:58Z

PTAL
Beefed up unit testing for batching, though it's fragile due to direct runner's own choices on how to bundle.
Added display metadata.

dhalperi · 2016-05-11T22:50:41Z

sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubUnboundedSink.java

        throws IOException {
      long nowMsSinceEpoch = System.currentTimeMillis();
      int n = pubsubClient.publish(topic, messages);
-      Preconditions.checkState(n == messages.size());
+      checkState(n == messages.size(), "Attempted to publish %d messaged but %d were successful",


typo: messages

mshields822 · 2016-05-12T00:04:25Z

PTAL

…th-value-sorting 158 extend reduce by key with value sorting

Co-authored-by: Elias Segundo <elias.segundo@luisrazo.local> Co-authored-by: Elias Segundo Antonio <eliassegundo.segundo@gmail.com> Co-authored-by: Elias Segundo <elias.segundo@luisrazo.local> Co-authored-by: Danny McCormick <dannymccormick@google.com>

tgroh reviewed Apr 13, 2016
View reviewed changes

tgroh reviewed Apr 15, 2016
View reviewed changes

swegner pushed a commit to swegner/beam that referenced this pull request Apr 22, 2016

Merge pull request apache#171 from dhalperi/backport-30

339eb0e

Backport apache#30

kennknowles reviewed May 6, 2016
View reviewed changes

dhalperi reviewed May 7, 2016
View reviewed changes

dhalperi reviewed May 11, 2016
View reviewed changes

Mark Shields added 6 commits May 11, 2016 17:03

Add PubsubUnboundedSink and tests

275d0f0

Quick polish

915813a

Make the factories static classes.

ef37b4b

Dan and Ken's comments. Make sure all classes are static.

12cb6fe

Dan's comments. Unit test batching by number and byte size

e47ebeb

More Dan comments

128b7e8

asfgit closed this in 2fbc0ea May 12, 2016

mshields822 mentioned this pull request May 19, 2016

Backport Java-only Pubsub GoogleCloudPlatform/DataflowJavaSDK#273

Closed

dhalperi mentioned this pull request May 20, 2016

PubsubIO: implement an unbounded source and sink GoogleCloudPlatform/DataflowJavaSDK#278

Merged

iemejia pushed a commit to iemejia/beam that referenced this pull request Jan 12, 2018

This closes apache#171

dd1ca3f

mareksimunek pushed a commit to mareksimunek/beam that referenced this pull request May 9, 2018

Merge pull request apache#171 from seznam/158-extend-reduce-by-key-wi…

16daaee

…th-value-sorting 158 extend reduce by key with value sorting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-53] Java-only Pubsub sink for streaming. #171

[BEAM-53] Java-only Pubsub sink for streaming. #171

mshields822 commented Apr 12, 2016

mshields822 commented Apr 12, 2016

mshields822 commented Apr 12, 2016

tgroh Apr 13, 2016

mshields822 May 4, 2016

kennknowles commented Apr 14, 2016

mshields822 commented Apr 14, 2016

tgroh Apr 15, 2016

mshields822 May 4, 2016

dhalperi commented Apr 25, 2016

mshields822 commented May 4, 2016

kennknowles May 6, 2016

mshields822 May 10, 2016

kennknowles commented May 6, 2016

dhalperi May 7, 2016

mshields822 May 10, 2016

mshields822 commented May 10, 2016

dhalperi May 11, 2016

mshields822 May 11, 2016

dhalperi commented May 11, 2016

mshields822 commented May 11, 2016

dhalperi May 11, 2016

mshields822 May 11, 2016

mshields822 commented May 12, 2016

[BEAM-53] Java-only Pubsub sink for streaming. #171

[BEAM-53] Java-only Pubsub sink for streaming. #171

Conversation

mshields822 commented Apr 12, 2016

mshields822 commented Apr 12, 2016

mshields822 commented Apr 12, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennknowles commented Apr 14, 2016

mshields822 commented Apr 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhalperi commented Apr 25, 2016

mshields822 commented May 4, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennknowles commented May 6, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mshields822 commented May 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhalperi commented May 11, 2016

mshields822 commented May 11, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mshields822 commented May 12, 2016