Google Pub Sub connector built on Akka Grpc #1162

2m · 2018-08-23T07:58:55Z

This is an alternative connector to Google Pub Sub that uses gRPC protocol.

This is a continuation of #759 with the difference that it uses Akka Grpc instead of Google Java gRPC libraries.

This is still WIP, as it needs the following:

separate Java and Scala extensions, so Java sub/pub client is not created when only Scala one is used or vice versa.

2m · 2018-08-24T07:57:03Z

Code is ready to be reviewed.

Docs still need an update, since those have been copied from the current Google Pub Sub connector.

...-grpc/src/main/scala/akka/stream/alpakka/googlecloud/pubsub/grpc/impl/AkkaGrpcSettings.scala

build.sbt

2m · 2018-08-24T11:37:05Z

Updated the docs to reflect the configuration of this connector.

ennru

Impressive how little is needed with Akka gRPC's support!

The module would improve a lot with a bit of Scaladoc. Generating Scaladoc fails for me with the generated code being the culprit. Didn't dig into that, but it may be because of multiple older Protobuf jars on the Alpakka classpath.

The generated code is created in c.g.pubsub.v1 for Java and c.g.pubsub.v1.pubsub for Scala. Is there a way to control that? Make it javadsl and scaladsl?

Sprinkle a few @ApiMayChange over this as it depends on the 0.2 version of Akka gRPC.

ennru · 2018-08-31T13:33:29Z

build.sbt

@@ -11,6 +11,7 @@ lazy val modules: Seq[ProjectReference] = Seq(
  geode,
  googleCloudPubSub,
  googleFcm,
+  googleCloudPubSubGrpc,


Something with this sort algorithm looks broken.

ennru · 2018-08-31T13:40:36Z

...cloud-pub-sub-grpc/src/main/scala/akka/stream/alpakka/googlecloud/pubsub/grpc/settings.scala

+    new PubSubConfig(host, port)
+
+  def apply(conf: Config): PubSubConfig = {
+    val c = conf.getConfig("alpakka.google.cloud.pubsub.grpc")


I'd rather make this expect to have selected the config scope already. That allows creating multiple settings from config.

ennru · 2018-08-31T14:04:05Z

google-cloud-pub-sub-grpc/src/test/scala/docs/scaladsl/ExampleUsage.scala

+class ExampleUsage {
+
+  //#init-mat
+  implicit val system = ActorSystem()


Annotate with types, please.

ennru · 2018-08-31T14:05:40Z

google-cloud-pub-sub-grpc/src/test/scala/docs/scaladsl/ExampleUsage.scala

+  //#init-client
+
+  //#publish-single
+  val publishMessage: PubsubMessage =


Make the relevant imports show in the snippet.

ennru · 2018-08-31T14:11:05Z

google-cloud-pub-sub-grpc/src/test/java/docs/javadsl/ExampleUsageJava.java

+    // #publish-fast
+    Source<PubsubMessage, NotUsed> messageSource = Source.single(publishMessage);
+    messageSource
+        .groupedWithin(1000, FiniteDuration.apply(1, "min"))


Use java.time.Duration instead.

ennru · 2018-08-31T14:20:31Z

...rc/main/scala/akka/stream/alpakka/googlecloud/pubsub/grpc/javadsl/GooglePubSubExternal.scala

+          .single(request)
+          .concat(
+            Source
+              .tick(Duration.ZERO, pollInterval, ())


Shouldn't this be pollInterval as initial delay, as well?

Yup. Nice catch.

ennru · 2018-08-31T14:21:46Z

...rc/main/scala/akka/stream/alpakka/googlecloud/pubsub/grpc/javadsl/GooglePubSubExternal.scala

+          .concat(
+            Source
+              .tick(Duration.ZERO, pollInterval, ())
+              .map(_ => subsequentRequest)


Why not use subsequentRequest in tick?

docs/src/main/paradox/google-cloud-pub-sub-grpc.md

ennru · 2018-08-31T14:27:56Z

docs/src/main/paradox/google-cloud-pub-sub-grpc.md

+
+The google cloud pub/sub connector provides a way to connect to google clouds managed pub/sub https://cloud.google.com/pubsub/.
+
+This connector communicates to PubSub via gRPC protocol. The integration between Akka Stream and gRPC is handled by the


Possibly add a callout with Google's explanation of what Pub/Sub offers. Good for googleability.

And a note on the difference to the other Alpakka connector...
And vice versa.

2m · 2018-09-18T20:51:00Z

Pushed commits that address feedback comments.

I had no problem generating api docs when running google-cloud-pub-sub-grpc/doc task. Which task did you use?

I see that scalapb has support for generating scala files under a custom package. However those scalapb specific options need to be defined in the proto files. I did not find a way how to control the package name from the outside when proto files come from somewhere else.

Added ApiMayChange on the clients that come from akka-grpc. I do not think ApiMayChange is needed on the akka stream operator factories, as those do not expose anything from akka-grpc.

raboof · 2018-09-19T15:38:45Z

Looked a bit at generating Scala files into a custom package, but didn't see a way to achieve this. Submitted scalapb/ScalaPB#495 . Of course to have scaladsl/javadsl, we'd need the same for the protoc Java code generation as well.

ennru

Mostly spelling.

ennru · 2018-09-19T15:53:52Z

docs/src/main/paradox/google-cloud-pub-sub-grpc.md

+Further information at the official [Google Cloud documentation website](https://cloud.google.com/pubsub/docs/overview).
+@@@
+
+This connector communicates to Pub/Sub via gRPC protocol. The integration between Akka Stream and gRPC is handled by the


s/via gRPC/via the gRPC/

ennru · 2018-09-19T15:54:43Z

docs/src/main/paradox/google-cloud-pub-sub-grpc.md

+## Configuration
+
+The connector comes with the default settings configured to work with the Google Pub Sub endpoint and uses the default way of
+locating credetianls by looking at the `GOOGLE_APPLICATION_CREDENTIAL` environment variable. Please check


s/credetianls/credentials/

ennru · 2018-09-19T15:56:59Z

docs/src/main/paradox/google-cloud-pub-sub-grpc.md

+Java
+: @@snip (/google-cloud-pub-sub-grpc/src/test/java/docs/javadsl/ExampleUsageJava.java) { #publish-fast }
+
+Finally, to automatically acknowledge messages and send messages to your own sink, once can do the following:


s/once/one/

ennru · 2018-09-19T15:57:58Z

docs/src/main/paradox/google-cloud-pub-sub-grpc.md

+## Subscribing
+
+To receive message from a subscription, first we create a `StreamingPullRequest` with a FQRS of a subscription and
+a deadline for acknowledgemts in seconds. Google requires that only the first `StreamingPullRequest` has the subscription


s/acknowledgemts/acknowledgements/

ennru · 2018-09-19T15:58:53Z

docs/src/main/paradox/google-cloud-pub-sub-grpc.md

+Java
+: @@snip (/google-cloud-pub-sub-grpc/src/test/java/docs/javadsl/ExampleUsageJava.java) { #subscribe }
+
+Here `pollInterval` is the time between `StreamingPullRequest` are sent when there are no messages in the subscription.


s/StreamingPullRequest/StreamingPullRequests/

docs/src/main/paradox/google-cloud-pub-sub-grpc.md

ennru

LGTM

docs/src/main/paradox/google-cloud-pub-sub-grpc.md

ennru · 2018-09-20T07:03:25Z

google-cloud-pub-sub-grpc/src/main/resources/reference.conf

+  port = 443
+
+  # Set to "none" to disable TLS
+  rootCa = "GoogleInternetAuthorityG3.crt"


Wouldn't it be better to put the certificate in a sub-directory, so that it works well even with über-jars?

Why would having the certificate in the top directory not work with uber-jars?

It works. Just thought it might be good to show context to where it is used. Maybe irrelevant.

This reverts commit 200afb8

2m · 2018-09-20T20:46:33Z

I just realized, that there is no akka-grpc for 2.11 released currently (akka/akka-grpc#345). But we need to discuss if we really need it. Maybe Alpakka 1.0 will drop 2.11 and be released for Scala 2.12 and 2.13.

ennru · 2018-09-21T06:19:09Z

Is there a way to build just this connector with Scala 2.12, only?
We will not drop Scala 2.11 support before 2.13 is final.

2m · 2018-09-24T20:59:39Z

Refactored compiled-only tests that are used in the documentation snippets to runnable ones.

* test using Google PubSub emulator and client docker containers

2m · 2018-09-26T19:56:22Z

Everything is green. This is ready to go in!

2m · 2018-09-27T12:46:13Z

Thank you @CremboC for the initial implementation of this. :) I hope you can try it out and see how it works in your use-case.

2m changed the title ~~[WIP] Google Pub Sub connector built on Akka Grpc~~ Google Pub Sub connector built on Akka Grpc Aug 24, 2018

2m mentioned this pull request Aug 24, 2018

Introduce secondary API utilising gRPC instead of HTTP and more configuration options #759

Closed

2m commented Aug 24, 2018

View reviewed changes

...-grpc/src/main/scala/akka/stream/alpakka/googlecloud/pubsub/grpc/impl/AkkaGrpcSettings.scala Show resolved Hide resolved

2m commented Aug 24, 2018

View reviewed changes

build.sbt Show resolved Hide resolved

ennru reviewed Aug 31, 2018

View reviewed changes

2m mentioned this pull request Sep 19, 2018

Google Cloud deserialisation needs better error handling #834

Closed

ennru reviewed Sep 19, 2018

View reviewed changes

ennru approved these changes Sep 20, 2018

View reviewed changes

2m force-pushed the wip-akka-grpc-google-pubsub-2m branch from 5dadb0d to 3c303ad Compare September 20, 2018 14:36

Google Cloud Pub/Sub GRPC implementation PoC

3fdba97

This reverts commit 200afb8

2m force-pushed the wip-akka-grpc-google-pubsub-2m branch 2 times, most recently from f77a82a to b8001b6 Compare September 20, 2018 15:56

2m mentioned this pull request Sep 21, 2018

Cross-build for Scala 2.11 akka/akka-grpc#345

Closed

Move Google PubSub gRPC connector to a separate module

2233374

* test using Google PubSub emulator and client docker containers

2m force-pushed the wip-akka-grpc-google-pubsub-2m branch from 33e26b7 to 2233374 Compare September 26, 2018 17:58

ennru added p:new p:google-cloud-pub-sub-grpc labels Sep 27, 2018

ennru merged commit f26a6b2 into akka:master Sep 27, 2018

ennru added this to the 1.0-M1 milestone Sep 27, 2018

2m deleted the wip-akka-grpc-google-pubsub-2m branch September 27, 2018 12:44

2m restored the wip-akka-grpc-google-pubsub-2m branch September 27, 2018 14:33

2m mentioned this pull request Sep 27, 2018

Google Pub/Sub with akka-grpc re-merge #1245

Merged

2m deleted the wip-akka-grpc-google-pubsub-2m branch September 27, 2018 15:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Google Pub Sub connector built on Akka Grpc #1162

Google Pub Sub connector built on Akka Grpc #1162

2m commented Aug 23, 2018 •

edited

Loading

2m commented Aug 24, 2018

2m commented Aug 24, 2018

ennru left a comment

ennru Aug 31, 2018

ennru Aug 31, 2018

ennru Aug 31, 2018

ennru Aug 31, 2018

ennru Aug 31, 2018

ennru Aug 31, 2018

2m Sep 18, 2018

ennru Aug 31, 2018

ennru Aug 31, 2018

ennru Aug 31, 2018

2m commented Sep 18, 2018

raboof commented Sep 19, 2018

ennru left a comment

ennru Sep 19, 2018

ennru Sep 19, 2018

ennru Sep 19, 2018

ennru Sep 19, 2018

ennru Sep 19, 2018

ennru left a comment

ennru Sep 20, 2018

2m Sep 20, 2018

ennru Sep 20, 2018

2m commented Sep 20, 2018

ennru commented Sep 21, 2018

2m commented Sep 24, 2018

2m commented Sep 26, 2018

2m commented Sep 27, 2018


		The google cloud pub/sub connector provides a way to connect to google clouds managed pub/sub https://cloud.google.com/pubsub/.

		This connector communicates to PubSub via gRPC protocol. The integration between Akka Stream and gRPC is handled by the

Google Pub Sub connector built on Akka Grpc #1162

Google Pub Sub connector built on Akka Grpc #1162

Conversation

2m commented Aug 23, 2018 • edited Loading

2m commented Aug 24, 2018

2m commented Aug 24, 2018

ennru left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2m commented Sep 18, 2018

raboof commented Sep 19, 2018

ennru left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ennru left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2m commented Sep 20, 2018

ennru commented Sep 21, 2018

2m commented Sep 24, 2018

2m commented Sep 26, 2018

2m commented Sep 27, 2018

2m commented Aug 23, 2018 •

edited

Loading