Backport Java-only Pubsub #273
Conversation
|
@dhalperi @davorbonaci |
| DataflowPipelineRunner.StreamingPubsubIOWrite.class, | ||
| new PubsubIOTranslator.WriteTranslator()); | ||
|
|
||
| PubsubUnboundedSink.class, new PubsubIOTranslator.WriteTranslator()); |
There was a problem hiding this comment.
@peihe
Pei, I could do with your help here. Despite my efforts these rewrites never seem to apply when running on Google Cloud Dataflow.
There was a problem hiding this comment.
PubsubIOTranslator.ReadTranslator is a TransformTranslator of PubsubIO.Read.Bound
The type doesn't match for the changes above.
Are you trying to override the new pubsubIO back to nativeIO for Dataflow?
If so, I think you want to remove the two registerTransformTranslator above, and add the overide in DataflowPipelineRunner.
Here is how I did it for BigQuery,Write:
https://github.com/GoogleCloudPlatform/DataflowJavaSDK/pull/205/files
There was a problem hiding this comment.
Thanks, I copied your style and it looks good.
|
Ran beautifully overnight on dataflow + java pubsub (both sink and source). |
|
Confirmed works, porting back to BEAM. |
|
Merged. Woohoo! |
|
Thanks! On Fri, May 20, 2016 at 12:35 AM, Daniel Halperin notifications@github.com
|
apache/beam#120
apache/beam#213
apache/beam#171
apache/beam#346