Skip to content

Conversation

@aromanenko-dev
Copy link
Contributor

Added withTopicFn() to set sink topics dynamically based upon contents of the data.
It's not supported when withEOS() is using since we can't obtain exact number of all partitions in this case.

@rangadi please, take a look

Follow this checklist to help us incorporate your contribution quickly and easily:

  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- --- --- --- ---
Java Build Status Build Status Build Status Build Status Build Status Build Status Build Status Build Status
Python Build Status --- Build Status
Build Status
Build Status --- --- ---

@aromanenko-dev aromanenko-dev changed the title [BEAM-5798] Added "withTopicFn() to set sink topics dynamically [BEAM-5798] Added "withTopicFn()" to set sink topics dynamically Oct 23, 2018
@aromanenko-dev
Copy link
Contributor Author

Retest this please

@lukecwik
Copy link
Member

Just waiting on tests to pass, and will merge on green.

@lukecwik
Copy link
Member

R: @lukecwik

@rangadi
Copy link
Contributor

rangadi commented Oct 30, 2018

I don't think we want to merge this. @aromanenko-dev, could you confirm?

On the mailing list, we agreed that it is better to support writing ProducerRecords, which get all this flexibility and more.

Copy link
Member

@lukecwik lukecwik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marked with request changes based upon thread feedback.

@lukecwik
Copy link
Member

@rangadi Thanks for pointing that out.

@aromanenko-dev aromanenko-dev changed the title [BEAM-5798] Added "withTopicFn()" to set sink topics dynamically WIP: [BEAM-5798] Added "withTopicFn()" to set sink topics dynamically Nov 5, 2018
@aromanenko-dev
Copy link
Contributor Author

@lukecwik Yes, we had a discussion about better solution with @rangadi on user mailing list (see: https://lists.apache.org/thread.html/a89d1d32ecdb50c42271e805cc01a651ee3623b4df97c39baf4f2053@%3Cuser.beam.apache.org%3E), so, I think I'll create another PR for this Jira.
Sorry for inconvenience with that.

@lukecwik
Copy link
Member

lukecwik commented Nov 7, 2018

Closed PR as per authors comment.

@lukecwik lukecwik closed this Nov 7, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants