-
Notifications
You must be signed in to change notification settings - Fork 31
Updating streamers #106
Updating streamers #106
Conversation
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project, in which case you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed, please reply here (e.g.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think Nicole and Dion previously discussed issues with hardcoding it, but maybe having a final constant and just manipulating that might be better.
PTAL |
LGTM once Travis is happy |
Dion, what do you think about having it be public and not final to play around with it, until we determine an algorithm for the optimal shard size? |
I think it's fine private now. Public constants are against our coding style, I'd rather not have people depend on it, and hopefully this is just very temporary while we figure something better out. |
That's understandable - thanks. |
LGTM |
@Careyjmac merge this anytime you like |
Updating streamers
Updating streamers to remove auth issue (jobs that take over an hour should be able to finish now) add aggregators (to be able to view the progress of the job in the dataflow console better) and lower the shard size (for now).