Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What is the best way to manage the inputs from more than one bolt? #317

Closed
fedelemantuano opened this issue Oct 1, 2016 · 1 comment
Closed
Labels

Comments

@fedelemantuano
Copy link

Is there a standard method to manage the inputs from more than one bolt?

I have three bolts (A, B, C) that do something. Then there is bolt MERGE that should merge A, B and C emits and output merge result.

I'm using a RAM storage in MERGE bolt, and I waiting tuple from A, B, C. But is this the best way?

@dan-blanchard
Copy link
Member

In the past I've done pretty much what you have when dealing with this issue:

  1. Use a fields grouping on the ID that we will use for joining.
  2. Store the values we've received for a given ID until we have all of the ones we were expecting.
  3. Process the values we stored when we have them all for a given ID.

One tip I can offer is that rather than hard-coding the list of inputs your bolt is expecting, you can actually just set that list to list(self.context['source->stream->grouping'].keys()) if you're using a version of Storm newer than 0.10.0. That will contain a list of all of the bolts that feed into the current one. You can see more details about what kind of info you can pull out of Bolt.context here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants