-
Notifications
You must be signed in to change notification settings - Fork 256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ChannelManager for local channel management and constants to Spout.java #982
Conversation
…t.java. Signed-off-by: Felix Engl <felix.engl@uni-bamberg.de>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks! apart from the use of constants, which is great, the main addition by this PR is the use of the ChannelManager. Can you please explain what it does and why it is needed?
external/urlfrontier/src/main/java/com/digitalpebble/stormcrawler/urlfrontier/Spout.java
Outdated
Show resolved
Hide resolved
external/urlfrontier/src/main/java/com/digitalpebble/stormcrawler/urlfrontier/Spout.java
Show resolved
Hide resolved
external/urlfrontier/src/main/java/com/digitalpebble/stormcrawler/urlfrontier/Spout.java
Show resolved
Hide resolved
A managed channel provides livecycle management of a connection to a server. If we have the Spout and Status on the same JVM we should use the same ManagedChannel in both nodes. https://grpc.github.io/grpc-java/javadoc/io/grpc/ManagedChannel.html (Unless I misunderstood the documentation. But as far as I understood it, best practice would be one instance per Frontier-Host, per JVM) |
Thanks for the explanation. You probably don't want it per Frontier host as multiple instance could be running on that host with different ports. There is an opposite argument for having separate channels, in fact, this is why I recently allowed the StatusUpdaterBolt instances to be a multiple of the number of target frontiers. See https://grpc.io/docs/guides/performance/. Since both the spout and the statusupdater operations are streaming ones, the server uses a single thread to deal with the sequence of events. If you have multiple channels, the Frontier runs the operations in parallel. If you run a large crawl hitting a single Frontier and look at the CPU usage, you will notice that it doesn't get very high - that's because often it won't be using all the threads it can use. See crawler-commons/url-frontier#41 for a related issue. A similar one could be made for the putting of the URLs, so that when dealing with a stream, it would use a pool of threads to deal with each update one by one. |
Thank you, I didn't know that page. (I'll read into this a little bit more, the Grpc protocol seems quite convenient. They even got Kotlin support.) What do you think about writing a comment or long term issue relating to that point? Because at some point this Grpc-problem will be fixed and then we should start reusing channels as described in best practices. |
yes, a comment to that effect would be good |
Signed-off-by: Felix Engl <felix.engl@uni-bamberg.de>
Signed-off-by: Felix Engl <felix.engl@uni-bamberg.de>
Signed-off-by: Felix Engl <felix.engl@uni-bamberg.de>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few minor points, main one being the name of ManagedChannelToolkit but this can be changed later on. Thanks
external/urlfrontier/src/main/java/com/digitalpebble/stormcrawler/urlfrontier/Spout.java
Show resolved
Hide resolved
Signed-off-by: Felix Engl <felix.engl@uni-bamberg.de>
Merged. thanks @FelixEngl |
Add constants for Spout & ChannelManager. Modify pom.xml.
(see #980)
Signed-off-by: Felix Engl felix.engl@uni-bamberg.de
Thanks for contributing to StormCrawler, your efforts are appreciated!
Developer Certificate of Origin
By contributing to StormCrawler, you accept and agree to the following terms and conditions (the Developer Certificate of Origin) for your present and future contributions submitted to StormCrawler.
Please refer to the Developer Certificate of Origin section in
CONTRIBUTING.md
for details.Before opening a PR, please check that:
Thanks!