This repository has been archived by the owner on May 12, 2021. It is now read-only.
METRON-2332 Enable Tuning of the Profiler's Parallelism from Ambari #1574
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When running the Streaming Profiler in Storm, tuning the parallelism of each component requires a user to edit the Flux file at
$METRON_HOME/flux/profiler/remote.yaml
. This change allows a user to tune the parallelism from within Ambari like the other topologies.Changes
profiler.executors
toprofiler.acker.executors
as this more accurately reflects the use of this parameter.Try It Out
Launch the development environment.
Kill the Profiler
Go to Ambari > Metron > Config > Profiler.
Reduce the Profile Period to 1 minute.
Alter the Spout, Splitter, Builder, Kafka Writer, and HBase Writer parallelism values to something other than the default of 1.
Restart the Profiler.
Open the Storm UI and ensure the parallelism values were accepted by Storm.
Test a profile in the REPL according to these instructions.
Deploy that profile to the Streaming Profiler in Storm.
Wait for the Streaming Profiler in Storm to flush and retrieve the measurement from HBase.
Retrieve the measurement from HBase. Prior to this PR, it was not possible to query HBase from the REPL.
Pull Request Checklist