upstream fixes #5
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…gnments; change red glow color
A screen created in order to manually define an arrangement of partitions assignment among the availabe brokers. This may be preferable than trusting the current automated automated way of generating partitions assignments.
* Created new BVGetViews in ActorModel.scala, which is the "plural" of BVGetView(id: Int). * In class BrokerViewCacheActor created new prived method allBrokerViews(): Seq[BVView] which is used by processActorRequest when treating the BVGetViews query. * In KafkaManager the getBrokersView(clusterName:String) query was added to wrap the BVGetViews query.
The Manual Partitions Assignment view was expanded to include a short version of brokers metrics to ease assignment decisions.
The very useful repeatWithIndex helper located in https://gist.github.com/benoit-ponsero/4484313 has been added to the project.
* conf/routes -> POST route added in /cluster/:c/assignments/manual to
controllers.ReassignPartitions.handleManualAssignment(c:String).
* New action handleManualAssignment(c:String) which takes a manual
reassignment form from request and applies it to
KafkaManager#manualPartitionAssignments(c:String,
assignment:List[(String,List[(Int, List[Int])])]) which takes the
assignment from the submitted form and sends to actor
CMManualPartitionAssignments below KMClusterCommandRequest.
* New actor CMManualPartitionAssignments which take an assignment for a
particular cluster and handles the update to zookeeper so that
runPartitionAssignment may later establish it.
* View topic/manualMultipleAssignments.scala.html was fully adapted to
conform the changes made.
Previously there was a bug that if a broker X had a number of topics and after a reassignment it didn't have any topic anymore, it would keep appearing to have the same topics (partitions replicas) as before because the map brokertopicPartitions wasn't updated for such brokers in the method updateView():Unit in BrokerViewActor.scala. It has now been fixed by performing the right mapping for such brokers.
* Error feedback screen when there's some wrong data in the submitted reassignment form or when validation goes wrong. * Manual reassignment form validation added, this validation verifies if two or more replicas of the same partition are being assigned to the same broker.
* Minor visual improvements in Manual Partitions Assignment screen. * Refactoring inside the method handleManualAssignment(String), which contained to much duplicated code, which was wrapped in the function responseScreen(String, IndexedSeq[ApiError] \/ Unit) inside the same method.
Long story short, after allowing the user to manually set partitions assignment among the brokers, the need to better analyze the brokers arise because one of the motivations to manually assign partitions is to understand some of the broker's low level stats such as CPU load and memory available, and thus improve the distributions of partitions among the brokers by taking into consideration such informations. That being said, kafka's MBeans are not the only ones being used now; java.lang OperatingSystem MBean is used to get Operating System metrics through JMX. * New class OSMetric has been created to keep the Operating System stats. * Class BrokerMetrics now also wraps a OSMetric object. * New KafkaMetrics method getOSMetric(MBeanServerConnection) was created to fetch OS stats through JMX. * New View app/views/common/expandedBrokerMetrics.scala.html was created, based on brokerMetrics.scala.html, to display OS stats and to only display the Mean of other metrics, the 1, 5, or 15 minutes threshold are not displayed in this view. TODO: Implement a filtering, possibly with AngularJS to better filter the metrics and topics seen, and decrease the visual clutter.
Two filters were added to Manual Partitions Assignment: one to filter topics, and the other to filter metrics thus reducing the visual clutter of the page. A new button was also added: Suggest Partition Assignment, which must be implemented yet. When clicking on it, a suggestion of configuration taking into consideration the metrics available must be set on the form.
Two OS-level metrics were removed: Free Physical Memory Size and Free Swap Space Size, both turned out to not be too useful when designing a manual partition assignment. Moreover, I removed from the interface elements meant to be used for suggestions of assignment. They were removed for separation of concerns, they must come in further contributions, not this one.
Both the method allBrokerViews() and the actor request processing for BVGetView had similar body code, this has now been enhanced by wrapping this particular code fragment in produceBViewWithBrokerClusterState(bv: BVView) method
Both actors command process for CMGeneratePartitionAssignments and CMManualPartitionAssignments updates the current broker assignment, previous making use of similar codes, i.e., code duplication, which is now wrapped inside the updateAssignmentInZk method.
Added unit test for CMManualPartitionAssignments in test/kafka/manager/TestClusterManagerActor.scala
Manual partition assignment
add check for topics undergoing reassignment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@abiletskyi can you take a look at these with your branch changes too please