ClusterRouterPool AdjustPoolSize can bypass cluster deployment constraints

[He-Pin]

Summary

AdjustPoolSize sent to a ClusterRouterPool can create a local routee without applying the cluster pool's deployment constraints. In particular, it can create a local child when allowLocalRoutees is false. The same inherited path also does not consult useRoles, totalInstances, or maxInstancesPerNode.

Reproducer

On current main (1fa250d450e18bbebd0374378b6e2a8069473e46), this directional test in ClusterRouterSupervisorSpec exposes the issue:

import pekko.routing.{ AdjustPoolSize, GetRoutees, Routees, RoundRobinPool }

"not create a local routee through AdjustPoolSize when local routees are disabled" in {
  val router = system.actorOf(
    ClusterRouterPool(
      RoundRobinPool(nrOfInstances = 0),
      ClusterRouterPoolSettings(
        totalInstances = 1,
        maxInstancesPerNode = 1,
        allowLocalRoutees = false))
      .props(Props(classOf[KillableActor])))

  router ! AdjustPoolSize(1)
  router ! GetRoutees

  expectMsg(Routees(Vector.empty))
}

Run with JDK 17:

sbt "cluster / Test / testOnly org.apache.pekko.cluster.routing.ClusterRouterSupervisorSpec"

The assertion fails because a local routee named c1 is present:

expected Routees(Vector()), found Routees(Vector(ActorRefRoutee(Actor[pekko://.../c1])))

Cause

ClusterRouterPoolActor inherits RouterPoolActor and delegates unmatched messages, including AdjustPoolSize, to super.receive:

• pekko/cluster/src/main/scala/org/apache/pekko/cluster/routing/ClusterRouterConfig.scala

  Lines 403 to 409 in 1fa250d

  	private[pekko] class ClusterRouterPoolActor(
  	supervisorStrategy: SupervisorStrategy,
  	val settings: ClusterRouterPoolSettings)
  	extends RouterPoolActor(supervisorStrategy)
  	with ClusterRouterActor {
  	override def receive = clusterReceive.orElse(super.receive)

The inherited handler calls pool.newRoutee directly for each requested addition:

• pekko/actor/src/main/scala/org/apache/pekko/routing/RoutedActorCell.scala

  Lines 203 to 214 in 1fa250d

  	override def receive =
  	({
  	case AdjustPoolSize(change: Int) =>
  	if (change > 0) {
  	val newRoutees = Vector.fill(change)(pool.newRoutee(cell.routeeProps, context))
  	cell.addRoutees(newRoutees)
  	} else if (change < 0) {
  	val currentRoutees = cell.router.routees
  	val abandon = currentRoutees.drop(currentRoutees.length + change)
  	cell.removeRoutees(abandon, stopChild = true)
  	}
  	}: Actor.Receive).orElse(super.receive)

For a ClusterRouterPool, newRoutee attaches a local child without checking cluster settings:

• pekko/cluster/src/main/scala/org/apache/pekko/cluster/routing/ClusterRouterConfig.scala

  Lines 342 to 360 in 1fa250d

  	override private[pekko] def newRoutee(routeeProps: Props, context: ActorContext): Routee = {
  	val name = "c" + childNameCounter.incrementAndGet
  	val ref = context
  	.asInstanceOf[ActorCell]
  	.attachChild(local.enrichWithPoolDispatcher(routeeProps, context), name, systemService = false)
  	ActorRefRoutee(ref)
  	}
  	/**
  	* Initial number of routee instances
  	*/
  	override def nrOfInstances(sys: ActorSystem): Int =
  	if (settings.allowLocalRoutees && settings.useRoles.nonEmpty) {
  	if (settings.useRoles.subsetOf(Cluster(sys).selfRoles)) {
  	settings.maxInstancesPerNode
  	} else 0
  	} else if (settings.allowLocalRoutees && settings.useRoles.isEmpty) {
  	settings.maxInstancesPerNode
  	} else 0

In contrast, the normal cluster-pool allocation path calls selectDeploymentTarget, which applies available-node, total-instance, and per-node limits:

• pekko/cluster/src/main/scala/org/apache/pekko/cluster/routing/ClusterRouterConfig.scala

  Lines 411 to 450 in 1fa250d

  	/**
  	* Adds routees based on totalInstances and maxInstancesPerNode settings
  	*/
  	override def addRoutees(): Unit = {
  	@tailrec
  	def doAddRoutees(): Unit = selectDeploymentTarget match {
  	case None => // done
  	case Some(target) =>
  	val routeeProps = cell.routeeProps
  	val deploy =
  	Deploy(config = ConfigFactory.empty(), routerConfig = routeeProps.routerConfig, scope = RemoteScope(target))
  	val routee = pool.newRoutee(routeeProps.withDeploy(deploy), context)
  	// must register each one, since registered routees are used in selectDeploymentTarget
  	cell.addRoutee(routee)
  	// recursion until all created
  	doAddRoutees()
  	}
  	doAddRoutees()
  	}
  	def selectDeploymentTarget: Option[Address] = {
  	val currentRoutees = cell.router.routees
  	val currentNodes = availableNodes
  	if (currentNodes.isEmpty || currentRoutees.size >= settings.totalInstances) {
  	None
  	} else {
  	// find the node with least routees
  	val numberOfRouteesPerNode: Map[Address, Int] = {
  	val nodeMap: Map[Address, Int] = currentNodes.map(_ -> 0).toMap.withDefaultValue(0)
  	currentRoutees.foldLeft(nodeMap) { (acc, x) =>
  	val address = fullAddress(x)
  	acc + (address -> (acc(address) + 1))
  	}
  	}
  	val (address, count) = numberOfRouteesPerNode.minBy(_._2)
  	if (count < settings.maxInstancesPerNode) Some(address) else None
  	}

The documentation also states that max-nr-of-instances-per-node will not be exceeded:

• pekko/docs/src/main/paradox/cluster-routing.md

  Lines 171 to 177 in 1fa250d

  	It is possible to limit the deployment of routees to member nodes tagged with a particular set of roles by
  	specifying `use-roles`.
  	`max-total-nr-of-instances` defines total number of routees in the cluster, but the number of routees
  	per node, `max-nr-of-instances-per-node`, will not be exceeded. By default `max-total-nr-of-instances`
  	is set to a high value (10000) that will result in new routees added to the router when nodes join the cluster.
  	Set it to a lower value if you want to limit total number of routees.

Expected behavior

AdjustPoolSize must not create a routee on a node excluded by allowLocalRoutees or useRoles, and must not silently bypass cluster pool limits.

The exact management semantics need a maintainer decision. Possible approaches include rejecting/ignoring AdjustPoolSize for cluster pools, or interpreting it through the cluster allocation path while preserving totalInstances and maxInstancesPerNode.

Additional context

This was found while investigating #3253. It is separate from the generic pool/resizer-bounds question there: the generic AdjustPoolSize contract is a direct management override, whereas cluster deployment settings describe node eligibility and per-node limits.