Skip to content

LinkConfig.toPartitioning emits broadcast channels via zip instead of cross product #4802

@Yicong-Huang

Description

@Yicong-Huang

What happened?

amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/config/LinkConfig.scala::LinkConfig.toPartitioning builds the channel list for the BroadcastPartition arm with fromWorkerIds.zip(toWorkerIds) — i.e. 1:1 pairing — instead of the full cross product. Two issues follow:

  1. The semantics of broadcast are "every sender targets every receiver", which a zip pairing does not encode.
  2. The companion helper ChannelConfig.generateChannelConfigs in the same package emits a full cross product for BroadcastPartition. The two helpers are expected to agree on routing; today they diverge for this arm.

For unequal-length input lists List.zip also silently drops the surplus side, parallel to issue #4799 for OneToOnePartition.

How to reproduce?

import org.apache.texera.amber.core.virtualidentity.ActorVirtualIdentity
import org.apache.texera.amber.core.workflow.BroadcastPartition
import org.apache.texera.amber.engine.architecture.scheduling.config.LinkConfig
import org.apache.texera.amber.engine.architecture.sendsemantics.partitionings.BroadcastPartitioning

val w = (1 to 3).map(i => ActorVirtualIdentity(s"w$i")).toList
val u = (1 to 3).map(i => ActorVirtualIdentity(s"u$i")).toList
val out = LinkConfig.toPartitioning(w, u, BroadcastPartition(), 64)
val bp = out.asInstanceOf[BroadcastPartitioning]
bp.channels.map(c => (c.fromWorkerId.name, c.toWorkerId.name))
// Returns: ((w1,u1), (w2,u2), (w3,u3))
// Broadcast semantics would predict the full 3*3 cross product.

Version

1.1.0-incubating (Pre-release/Master)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions