Skip to content

[Bug] ShardedDaemonProcess sets numShards = numDaemons but the region hashes the entity id, so 100 daemons land on 58 shards and up to 4 permanently co-locate instead of the documented 1-to-1 allocation #951

Description

@pathosDev

Problem

ShardedDaemonProcess sets numShards equal to numDaemons and extractEntityId to String(env.index), on the assumption that daemon i lands on shard i. It does not: ShardRegion.route computes hashShardId(entityId, numShards), an FNV-1a hash modulo the shard count. Hashing N distinct ids into N buckets is the birthday problem — roughly 1/e of the buckets stay empty and the daemons that collide are pinned onto one shard for the lifetime of the cluster.

Measured on the real hashShardId (numbers below are exact, not estimates):

numDaemons distinct shards empty shards worst shard holds
10 8 2 2
50 36 14 3
100 58 42 4
256 188 68 4
1000 532 468 6

The consequence is not cosmetic. A shard is the unit of allocation, so co-located daemons can never be separated: LeastShardAllocationStrategy balances shards, and it sees 58 shards for 100 daemons, four of which carry four daemons each. The documented purpose — "N background reconcilers each owning a slice of work-IDs", "spread evenly across the cluster" — is defeated for 72 of those 100 daemons, and any per-daemon resource budgeting (one DB connection each, one topic-partition consumer each) is off by up to 4× on the unlucky node. Rebalancing cannot repair it; there is no split.

It compounds with the ensureCoordinator ordering defect (ClusterSharding.ts:124/133/404, filed separately in this batch): a coordinator pinned to DEFAULT_NUM_SHARDS = 64 rejects every GetShardHome for an id ≥ 64. With numDaemons = 100, 41 of the 100 daemons hash to a shard id ≥ 64 and therefore never start at all — their wake-ups sit in the region's buffer and the liveness tick re-sends them forever.

Both the class doc and the docs page state the guarantee that does not hold. ShardedDaemonProcess.ts:41-44"each daemon becomes an entity, each entity gets its own shard via a 1-to-1 allocation". docs/src/content/docs/cluster/sharding/sharded-daemon-process.mdx:15-18"gives you exactly N, indexed 0..N-1, spread evenly across the cluster".

Evidence

src/cluster/sharding/ShardedDaemonProcess.ts:61-74
    const startOptions = StartShardingOptions.create<DaemonEnvelope<T>>()
      .withTypeName(`daemon-${resolvedOptions.name}`)
      .withEntityActor(() => new DaemonHost<T>(resolvedOptions.actorFor) as unknown as Actor<DaemonEnvelope<T>>)
      .withExtractEntityId((env) => String(env.index))
      .withExtractEntityMessage((env) => env.body)
      .withNumShards(resolvedOptions.numDaemons)
      .withRememberEntities(true)
      // A daemon is supposed to run continuously, so the node-wide idle sweep
      // must not apply to it: a daemon that only wakes on its own schedule
      // looks idle, and passivating it would both drop it from the
      // remember-entities registry and leave `wakeAll` resurrecting it on
      // every liveness tick.  Explicit, so it beats HOCON as well.
      .withPassivationIdleMs(0)
      .withAllocationStrategy(new LeastShardAllocationStrategy());

The region hashes rather than using the id as the shard:

src/cluster/sharding/ShardRegion.ts:280-286
  private route(
    entityId: string,
    entityMessage: TMessage,
    forwardMessage: RoutableMessage<TMessage>,
    sender: ActorRef | null,
  ): void {
    const shardId = hashShardId(entityId, this.config.numShards);
src/cluster/sharding/ShardAllocator.ts:38-53
/**
 * Helper used by the default ShardRegion to map an entityId to a shard.
 * Uses a stable string hash; callers may supply their own extractShardId.
 */
export function hashShardId(entityId: string, numShards: number): number {
  return Math.abs(stringHash(entityId)) % numShards;
}

function stringHash(text: string): number {
  let hash = 2166136261; // FNV-1a 32-bit basis
  for (let i = 0; i < text.length; i++) {
    hash ^= text.charCodeAt(i);
    hash = Math.imul(hash, 16777619);
  }
  return hash;
}

There is no escape hatch: the JSDoc's "callers may supply their own extractShardId" is false — extractShardId appears nowhere else in src/, and ShardRegionConfig has no such field.

Proposal

The daemon index already is a shard id; it should be used as one rather than hashed back into the same space.

  • Add an optional extractShardId?: (entityId: string) => number to ShardingOptionsType / ShardRegionConfig and honour it in route (and in EntityRef, which calls hashShardId at EntityRef.ts:46). ShardedDaemonProcess then passes (id) => Number(id), restoring the documented 1-to-1 mapping exactly. This also makes the existing JSDoc true.
  • Failing that, the wrapper can allocate numShards large enough that collisions are rare and accept the imbalance — but that trades a correctness guarantee for a probability and should not be the resolution for a feature whose entire contract is "exactly N, spread evenly".
  • Either way, validate numDaemons against the coordinator's effective numShards and fail loudly rather than silently stranding daemons.

Acceptance sketch

  • ShardedDaemonProcess.init with numDaemons = 100 places the 100 daemons on 100 distinct shards.
  • A test asserts a bijection between daemon index and shard id for at least numDaemons ∈ {10, 100, 1000}.
  • Every daemon index receives its wake-up and its preStart runs, for numDaemons > 64.
  • The "1-to-1 allocation" claim in ShardedDaemonProcess.ts and the "spread evenly" claim in the docs page are true or removed — EN + DE.

Reference issues: #193 is the original scoping issue for the feature ("verify implementation completeness") and predates the implementation being reviewed; this is the concrete defect its verify-and-fix task list would surface. #854 covers the SDP configuration keys (keep-alive interval, role) and is unrelated to placement. #679 proposes deleting the dead ShardAllocator surface — the extractShardId hook proposed here lives next to it and should be reconciled in the same pass.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: reproduced by execution against the current tree — the real hashShardId from src/cluster/sharding/ShardAllocator.ts applied to String(0..N-1) with numShards = N, exactly as ShardedDaemonProcess configures it.

numDaemons=  10  distinct shards=   8  empty shards=   2  max daemons on one shard=2  daemons sharing a shard=4    worst shard 5 hosts [6, 8]
numDaemons=  50  distinct shards=  36  empty shards=  14  max daemons on one shard=3  daemons sharing a shard=26   worst shard 21 hosts [26, 39, 44]
numDaemons= 100  distinct shards=  58  empty shards=  42  max daemons on one shard=4  daemons sharing a shard=72   worst shard 95 hosts [22, 48, 75, 84]
numDaemons= 256  distinct shards= 188  empty shards=  68  max daemons on one shard=4  daemons sharing a shard=128  worst shard 176 hosts [32, 87, 225, 250]
numDaemons=1000  distinct shards= 532  empty shards= 468  max daemons on one shard=6  daemons sharing a shard=741  worst shard 595 hosts [22, 242, 259, 295, 363, 378]

numDaemons=100: 41 daemons hash to a shard id >= 64

These are structural counts over a deterministic hash, so they are exactly reproducible and independent of machine load.

Part of the production-readiness review batch — tracked in #913.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: highTop priority — high impact, plan nextproduction-goalBlocks or defines the path to production readiness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions