Problem
ShardedDaemonProcess sets numShards equal to numDaemons and extractEntityId to String(env.index), on the assumption that daemon i lands on shard i. It does not: ShardRegion.route computes hashShardId(entityId, numShards), an FNV-1a hash modulo the shard count. Hashing N distinct ids into N buckets is the birthday problem — roughly 1/e of the buckets stay empty and the daemons that collide are pinned onto one shard for the lifetime of the cluster.
Measured on the real hashShardId (numbers below are exact, not estimates):
numDaemons |
distinct shards |
empty shards |
worst shard holds |
| 10 |
8 |
2 |
2 |
| 50 |
36 |
14 |
3 |
| 100 |
58 |
42 |
4 |
| 256 |
188 |
68 |
4 |
| 1000 |
532 |
468 |
6 |
The consequence is not cosmetic. A shard is the unit of allocation, so co-located daemons can never be separated: LeastShardAllocationStrategy balances shards, and it sees 58 shards for 100 daemons, four of which carry four daemons each. The documented purpose — "N background reconcilers each owning a slice of work-IDs", "spread evenly across the cluster" — is defeated for 72 of those 100 daemons, and any per-daemon resource budgeting (one DB connection each, one topic-partition consumer each) is off by up to 4× on the unlucky node. Rebalancing cannot repair it; there is no split.
It compounds with the ensureCoordinator ordering defect (ClusterSharding.ts:124/133/404, filed separately in this batch): a coordinator pinned to DEFAULT_NUM_SHARDS = 64 rejects every GetShardHome for an id ≥ 64. With numDaemons = 100, 41 of the 100 daemons hash to a shard id ≥ 64 and therefore never start at all — their wake-ups sit in the region's buffer and the liveness tick re-sends them forever.
Both the class doc and the docs page state the guarantee that does not hold. ShardedDaemonProcess.ts:41-44 — "each daemon becomes an entity, each entity gets its own shard via a 1-to-1 allocation". docs/src/content/docs/cluster/sharding/sharded-daemon-process.mdx:15-18 — "gives you exactly N, indexed 0..N-1, spread evenly across the cluster".
Evidence
src/cluster/sharding/ShardedDaemonProcess.ts:61-74
const startOptions = StartShardingOptions.create<DaemonEnvelope<T>>()
.withTypeName(`daemon-${resolvedOptions.name}`)
.withEntityActor(() => new DaemonHost<T>(resolvedOptions.actorFor) as unknown as Actor<DaemonEnvelope<T>>)
.withExtractEntityId((env) => String(env.index))
.withExtractEntityMessage((env) => env.body)
.withNumShards(resolvedOptions.numDaemons)
.withRememberEntities(true)
// A daemon is supposed to run continuously, so the node-wide idle sweep
// must not apply to it: a daemon that only wakes on its own schedule
// looks idle, and passivating it would both drop it from the
// remember-entities registry and leave `wakeAll` resurrecting it on
// every liveness tick. Explicit, so it beats HOCON as well.
.withPassivationIdleMs(0)
.withAllocationStrategy(new LeastShardAllocationStrategy());
The region hashes rather than using the id as the shard:
src/cluster/sharding/ShardRegion.ts:280-286
private route(
entityId: string,
entityMessage: TMessage,
forwardMessage: RoutableMessage<TMessage>,
sender: ActorRef | null,
): void {
const shardId = hashShardId(entityId, this.config.numShards);
src/cluster/sharding/ShardAllocator.ts:38-53
/**
* Helper used by the default ShardRegion to map an entityId to a shard.
* Uses a stable string hash; callers may supply their own extractShardId.
*/
export function hashShardId(entityId: string, numShards: number): number {
return Math.abs(stringHash(entityId)) % numShards;
}
function stringHash(text: string): number {
let hash = 2166136261; // FNV-1a 32-bit basis
for (let i = 0; i < text.length; i++) {
hash ^= text.charCodeAt(i);
hash = Math.imul(hash, 16777619);
}
return hash;
}
There is no escape hatch: the JSDoc's "callers may supply their own extractShardId" is false — extractShardId appears nowhere else in src/, and ShardRegionConfig has no such field.
Proposal
The daemon index already is a shard id; it should be used as one rather than hashed back into the same space.
- Add an optional
extractShardId?: (entityId: string) => number to ShardingOptionsType / ShardRegionConfig and honour it in route (and in EntityRef, which calls hashShardId at EntityRef.ts:46). ShardedDaemonProcess then passes (id) => Number(id), restoring the documented 1-to-1 mapping exactly. This also makes the existing JSDoc true.
- Failing that, the wrapper can allocate
numShards large enough that collisions are rare and accept the imbalance — but that trades a correctness guarantee for a probability and should not be the resolution for a feature whose entire contract is "exactly N, spread evenly".
- Either way, validate
numDaemons against the coordinator's effective numShards and fail loudly rather than silently stranding daemons.
Acceptance sketch
Reference issues: #193 is the original scoping issue for the feature ("verify implementation completeness") and predates the implementation being reviewed; this is the concrete defect its verify-and-fix task list would surface. #854 covers the SDP configuration keys (keep-alive interval, role) and is unrelated to placement. #679 proposes deleting the dead ShardAllocator surface — the extractShardId hook proposed here lives next to it and should be reconciled in the same pass.
Verification status
Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: reproduced by execution against the current tree — the real hashShardId from src/cluster/sharding/ShardAllocator.ts applied to String(0..N-1) with numShards = N, exactly as ShardedDaemonProcess configures it.
numDaemons= 10 distinct shards= 8 empty shards= 2 max daemons on one shard=2 daemons sharing a shard=4 worst shard 5 hosts [6, 8]
numDaemons= 50 distinct shards= 36 empty shards= 14 max daemons on one shard=3 daemons sharing a shard=26 worst shard 21 hosts [26, 39, 44]
numDaemons= 100 distinct shards= 58 empty shards= 42 max daemons on one shard=4 daemons sharing a shard=72 worst shard 95 hosts [22, 48, 75, 84]
numDaemons= 256 distinct shards= 188 empty shards= 68 max daemons on one shard=4 daemons sharing a shard=128 worst shard 176 hosts [32, 87, 225, 250]
numDaemons=1000 distinct shards= 532 empty shards= 468 max daemons on one shard=6 daemons sharing a shard=741 worst shard 595 hosts [22, 242, 259, 295, 363, 378]
numDaemons=100: 41 daemons hash to a shard id >= 64
These are structural counts over a deterministic hash, so they are exactly reproducible and independent of machine load.
Part of the production-readiness review batch — tracked in #913.
Problem
ShardedDaemonProcesssetsnumShardsequal tonumDaemonsandextractEntityIdtoString(env.index), on the assumption that daemon i lands on shard i. It does not:ShardRegion.routecomputeshashShardId(entityId, numShards), an FNV-1a hash modulo the shard count. Hashing N distinct ids into N buckets is the birthday problem — roughly 1/e of the buckets stay empty and the daemons that collide are pinned onto one shard for the lifetime of the cluster.Measured on the real
hashShardId(numbers below are exact, not estimates):numDaemonsThe consequence is not cosmetic. A shard is the unit of allocation, so co-located daemons can never be separated:
LeastShardAllocationStrategybalances shards, and it sees 58 shards for 100 daemons, four of which carry four daemons each. The documented purpose — "N background reconcilers each owning a slice of work-IDs", "spread evenly across the cluster" — is defeated for 72 of those 100 daemons, and any per-daemon resource budgeting (one DB connection each, one topic-partition consumer each) is off by up to 4× on the unlucky node. Rebalancing cannot repair it; there is no split.It compounds with the
ensureCoordinatorordering defect (ClusterSharding.ts:124/133/404, filed separately in this batch): a coordinator pinned toDEFAULT_NUM_SHARDS = 64rejects everyGetShardHomefor an id ≥ 64. WithnumDaemons = 100, 41 of the 100 daemons hash to a shard id ≥ 64 and therefore never start at all — their wake-ups sit in the region's buffer and the liveness tick re-sends them forever.Both the class doc and the docs page state the guarantee that does not hold.
ShardedDaemonProcess.ts:41-44— "each daemon becomes an entity, each entity gets its own shard via a 1-to-1 allocation".docs/src/content/docs/cluster/sharding/sharded-daemon-process.mdx:15-18— "gives you exactly N, indexed0..N-1, spread evenly across the cluster".Evidence
The region hashes rather than using the id as the shard:
There is no escape hatch: the JSDoc's "callers may supply their own extractShardId" is false —
extractShardIdappears nowhere else insrc/, andShardRegionConfighas no such field.Proposal
The daemon index already is a shard id; it should be used as one rather than hashed back into the same space.
extractShardId?: (entityId: string) => numbertoShardingOptionsType/ShardRegionConfigand honour it inroute(and inEntityRef, which callshashShardIdatEntityRef.ts:46).ShardedDaemonProcessthen passes(id) => Number(id), restoring the documented 1-to-1 mapping exactly. This also makes the existing JSDoc true.numShardslarge enough that collisions are rare and accept the imbalance — but that trades a correctness guarantee for a probability and should not be the resolution for a feature whose entire contract is "exactly N, spread evenly".numDaemonsagainst the coordinator's effectivenumShardsand fail loudly rather than silently stranding daemons.Acceptance sketch
ShardedDaemonProcess.initwithnumDaemons = 100places the 100 daemons on 100 distinct shards.numDaemons ∈ {10, 100, 1000}.preStartruns, fornumDaemons > 64.ShardedDaemonProcess.tsand the "spread evenly" claim in the docs page are true or removed — EN + DE.Reference issues: #193 is the original scoping issue for the feature ("verify implementation completeness") and predates the implementation being reviewed; this is the concrete defect its verify-and-fix task list would surface. #854 covers the SDP configuration keys (keep-alive interval, role) and is unrelated to placement. #679 proposes deleting the dead
ShardAllocatorsurface — theextractShardIdhook proposed here lives next to it and should be reconciled in the same pass.Verification status
Found in the ten-lens production-readiness review of 2026-08-05 (
v0.13.0) and re-verified before filing: reproduced by execution against the current tree — the realhashShardIdfromsrc/cluster/sharding/ShardAllocator.tsapplied toString(0..N-1)withnumShards = N, exactly asShardedDaemonProcessconfigures it.These are structural counts over a deterministic hash, so they are exactly reproducible and independent of machine load.
Part of the production-readiness review batch — tracked in #913.