spike: Phase 4 — measure parallel rendering, decide against sharding - #21
Open
TheMeinerLP wants to merge 2 commits into
Open
spike: Phase 4 — measure parallel rendering, decide against sharding#21TheMeinerLP wants to merge 2 commits into
TheMeinerLP wants to merge 2 commits into
Conversation
Phase 4 was gated on a spike into whether two BlueMap processes can render disjoint region sets of the same map into the same map storage without damaging the zoomed-out (lowres) views. No sharding is implemented, deliberately. - Lowres levels are built by aggregation across region boundaries, unlike granularly stored render state. Measured directly: 7 of 24 lowres tiles diverge from a single-pass reference render when two containers render disjoint adjacent region sets concurrently, reproduced identically three times; a sequential control run with no race at all still corrupts 10 of 24 tiles, confirming order dependence rather than a timing artifact. - Full report with setup, raw numbers and an honest assessment of scope: docs/superpowers/spikes/2026-08-09-lowres-sharding-spike.md. The spike's own entrypoint script validates its environment variables with the same single-mention helper function as the Phase 1 runner image. - Decision: scale vertically via render-threads instead of the two-stage alternative (shards render hires only, a final pass builds lowres) — the two-stage route needs a custom runner bound to BlueMap-Core, which is not a stable public API, and no world has yet taken long enough to justify the work. The architecture stays sharding-capable (region list in the bundle manifest, BlueMapMap.spec.shards exists) but shards stays at 1 until a world actually needs it. This branch replaces clean/phase-4-sharding (PR #13), stacked on clean2/phase-3-hosting instead of clean/phase-3-hosting, and squashed on a base where the spike entrypoint's environment-variable validation already uses the helper function, so no follow-up fix commit is needed to clear the scanner finding.
TheMeinerLP
force-pushed
the
clean2/phase-3-hosting
branch
from
August 9, 2026 10:30
cb571e5 to
0a64b11
Compare
TheMeinerLP
force-pushed
the
clean2/phase-4-sharding
branch
from
August 9, 2026 10:30
570d358 to
ddd0c69
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 4 of Apus was gated on a spike. It ran, it came out negative, and this PR records that plus the resulting decision. No sharding is implemented, deliberately.
The question
Can two BlueMap processes render disjoint region sets of the same map into the same map storage without damaging the zoomed-out views?
Render state is stored granularly per tile and chunk, so disjoint regions do not collide there. But lowres levels are built by aggregation — a lowres tile averages colour, height and light across a group of higher-resolution tiles, and those groups span region boundaries. Earlier research called conflicts "unlikely" because lowres tiles are also stored granularly. That reasoning conflates two different things, which is why this was measured rather than assumed.
The measurement
A single-pass reference render of the whole world, against two concurrently running containers rendering disjoint, adjacent region sets into a second bucket.
Granular storage prevents corruption of individual tiles. It does not prevent two shards overwriting the same aggregated lowres tile.
Full report with setup, raw numbers and an honest assessment of what the experiment does and does not cover:
docs/superpowers/spikes/2026-08-09-lowres-sharding-spike.md.The decision
The spec offered two fallbacks. This picks the second — vertical scaling via
render-threads— over the two-stage alternative (shards render hires only, a final pass builds lowres):The architecture stays sharding-capable — the region list is in the bundle manifest,
BlueMapMap.spec.shardsexists. If a world ever does take too long, the two-stage approach is what to evaluate, and this spike is the ground to build on. Until thenshardsstays at 1.Limits of the experiment, as stated in the report
Small test world,
render-maskused to split rather than thescheduleMapUpdateTaskAPI a real implementation would use, and only one split topology tested.