Skip to content

fix: clear the open defect backlog (#334, #348, #312, #313, #357) - #370

Merged
Taure merged 7 commits into
mainfrom
fix/defect-backlog-334-348-312-313-357
Aug 4, 2026
Merged

fix: clear the open defect backlog (#334, #348, #312, #313, #357)#370
Taure merged 7 commits into
mainfrom
fix/defect-backlog-334-348-312-313-357

Conversation

@Taure

@Taure Taure commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Five open defects, none related to each other in code. One PR because they are all small, they touch disjoint files, and splitting them into five would cost five review contexts for one afternoon's backlog. Each is its own commit, so they read independently and revert independently.

Closes #334
Closes #348
Closes #312
Closes #313
Closes #357


#334 - around/3 ranks were offset by the size of the board

count_before/2 walks the ordered_set comparing ETS keys ({-Score, PlayerId}); entries_around/3 handed it the first entry's player id. No key ever equals a player id, so the walk always ran to '$end_of_table' and returned the board size - every rank in the window was numbered from BoardSize + 1.

GET /api/v1/leaderboards/:id/around/:player_id served those ranks straight to clients, so an "around me" panel showed every player a rank worse than they are, disagreeing with the rank the same player got back from submit.

The fix derives the start rank from the queried player's own key and steps back by the number of entries that precede them in the window, which stays correct when the window is truncated at either end of the board.

Tests: 2 new cases (around_ranks_are_absolute, around_at_board_edges), both fail on the old code and pass on the new. The first also cross-checks around/3 against rank/2 for all 10 players. The pre-existing around_query passes either way - it only asserts length(Entries) =:= 5, which is why the defect survived.

#348 - reduction budget on Lua callbacks

The wall-clock timeout bounds latency, not work: a spinning callback is killed at its deadline and then does it again on the next tick, holding a scheduler for its full budget forever. luerl_sandbox:run/3 offers a reduction limit upstream, but only over luerl:do/2; asobi's hot path is luerl:call_function/3 against an already-loaded state, so the poll went into bounded_eval/2, which already spawned and monitored the worker.

What the budget is. asobi_lua.max_reductions_per_ms (default 50,000) times that callback's own wall-clock budget. Per-ms rather than absolute because the surface spans 50 ms (think) to 5000 ms (generate_world) and scaling keeps their relative weights. The rate is measured, not guessed: a 200-entity tick costs ~24k reductions in total, a spin sustains ~284k reductions/ms, so the budget sits ~1000x above a normal tick and cuts a spin at roughly 18% of its wall-clock window. Setting it to 0 restores the previous single-receive behaviour exactly.

What happens when it is exceeded. The same thing that happens on a timeout: the result is discarded, the previous Luerl state is kept, the error is logged, and the tick is skipped. The callers already have a catch-all clause next to their {error, timeout} clause, so no match or zone is ever torn down for this - killing a match because one callback overran would be worse than the spin it prevents. reductions_exhausted is a distinct tag purely so an operator can tell "burned CPU" from "was slow" and from "allocated too much" in a log line.

Sampled every 10 ms, so overshoot is bounded by one interval - the budget bounds sustained CPU, not the instantaneous peak. handle_input/3 is still unbounded by design (ADR 0002: it runs in the calling process, no child).

guides/security-lua-known-limitations.md changes again: the section is no longer "the reduction limit Luerl offers is not applied" but "the CPU bound is sampled, not exact", and now documents the knob, the failure mode, and the two remaining limits. guides/self-hosting.md gains the config row.

Tests: 4 new. reduction_budget_kills_spin_before_deadline_test is the one that fails on the old code (1 failure when the loader change is reverted). Stated plainly:

  • reduction_budget_disabled_falls_back_to_timeout_test is a negative control and passes either way by design.
  • reduction_budget_passes_normal_callback_test also passes either way - it is a regression guard that the default rate does not kill legitimate work, not coverage of the new bound.
  • reduction_exhaustion_renders_distinctly_test covers the new format_reason/1 clause and fails without it.

Three existing tests asserted {error, timeout} on a spinning callback and now accept either bound, because which one fires first is machine-dependent. Their intent (the parent is not wedged; the callback is killed) is unchanged.

#312 - the debug logger attached 24 of 35 events

The attach list was hand-maintained beside the emitters and had drifted. The eleven it missed were - auth_cache aside - exactly the failure and abuse signals someone turns the debug logger on to find: rate limits, anticheat violations, rejected origins, [asobi, error] itself.

The list moves to an exported asobi_telemetry:events/0, so a consumer attaches to the whole surface without restating it. Restating it is how the drift happened, and how opentelemetry_asobi ended up with the same stale copy - that repo should now switch to events/0 (out of scope here, noted in the ADR).

Tests: attach_list_covers_every_emitted_event_test derives the emitted set from the module's own compiled abstract code (every literal event name passed to telemetry:execute/3) and asserts set equality both ways, so a new emitter without a matching entry fails the build. Verified by deleting [asobi, error] from events/0: the test fails. Plus setup_attaches_a_handler_to_every_event_test.

#313 - no world-loop telemetry

asobi_world_ticker, asobi_zone_manager and asobi_zone_spawner emitted nothing. Adds [asobi, world, tick] and the [asobi, zone, opened | closed] pair. asobi_zone_spawner is a pure entity-template registry, not a zone lifecycle owner, and still emits nothing by design.

Settling the shape the issue left open:

  • world/tick measures the full fan-out/fan-in cycle (dispatch to every zone, then the last tick_done), which is the thing that degrades under entity load. Measurements #{duration_ms, max_duration_ms, zone_count, count}, metadata #{world_id}. Sampled - 20 events/s/world raw is too hot for a sink - roughly once a second (tick_sample_interval_ms / tick rate). It carries max_duration_ms for the sampled window because a sampled duration alone hides exactly the spikes worth paging on. entity_count from the issue's sketch is not available at the ticker; zone_count is what it actually fans out to. A tick that never fans in (a zone died mid-tick) is not sampled rather than reported with a fabricated duration.
  • zone/opened / zone/closed are a counter pair, since zones are lazy and a live count is not derivable from world count. closed is gated on the zone still being in the manager's ETS table, because cleanup_zone/2 is reachable more than once for the same coords - otherwise a gauge drifts negative.

The gauge can still drift upward, and I could not honestly fix it here. A world teardown emits no closed at all: asobi_zone_manager does not trap exits, so a supervisor shutdown kills it without running terminate/2, and asobi_world_instance stops the manager before the zone supervisor, so it never processes the zones' DOWNs either. My first attempt put the emit in terminate/2; the test I wrote for it failed, which is how I found this. Making the manager trap exits would close the gap but is a supervision-tree change with its own shutdown-latency cost, so it is documented instead: key the gauge on world_id and drop a world's counters on [asobi, world, finished]. Both the ADR and a test now pin that.

ADR 0005 is updated: new events documented, coords classified unbounded, the count moves 35 -> 38, the gauge caveat recorded, and the two "known gaps" entries struck through. Additive, so a minor bump under the ADR's stability rule.

Tests: 6 new (3 ticker, 3 zone manager). 5 fail when the emitters are reverted. The 6th, manager_shutdown_emits_no_close, passes either way - it pins the documented limitation above so the ADR and the code cannot disagree silently, and is not coverage of the new events.

Worth noting for anyone writing telemetry tests here: eunit's per-test process exiting normal does not kill a start_linked gen_server, so a previous test's ticker keeps emitting into the next test's subscription. The subscriptions filter on world_id / coords for that reason; the sampling test was green, then red, until I found it.

#357 - suites leaving uniquely-constrained rows behind

asobi_iap_SUITE inserted iap_transactions with a fixed txn-1 and never deleted it. The table is unique on (provider, transaction_id), so the first run passed and every later run against the same database got a 409. CI never saw it because CI gets a fresh database - which is why it survived three weeks costing only local developers, and looking like a regression in whatever unrelated work happened to be in flight.

The sweep found a second, quieter cause. erlang:unique_integer([positive]) is unique within one runtime instance, and each rebar3 ct run is a new one, so two runs hand out the same low integers (measured: three fresh VMs, two produced the identical 2242, 2306, 2370). Anything keyed on it was never run-unique despite comments in asobi_storage_SUITE claiming exactly that. Affected fixtures under a unique index:

  • asobi_oauth_SUITE - provider_uid, unique on (provider, provider_uid). This suite was already failing on a first run here, for this reason.
  • asobi_store_SUITE - item_defs.slug.
  • asobi_storage_SUITE - the global row (player_id IS NULL), unique on (collection, key).

Also swept and cleared: group_members, wallets, friendships, cloud_saves, leaderboard_entries, zone_snapshots, players.username - all keyed on a per-run player id, uuid, or the existing random unique_username/1, so none can collide across runs.

Adds asobi_test_helpers:unique_id/1 (random bytes, like the existing unique_username/1). Preferred over an end_per_testcase cleanup because it also lets a suite run concurrently with itself, which cleanup does not. asobi_lua_storage_SUITE had grown a private copy of the same helper; it now shares this one.

Evidence: before, the first run of iap + storage + store + oauth failed 2 tests on a database that already had leftovers. After, three consecutive runs pass 59/59. The two unit tests on unique_id/1 are shape only, and say so in a comment - the cross-run property cannot be asserted from inside one run, and a peer-node test showing unique_integer colliding is flaky in both directions, so I dropped it rather than count it as coverage.


Checks

All green on the final tree: rebar3 fmt --check, xref, dialyzer, ex_doc (zero warnings - it caught two bad references in this branch, both fixed), full eunit (1511 tests, 0 failures), and ct for every suite touched (65 tests, 0 failures).

One process note, since it produced a red herring mid-way: running rebar3 ct and rebar3 eunit concurrently in the same worktree corrupts _build and surfaces as an unrelated undef on asobi_rate_limit_plugin. All the numbers above come from serial runs.

Not done

Taure added 6 commits August 4, 2026 15:17
count_before/2 walks ETS keys ({-Score, PlayerId}); entries_around/3 handed
it the first entry's player id instead. No key ever matched, so the walk ran
to '$end_of_table' and every rank in the window started at BoardSize + 1.

Derive the start rank from the queried player's own key and step back by the
number of entries that precede them in the window, which also stays correct
when the window is truncated at either end of the board.

Closes #334
…#348)

The wall-clock timeout kills a spinning callback at its deadline and the
script simply spins again on the next tick, holding a scheduler for its full
budget forever. luerl_sandbox:run/3 offers a reduction limit upstream but
only over luerl:do/2; asobi's hot path is luerl:call_function/3 against an
already-loaded state, so the poll goes into bounded_eval/2, which already
spawned and monitored the worker.

The budget is asobi_lua.max_reductions_per_ms (50,000) times the callback's
own wall-clock budget, so it scales with a surface that spans 50 ms (think)
to 5000 ms (generate_world). The rate is measured against Luerl: a
200-entity tick costs ~24k reductions while a spin sustains ~284k per ms, so
the budget sits ~1000x above a normal tick and cuts a spin at ~18% of its
window. Rate 0 restores the previous single-receive behaviour exactly.

Overrun is {error, reductions_exhausted} - a distinct tag for the log, but
handled by the same caller clause as a timeout: the result is discarded, the
previous Lua state kept, the tick skipped. Killing a match because one
callback overran would be worse than the spin it prevents.

Closes #348
…rld loop

#312: setup/0's attach list was hand-maintained beside the emitters and had
drifted to 24 of 35 events. The eleven it missed were - auth_cache aside -
exactly the failure and abuse signals someone turns the debug logger on to
find: rate limits, anticheat violations, rejected origins, [asobi, error].
The list moves to an exported events/0 so a consumer attaches to the surface
without restating it (restating it is how the drift happened, and how
opentelemetry_asobi got the same stale copy), and a test derives the emitted
set from the module's own compiled abstract code and asserts the two match.

#313: asobi_world_ticker, asobi_zone_manager and asobi_zone_spawner emitted
nothing, so an operator could not see how long a world tick takes or how many
zones a node carries. Adds [asobi, world, tick] and the
[asobi, zone, opened | closed] pair.

world/tick is sampled - 20 events/s/world raw is too hot for a sink - so it
carries max_duration_ms for the sampled window next to the sampled tick's own
duration_ms, because a sampled duration alone hides the spikes worth paging
on. zone/closed is gated on the zone still being in the manager's table:
cleanup_zone/2 is reachable twice for the same coords, and a gauge built from
opened minus closed would drift negative.

Additive under ADR 0005, which is updated for the new events and the closed
gaps.

Closes #312
asobi_iap_SUITE inserted iap_transactions with a fixed 'txn-1' and never
deleted it. The table is unique on (provider, transaction_id), so the first
run passed and every later run against the same database got a 409. CI never
saw it because CI gets a fresh database, which is why it survived three weeks
costing only local developers - and looking like a regression in whatever
unrelated work happened to be in flight.

Sweeping the other suites for the same shape turned up a second, quieter
cause: erlang:unique_integer/1 is unique within one runtime instance, and
each rebar3 ct run is a new one, so two runs hand out the same low integers.
Anything keyed on it was never run-unique despite comments saying so. That
covers oauth's provider_uid (unique on (provider, provider_uid)), the store's
item_defs.slug, and the global storage row (unique on (collection, key) where
player_id IS NULL). asobi_oauth_SUITE was in fact already failing on a first
run here for exactly this reason.

Adds asobi_test_helpers:unique_id/1 (random bytes, like the existing
unique_username/1) and points those fixtures at it, which also lets the
suites run concurrently with themselves - something an end_per_testcase
cleanup would not. asobi_lua_storage_SUITE had grown a private copy of the
same helper; it now shares this one.

Closes #357
bounded_eval/2 is private, and the ADR tree is not in ex_doc's extras, so
neither resolved as a link.
asobi_zone_manager does not trap exits, so a supervisor shutdown kills it
without running terminate/2, and asobi_world_instance stops the manager
before the zone supervisor, so it never processes the zones' DOWNs either.
An emit in terminate/2 would have been dead code.

Pin the real behaviour in a test and document in ADR 0005 that a live-zone
gauge has to be keyed on world_id and dropped on [asobi, world, finished],
rather than being one global counter pair. Making the manager trap exits to
close the difference is a supervision-tree change with its own
shutdown-latency cost and is deliberately not made here.
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

🟡 Code Coverage — 74.5%

6450 of 8661 lines covered.

@Taure
Taure merged commit abc1d86 into main Aug 4, 2026
15 checks passed
@Taure
Taure deleted the fix/defect-backlog-334-348-312-313-357 branch August 4, 2026 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment