Skip to content

Harden scheduler stealing and runtime fault containment - #29

Merged
kleeedolinux merged 43 commits into
poplanguage:masterfrom
kleeedolinux:master
Jul 15, 2026
Merged

Harden scheduler stealing and runtime fault containment#29
kleeedolinux merged 43 commits into
poplanguage:masterfrom
kleeedolinux:master

Conversation

@kleeedolinux

@kleeedolinux kleeedolinux commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR completes a resilience pass over Pop Lang’s bounded synchronized M:N
scheduler.

The scheduler is the runtime foundation for typed tasks, async functions,
coroutines, actors, supervision, timers, external events, and future VM
execution. This work strengthens that foundation without introducing a new
source-level scheduling contract or hiding unfinished production-GC work.

The scheduler already provides:

  • separate logical schedulers and bounded operating-system workers;
  • bounded scheduler-local ready queues and a global injection queue;
  • exact Ready, Running, Suspended, Completed, Cancelled, and
    Panicked task states;
  • cooperative per-dispatch work budgets;
  • coalesced wakeups and cancellation;
  • batch work stealing with bounded simultaneous searchers;
  • scheduler-affine and movable tasks;
  • explicit collector-controlled migration;
  • precise ready and suspended task-frame root containers;
  • worker mutator registration and collector epoch participation;
  • a bounded blocking pool;
  • host timers and external-event delivery;
  • deterministic recording, replay, and bounded schedule exploration;
  • structured scheduler telemetry and versioned benchmark workloads; and
  • task-panic and blocking-operation-panic isolation.

This PR closes three concrete scheduler resilience gaps.

Deterministic steal-victim selection

Steal searches previously started with a fixed neighboring scheduler. That
could repeatedly direct idle workers toward the same queue and create avoidable
contention.

Each logical scheduler now maintains a deterministic steal-search round.
Victim selection:

  • derives a pseudo-randomized starting offset from the worker and round;
  • uses no ambient randomness or allocation;
  • never selects the thief itself;
  • visits every peer at most once per search;
  • produces the same order for the same worker and round; and
  • changes the first victim across successive rounds.

The existing bound on simultaneous searching workers remains active.

Bounded migration during steal storms

A ready task could previously migrate repeatedly between queues before any
worker polled it. In an optimized 1,024-task steal-storm run this produced
between 1,881 and 2,048 migrations for only 1,024 ready publications.

A task may now migrate at most once during one ready publication. Migration
eligibility resets only when the task makes real progress and is published as
ready again, including:

  • admission;
  • wake from suspension;
  • cancellation wakeup; or
  • a nonterminal poll that republishes the task.

This prevents queue-to-queue ping-pong without permanently pinning a movable
task. A task can still migrate again during a later ready episode.

The optimized benchmark contract now rejects:

  • more steals than ready operations;
  • disagreement between successful steals and scheduler migrations;
  • repeated examination of a peer within one search;
  • lost or duplicated completions;
  • checksum disagreement; and
  • stale ready-queue entries.

After the change, three optimized 1,024-task samples recorded 1,012–1,017
migrations, compared with 1,881–2,048 before the bound.

Fail-closed runtime-transition handling

Recoverable migration refusal and collector/runtime failure are now distinct.

A normal migration refusal:

  • leaves the task on its current scheduler;
  • preserves its precise frame-root ownership;
  • records a GC-delayed migration; and
  • allows the scheduler to continue.

A runtime-transition failure:

  • propagates through injection selection or stealing;
  • terminates the affected worker loop;
  • closes the complete scheduler;
  • wakes parked workers so shutdown cannot strand them;
  • rejects all later task admission; and
  • remains the typed error returned by shutdown.

Worker registration or WorkerStarted failure now closes the complete
scheduler as well. Previously a startup transition could terminate only one
worker while leaving the scheduler open to admit work for that dead logical
scheduler.

Panics from trusted runtime-transition hooks are caught at the boundary and
converted into the same typed collector-state incident. They are not confused
with an ordinary task panic or recoverable migration refusal.

This preserves the intended failure hierarchy:

  • an ordinary task panic terminates only that task;
  • a blocking operation panic is contained by the blocking pool;
  • neither failure destroys scheduler workers;
  • a trusted runtime/collector integrity failure closes the scheduler globally
    instead of leaving a partially live runtime.

Reproducible park/publication stress

A new seeded stress test exercises the lost-wakeup boundary.

For each fixed replay seed, the test:

  1. waits until every normal worker is observably parked;
  2. concurrently admits tasks from multiple producer threads;
  3. waits for exact scheduler quiescence;
  4. verifies every task completed exactly once;
  5. verifies every terminal task can be released;
  6. verifies local and injection queues drain completely; and
  7. verifies no stale ready entries remain.

Failures report the seed, round, task identity, and scheduler telemetry. This
makes the workload reproducible instead of reporting an unexplained intermittent
timeout.

The complete scheduler suite also passed 25 consecutive stress iterations.

Research basis

The implementation follows the accepted scheduler research rather than copying
another runtime’s public semantics:

  • BEAM/ERTS contributes reduction-style bounded work, per-scheduler queues,
    observable scheduler state, failure isolation, and separate dirty execution
    domains.
  • Go contributes the separation between logical scheduler resources and
    operating-system threads, local/global queues, periodic global polling,
    batch stealing, and throttled searching workers.
  • Tokio contributes bounded local queues, coalesced wakeups, operation budgets,
    randomized victim starts, half-queue stealing, and protection against
    thundering-herd search behavior.
  • Cilk, Chase–Lev, weak-memory work-stealing research, and CHESS motivate batch
    stealing, explicit linearization requirements, and deterministic replay.

The implementation deliberately retains synchronized bounded queues. Replacing
them with a lock-free Chase–Lev-style deque remains an optimization gate
requiring a written linearization/reclamation proof, weak-memory tests, stress
coverage, and benchmark evidence.

Architecture traceability

  • Authorizing architecture section or ADR:
    • architecture/23.1-scheduler-runtime-implementation.md
    • ADR 0068: typed async tasks, actors, and distribution
    • ADR 0072: scheduler mutator and task-root binding
    • ADR 0073: native ABI 2 writable-root coexistence
    • architecture/19-architecture-conformance-and-regression-policy.md
  • New or changed public contract:
    • No new Pop source syntax or source-visible scheduler API.
    • Private runtime steal policy now varies victim order deterministically.
    • A ready publication may migrate at most once before being polled.
    • Runtime-transition failure closes the scheduler and is never treated as
      ordinary migration refusal.
    • Trusted transition panics become typed collector-state incidents.
    • The steal-storm benchmark rejects migration amplification.
  • Architecture documents, examples, or terminology updated:
    • ROADMAP.md records deterministic victim selection, bounded migration,
      fail-closed transition handling, and seeded park/publication stress.
    • Item/Module/Bubble/Package/Workspace and backend-neutral HIR/MIR terminology
      remain unchanged.

Verification

  • Tests were added or updated before implementation where behavior changed.
  • Positive behavior is covered.
  • Negative/rejection boundaries are covered.
  • Convention, consistency, and regression coverage is present where relevant.
  • Cross-backend or differential coverage is present where relevant.
  • cargo fmt --all -- --check
  • cargo check --workspace --all-targets
  • cargo test --workspace --all-targets
  • cargo clippy --workspace --all-targets -- -D warnings

Additional checks run successfully:

  • cargo test --workspace
  • cargo test -p pop-runtime-native
  • cargo test -p pop-runtime-collector --test task_frame_roots
  • cargo test -p pop-architecture-tests
  • 25 consecutive executions of the complete native scheduler test suite
  • optimized eight-worker, 1,024-task steal_storm benchmark
  • git diff --check

The native scheduler suite includes coverage for:

  • exact task-state transitions;
  • ready/suspended frame-root publication and restoration;
  • forced minor and major collection;
  • worker mutator registration and cleanup;
  • wake-during-poll coalescing;
  • concurrent wake/cancellation races;
  • task panic storms;
  • blocking-operation panic isolation;
  • work-budget fairness;
  • batch stealing and migration refusal;
  • transition rejection and transition panic;
  • timers and external events;
  • deterministic recording/replay/exploration;
  • shutdown root cleanup; and
  • seeded parked-worker publication races.

If a check was not run, explain why:

cargo check --workspace --all-targets, the exact --all-targets workspace test
command, and workspace Clippy were not run in this tranche. The full ordinary
workspace test suite, focused runtime/collector suites, architecture suite,
formatter, repeated scheduler stress, and optimized scheduler benchmark passed.

Review notes

  • No dynamic typing, runtime string lookup, broad reflection, or universal-table behavior was introduced.
  • HIR/MIR remain backend-neutral.
  • No generated artifacts, dependency caches, credentials, or editor files are included.
  • This is ready for technical review.

Known limitations and deliberately open gates:

  • This PR hardens the synchronized correctness scheduler; it does not claim the
    production concurrent collector is selectable.
  • The stable native facade still advertises ABI 1.11 and rejects ABI 2.0.
  • Production ABI 2 still needs moving-collector facade selection and complete
    unwind, coroutine, and FFI relocation proofs.
  • Adaptive collector-worker sizing, concurrent card refinement/page
    reclamation, active-stack watermarks, and declared supported-target latency
    gates remain open.
  • The experimental C backend continues to reject async/runtime-managed features
    rather than emitting a synchronous or dynamic fallback.
  • Native LLVM async task-state-machine lowering remains fail-closed until its
    complete scheduler/frame/GC contract is implemented.
  • The untracked user-owned popbook/ directory is not part of this PR.

@kleeedolinux
Document remaining heap bottlenecks
995c655 
List the concrete problems that still prevent the stable native collector from reaching production throughput: cross-workload regressions, detached payload storage, global serialization, repeated layout construction, barrier overhead, ABI 1 relocation limits, and incomplete concurrent integration.

Define observable completion targets for retained-object performance without allowing allocation churn, P99, memory, or correctness regressions.
@kleeedolinux
Define native scheduler runtime design
2a84b50 
Document the scheduler decisions drawn from BEAM, Go, Tokio, and the
work-stealing and deterministic-testing literature. Specify bounded
queues, logical scheduler ownership, fairness, blocking isolation, GC
transitions, and record/replay gates before implementation.
@kleeedolinux
Add bounded native task scheduler
7a13ff0 
Implement the synchronized M:N correctness scheduler with bounded local
and injection queues, cooperative polling and cancellation, wake
coalescing, work stealing, panic containment, deterministic record/replay,
typed collector transitions, and an isolated bounded blocking pool.

Exercise capacity, fairness, wake races, migration refusal, shutdown, and
worker survival after task and blocking-operation failures.
@kleeedolinux
Add scheduler event and resilience support
90ee3d7 
Add bounded host and virtual timers, exact external-event delivery, atomic
task batches, owner-compatible injection selection, and deterministic
resource cleanup. Keep migration disabled until a runtime transition hook
explicitly approves it.

Harden parking and stealing against affinity wake loss, park-gate
self-deadlock, admission contention, stale queue accounting, concurrent
wake/cancellation races, panicking work, and dormant-source shutdown.
@kleeedolinux
Add bounded scheduler exploration
7714b28 
Derive alternative decision prefixes from recorded enabled sets so concurrency failures can be explored and replayed without ambient timing or randomness. Reject invalid and unbounded exploration requests before executing them.
@kleeedolinux
Add versioned scheduler benchmark
96c0782 
Measure the synchronized reference scheduler with bounded, checksum-validated workloads for task control, ready work, injection, stealing, suspension, timers, events, and blocking work. Label every result with its schema, runtime stage, host profile, and exact latency scope so local evidence cannot be mistaken for a production GC or portable performance claim.
@kleeedolinux
Test scheduler panic-storm isolation
284f61d 
Exercise repeated task failures across four logical schedulers, release every retained terminal record, and require affine continuation work to complete on each worker. This guards against a panic storm shrinking or poisoning the normal scheduler pool.
@kleeedolinux
Add scheduler queue and lifecycle telemetry
64084b1 
Track exact current and high-water ready and blocking queues, bounded steal search outcomes, batch sizes, and worker lifecycle events. Publish the facts through the versioned benchmark and a final shutdown snapshot. Keep wake and cancellation readiness accounting inside queue publication so a fast worker cannot leave a phantom ready task after termination.
@kleeedolinux
Define scheduler mutator and task-root binding
ad8087d 
Require every non-running task frame to retain one precise collector-visible root container, including ready frames waiting in queues. Bind normal workers to detached mutator registrations, restore relocated root slots before dispatch, and make scheduler selection atomic with each native runtime operation.
@kleeedolinux
Move scheduler identities into PLRI
0265d42 
Make SchedulerId a backend-neutral runtime-interface identity instead of a collector-owned type. Add a distinct TaskFrameRootId so retained coroutine frames can cross the scheduler and collector boundary without string or integer-based resolution.
@kleeedolinux
Retain precise scheduler task roots
e60b464 
Keep every queued or suspended task frame alive through a bounded collector-owned container. Preserve exact stack-map shape, follow minor relocation through private strong handles, reject foreign ownership, and restore or release roots exactly once through both moving and native stable collector profiles.
@kleeedolinux
Resolve merged architecture identities
48cf17c 
Renumber the later native runtime decisions after the accepted ADR sequence grew through 0069, and update references by semantic owner. Keep one typed Actor and Cluster catalog entry so the public library inventory remains unique and consistent with ADR 0068.
@kleeedolinux
Prepare task root restoration atomically
33bc0d3 
Keep retained task-frame roots live while a compiler-generated frame adapter installs relocated slots. Release the collector container only after explicit completion so a rejected installation can retry without losing the last valid root set.
@kleeedolinux
Bind scheduler tasks to precise roots
99a0772 
Require every scheduler task to publish an exact frame map before
admission and after every nonterminal poll. Restore collector-updated
slots before dispatch so queued and suspended tasks remain precise
managed roots.

Make admission, migration, restoration, and shutdown failure-atomic.
Expose root lifecycle telemetry and keep the deterministic scheduler on
the same explicit contract. Add forced-collection, rejection, rollback,
cleanup, and repeated concurrency coverage.
@kleeedolinux
Bind scheduler workers to collector epochs
25f586f 
Register every normal worker as a detached mutator for its logical
scheduler. Enter managed state only around task polling and route native
ABI operations through the exact thread-local scheduler binding while
holding the serialized runtime lock.

Make managed safe points acknowledge each active epoch at most once,
detach before publishing a task state, and unregister on all worker exit
paths. Preserve precise task roots when dispatch fails and add allocation
ownership, epoch, lifecycle, and failure cleanup coverage.
@kleeedolinux
Check ABI safe-point identity conversion
4b9ec7a 
Convert the bounded churn-test index with a checked conversion instead of
a truncating cast. This keeps strict all-target runtime lint clean and
makes the safe-point identity assumption explicit.
@kleeedolinux
Enforce scheduler dispatch work budgets
3b17187 
Give every task dispatch a nonzero deterministic work budget and require
compiler or trusted-runtime progress to consume explicit units. Reject
zero-unit charges and zero-budget configurations before scheduling.

Requeue an exhausted nonterminal task at the ready tail in both native
and deterministic schedulers. Count every exhaustion and test that a hot
task cannot suspend or run again before an already-ready peer progresses.
@kleeedolinux
Count accepted scheduler migrations
cb51cf5 
Record each collector-approved scheduler ownership transfer separately
from queue steals and GC-delayed migration attempts. This makes migration
observability exact for injection and steal paths without treating a
refused transfer as success.
@kleeedolinux
Measure scheduler ready-to-run delay
b752e78 
Record one semantic work-unit delay sample for every successful task\ndispatch. Use a fixed logarithmic histogram so percentile telemetry\nremains bounded even under long-running scheduler workloads.\n\nKeep deterministic and native schedulers on the same transition-based\nmeasurement contract, and exclude rejected dispatches from the sample\ncount.
@kleeedolinux
Measure scheduler driver delivery delay
cae1080 
Count deterministic and native event/timer readiness inspections and\nmeasure handoff delay on the shared semantic work clock. Start event\ndelay at accepted signal publication and timer delay when the driver\nfirst observes an expired deadline.\n\nStore delivery observations in fixed histograms and keep cancelled or\nrejected sources out of the sample population.
@kleeedolinux
Measure blocking pool shutdown delay
789ad37 
Measure the accepted blocking-work drain and worker-join interval on the\nscheduler observation clock. Record exactly one bounded sample and make\nsubsequent drop cleanup idempotent.\n\nAdvance the clock for blocking completions so shutdown telemetry exposes\nwork drained after admission closes, then mark the scheduler observability\nroadmap item complete.
@kleeedolinux
Extend scheduler benchmark telemetry
2656376 
Advance the checksum-validated benchmark schema to v3 and emit bounded\nscheduler work-delay distributions, budget and migration counters, and\nlabelled operating-system resource observations.\n\nRead Linux memory and context-switch facts from procfs. Report an explicit\nunavailable source with zero values on unsupported or unreadable targets\ninstead of implying measurements that were not collected.
@kleeedolinux
Expand scheduler benchmark workloads
e8ecbc5 
Add checksum-validated local and foreign wake, atomic-turn ping-pong,\nshort-task steal storm, continuous event fairness, and scheduler/GC\ninteraction workloads. Sample process memory while minimal task frames\nare suspended so the explicit million-frame profile measures retained\nstate rather than only its post-release footprint.\n\nKeep logical operations distinct from retry polls in the ping-pong\nprofile and verify every workload completes without stale or duplicated\nwork.
@kleeedolinux
Record million-frame scheduler evidence
5bc1090 
Document the checksum-validated local scale run for one million\nsuspended minimal frames, including elapsed time, memory, context\nswitches, and stale-entry result.\n\nKeep the production benchmark gate open because the accepted ABI 2\nwritable-root transition is still required before collector-coupled\nperformance evidence can be claimed.
@kleeedolinux
Define native ABI 2 root coexistence
0353913 
Keep ABI 1.11 bootstrap safe points distinct from the ABI 2.0\nwritable-root entry. Define fixed capability negotiation, failure-atomic\nslot writeback, and mandatory LLVM post-safe-point alias rewriting.\n\nRequire emitted control-flow verification and forced relocation before\nLLVM or the native facade can advertise production relocation.
@kleeedolinux
Stage writable native ABI 2 roots
6d9ee90 
Publish distinct immutable ABI 1.11 and ABI 2.0 descriptors, add fixed\ncapability negotiation, and implement the failure-atomic writable root\nsafe-point entry.\n\nKeep the stable native facade advertising ABI 1.11 only. This permits\nABI 2 writeback tests without falsely enabling moving execution before\nLLVM reload proof and a production collector composition exist.
@kleeedolinux
Reload writable roots in LLVM lowering
32ef3b3 
Select the distinct ABI 2 safe-point entry for production-profile\nlowering. Spill exact roots before the poll branch, reject failed runtime\npublications, and reload successful writable slots into new SSA values.\n\nCarry relocated values through later instructions, terminators, control-flow\narguments, repeated safe points, and loop backedges while leaving ABI 1\nlowering unchanged.
@kleeedolinux
Force LLVM root relocation in native tests
70dc5b3 
Make the backend poll interval a typed lowering option so conformance\ntests can force every safe point without changing production defaults.\n\nLink optimized ABI 2 output against a deterministic native test runtime\nthat changes each published token and aborts on stale uses. Mutate the\nemitted retain path back to the old SSA token to prove the negative check\nactually fails. Keep the relocation capability disabled until the remaining\ncontrol-flow-wide verifier and transition proofs land.
@kleeedolinux
Preserve ABI 2 roots across divergent control flow
06f634a 
Track every writable managed root in a backend-private function-local\ncell. Initialize cells at definitions and block arguments, publish current\nvalues at safe points, store relocated values on success, and reload uses\nbefore instructions and control-flow edges. This lets LLVM form the needed\nphis without exposing relocation operations in MIR.\n\nReject direct old-token operands during lowering. Add optimized positive and\nnegative execution tests for divergent merges and loop backedges while\nkeeping the production capability disabled for the remaining transition\nproofs.
@kleeedolinux
Validate native ABI 2 before program entry
56ad20d 
Emit an exact pop_rt_supports_abi(2, 0) check in production-profile\nentry wrappers before argument decoding or user code. Route rejection through\nthe closed native trap path and leave ABI 1 entry wrappers unchanged.\n\nExtend the relocation conformance runtime to advertise descriptor 2.0 and\nprove that the stable ABI 1 archive rejects the same executable before normal\nentry.
@kleeedolinux
Add typed async task state machines
2ba242b 
Implement async function and await syntax, exact Task<T> typing, and\nbackend-neutral HIR/MIR suspension state. Preserve async identity in\nreference metadata and verify cold task creation, cancellation cleanup,\nprecise live frames, root maps, and coroutine state identities.\n\nReject async MIR explicitly in backends that do not yet implement task\nexecution so no synchronous or dynamic fallback can change semantics.
@kleeedolinux
Carry exact capture maps in task MIR
f7e1d33 
Derive cold task object maps from verified dispatch, argument, and completion types. Verifying these maps prevents runtime values from redefining precise tracing and keeps retained task completions visible to GC.
@kleeedolinux
Execute cold async tasks in MIR interpreter
0f7a288 
Model created, running, and retained terminal task states without a synchronous fallback. Execute direct and typed indirect task bodies on await, preserve exact live frames, route cancellation and panic exits, and publish managed completions through the verified task object map.
@kleeedolinux
Synchronize accepted async library examples
a2d5365 
Remove the obsolete proposed-syntax disclaimer after ADR 0068 accepted async and await spelling. Keep the catalog regression aligned by requiring the typed async task-group closure.
@kleeedolinux
Vary scheduler steal victim order
673c1ae 
Replace fixed-neighbor steal searches with deterministic per-worker rounds. Each round visits every peer exactly once from a mixed starting offset, reducing repeated victim contention without ambient randomness or weakening bounded search admission.
@kleeedolinux
Bound repeated scheduler task stealing
7083251 
Prevent a movable task from migrating more than once during one ready\npublication. Repeated batch re-stealing could otherwise amplify scheduler\nmigrations before useful polling occurred.\n\nReset migration eligibility only when the task is republished as ready, and\nmake both the benchmark contract and optimized steal-storm workload reject\nmigration amplification or repeated peer scans.
@kleeedolinux
Fail scheduler closed on runtime transition faults
c3c915a 
Keep recoverable migration refusal distinct from collector transition\nfailure. The scheduler previously collapsed both outcomes, and worker startup\nerrors could leave the remaining pool open for admission.\n\nPropagate transition failures through injection and stealing, close every\nworker on lifecycle failure, and convert panics in trusted transition hooks\nto the typed collector-state incident. Ordinary task panics remain isolated\ninside their task boundary.
@kleeedolinux
Stress scheduler park and publication races
35f2a38 
Publish concurrent task batches only after every normal worker is\nobservably parked. This exercises the lost-wakeup boundary repeatedly with\nfixed replay seeds and reports the seed, round, and telemetry on failure.\n\nVerify exact completion, queue drainage, retained terminal release, and zero\nstale ready entries after every round.
@kleeedolinux
Merge branch 'master' into master
f03ffb7
@kleeedolinux
Restore native task ABI test imports
fb8040b 
A merge retained the task ABI assertions but dropped their explicit imports, causing the runtime ABI test target to fail during compilation. Restore the five exported task symbols to the test module scope.
@kleeedolinux
Resolve conflicting async compiler paths
3f4ea3c 
The branch merge combined duplicate async syntax and type nodes with two incompatible MIR designs. This left repeated enum variants, partial call signatures, and a direct Await instruction alongside the accepted cold-task state machine.\n\nKeep the typed TaskCreate and explicit Suspend terminator pipeline, remove the duplicate and premature async-cleanup fragments, and update runtime-contract coverage to follow task creation.
@kleeedolinux
Renumber the experimental eBPF ADR
89b0306 
Assign the eBPF backend decision identity 0071 and remove duplicate shifted copies introduced by the branch merge. Move the later scheduler and native ABI decisions to unused identities and update every semantic reference so the accepted ADR inventory remains unique and all links resolve.
@kleeedolinux
Recognize backend diagnostic codes
7482fcc 
Include the accepted POP7000 backend range in the catalog contract and
keep style-warning assertions limited to the POP6400 entries. This keeps
the diagnostic phase partition aligned with ADR 0071 and the integrated
diagnostics architecture.
@kleeedolinux
kleeedolinux merged commit 8e3a079 into poplanguage:master Jul 15, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kleeedolinux