diff --git a/docs/issue-backlog.md b/docs/issue-backlog.md new file mode 100644 index 0000000..cb505e3 --- /dev/null +++ b/docs/issue-backlog.md @@ -0,0 +1,161 @@ +# GitHub Issue Backlog + +Draft issues ready to be copy-pasted into GitHub. Each entry lists a suggested title, labels, background, concrete scope, and acceptance criteria aligned with the canonical ABI reference implementation. + +--- + +## Issue: Implement canonical async runtime scaffolding +- **Labels:** `enhancement`, `async` + +### Background +The canonical ABI reference (`design/mvp/canonical-abi/definitions.py`) includes a `Store`, `Thread`, and scheduling loop that enable cooperative async execution via `tick()`. The current C++ headers lack any runtime to drive asynchronous component calls. + +### Scope +- Add `Store`, `Thread`, `Task`, and related scheduling types to `cmcpp` mirroring the canonical semantics. +- Implement queueing of pending threads and `tick()` to resume ready work. +- Expose hooks for component functions to register `on_start`/`on_resolve` callbacks and support cancellation tokens on the returned `Call` object. +- Provide doctest (or equivalent) coverage that simulates async invocation and verifies correct scheduling behavior. + +### Acceptance Criteria +1. New runtime types are available in the public API and documented. +2. Asynchronous component calls can progress via repeated `tick()` calls without blocking the host thread. +3. Unit tests demonstrate thread scheduling, `on_start` argument delivery, and cooperative cancellation. +4. Documentation explains how hosts drive async work. + +--- + +## Issue: Complete resource handle lifecycle +- **Labels:** `enhancement`, `abi` + +### Background +`ResourceHandle`, `ResourceType`, and `ComponentInstance.resources` are currently empty shells. The canonical implementation tracks ownership, borrow counts, destructors, and exposes `canon resource.{new,drop,rep}`. + +### Scope +- Flesh out `ResourceHandle` with `own`, `borrow_scope`, and `num_lends` semantics. +- Implement `ResourceType` destructor registration, including async callback support. +- Add `canon resource.new`, `canon resource.drop`, and `canon resource.rep` helper functions that update component tables and trap on misuse. +- Write tests covering own/borrow flows, lend counting, and destructor invocation. + +### Acceptance Criteria +1. Resource creation traps if the table would overflow or if ownership rules are violated. +2. Dropping resources invokes registered destructors exactly once and respects async/sync constraints. +3. Borrowed handles track lend counts and trap on invalid drops or reps. +4. Tests mirror canonical success and trap cases. + +--- + +## Issue: Add waitable, stream, and future infrastructure +- **Labels:** `enhancement`, `async` + +### Background +Canonical ABI defines waitables, waitable-sets, buffers, streams, and futures plus their cancellation behaviors. Our headers only contain empty structs. + +### Scope +- Model waitable and waitable-set state, including registration with the component store. +- Implement buffer management for {stream,future} types, covering readable/writable halves. +- Provide APIs for `canon waitable-set.{new,wait,poll,drop}`, `canon waitable.join`, and `canon {stream,future}.{new,read,write,cancel,drop}`. +- Add tests verifying read/write ordering, cancellation pathways, and polling semantics. + +### Acceptance Criteria +1. Streams and futures can be created, awaited, and canceled via the new APIs. +2. Waitable sets correctly block/unblock pending tasks and surface readiness in tests. +3. Cancellation propagates to queued operations per canonical rules. +4. Documentation describes how hosts integrate these constructs. + +--- + +## Issue: Implement backpressure and task lifecycle management +- **Labels:** `enhancement`, `async` + +### Background +`ComponentInstance` holds flags for `may_leave`, `backpressure`, and call-state tracking but they are unused. Canonical ABI specifies `canon task.{return,cancel}`, `canon yield`, and backpressure counters governing concurrent entry. + +### Scope +- Track outstanding synchronous/async calls and enforce `may_leave` invariants when entering/leaving component code. +- Implement `canon task.return` and `canon task.cancel` helpers wired to pending task queues. +- Support `canon yield` to hand control back to the embedder. +- Ensure backpressure counters gate `Store.invoke` while tasks are paused or pending. + +### Acceptance Criteria +1. Re-entrant sync calls are rejected per canonical rules (tests cover both allowed and disallowed cases). +2. Tasks marked for cancellation resolve promptly with `on_resolve(None)`. +3. `canon yield` transitions tasks to pending and requires a subsequent `tick()` to resume. +4. Backpressure metrics are exposed for debugging and verified in tests. + +--- + +## Issue: Support context locals and error-context APIs +- **Labels:** `enhancement`, `abi` + +### Background +`LiftLowerContext` currently omits instance references and borrow scopes, and `ContextLocalStorage`/`ErrorContext` types are unused. The canonical ABI exposes `canon context.{get,set}` and `canon error-context.{new,debug-message,drop}`. + +### Scope +- Extend `LiftLowerContext` to hold the active `ComponentInstance` and scoped borrow variant. +- Implement context-local storage getters/setters with bounds validation. +- Provide error-context creation, debug-message formatting via the host converter, and drop semantics that respect async callbacks. +- Add tests ensuring invalid indices and double drops trap. + +### Acceptance Criteria +1. Borrowed resources capture their scope and trap when accessed outside it. +2. Context locals persist across lift/lower calls and reset appropriately between tasks. +3. Error-context debug messages surface through the host trap mechanism. +4. Test coverage includes both success and failure paths for each API. + +--- + +## Issue: Finish function flattening utilities +- **Labels:** `enhancement`, `abi` + +### Background +`include/cmcpp/func.hpp` contains commented-out flattening helpers. Canonical ABI requires flattening functions to honor `MAX_FLAT_PARAMS/RESULTS` and spill to heap memory via the provided `realloc`. + +### Scope +- Implement `cmcpp::func::flatten`, `pack_flags_into_int`, and the associated load/store helpers. +- Respect max-flat thresholds and ensure out-params allocate via `LiftLowerContext::opts.realloc`. +- Add regression tests covering large tuples, records, and flag types that exceed the flat limit. +- Compare flattened signatures against outputs from the canonical Python definitions for validation. + +### Acceptance Criteria +1. Flattened core signatures match canonical expectations for representative WIT signatures. +2. Heap-based lowering is invoked automatically when flat limits are exceeded. +3. Flags marshal correctly between bitsets and flat integers. +4. Tests demonstrate both flat and heap pathways. + +--- + +## Issue: Wire canonical options and callbacks through lift/lower +- **Labels:** `enhancement`, `abi` + +### Background +`CanonicalOptions` exposes `post_return`, `callback`, and `sync` but they are currently unused. Canonical ABI requires these flags to control async vs sync paths and post-return cleanup. + +### Scope +- Invoke `post_return` after successful lowerings when provided. +- Enforce `sync` by trapping when async behavior would occur while `sync == true`. +- Invoke `callback` when async continuations schedule additional work. +- Ensure option fields propagate through `InstanceContext::createLiftLowerContext` and task lifecycles. + +### Acceptance Criteria +1. `post_return` is called exactly once per lowering when configured (verified via tests). +2. Async lowering attempts while `sync == true` trap with a descriptive error. +3. Registered callbacks fire for asynchronous continuations and can trigger host-side scheduling. +4. Documentation clarifies option usage and interaction with new runtime pieces. + +--- + +## Issue: Expand docs and tests for canonical runtime features +- **Labels:** `documentation`, `testing` + +### Background +New runtime pieces require supporting documentation and tests. Currently, README lacks guidance and test coverage mirrors only existing functionality. + +### Scope +- Update `README.md` (or add a new guide) summarizing the async runtime, resource management, and waitable APIs. +- Add doctest/ICU-backed unit tests covering the new behavior to `test/main.cpp` (or adjacent files). +- Optionally add a Python cross-check using `ref/component-model/design/mvp/canonical-abi/run_tests.py` for parity. + +### Acceptance Criteria +1. Documentation explains how to configure `InstanceContext`, allocate options, and drive async flows. +2. New tests pass in CI and cover at least one example for each newly implemented feature. +3. The backlog of canonical ABI features is reflected as "Done" within this issue once merged. diff --git a/include/cmcpp.hpp b/include/cmcpp.hpp index dfd5c21..c664e21 100644 --- a/include/cmcpp.hpp +++ b/include/cmcpp.hpp @@ -1,6 +1,7 @@ #ifndef CMCPP_HPP #define CMCPP_HPP +#include #include #include #include diff --git a/include/cmcpp/context.hpp b/include/cmcpp/context.hpp index 8523387..860fc26 100644 --- a/include/cmcpp/context.hpp +++ b/include/cmcpp/context.hpp @@ -4,6 +4,10 @@ #include "traits.hpp" #include +#include +#include +#include +#include #if __has_include() #include #else @@ -53,9 +57,166 @@ namespace cmcpp bool allways_task_return = false; }; - // Runtime State --- - struct ResourceHandle + struct ComponentInstance; + struct HandleElement; + + class LiftLowerContext + { + public: + HostTrap trap; + HostUnicodeConversion convert; + + LiftLowerOptions opts; + ComponentInstance *inst = nullptr; + std::vector lenders; + uint32_t borrow_count = 0; + + LiftLowerContext(const HostTrap &host_trap, const HostUnicodeConversion &conversion, const LiftLowerOptions &options, ComponentInstance *instance = nullptr) + : trap(host_trap), convert(conversion), opts(options), inst(instance) {} + + void track_owning_lend(HandleElement &lending_handle); + void exit_call(); + }; + + inline void trap_if(const LiftLowerContext &cx, bool condition, const char *message = nullptr) noexcept(false) + { + if (condition) + { + const char *msg = message == nullptr ? "Unknown trap" : message; + if (cx.trap) + { + cx.trap(msg); + return; + } + throw std::runtime_error(msg); + } + } + + inline LiftLowerContext make_trap_context(const HostTrap &trap) + { + HostUnicodeConversion convert{}; + LiftLowerOptions opts{}; + return LiftLowerContext(trap, convert, opts); + } + + struct ResourceType + { + ComponentInstance *impl = nullptr; + std::function dtor; + + ResourceType() = default; + + explicit ResourceType(ComponentInstance &instance, std::function destructor = {}) + : impl(&instance), dtor(std::move(destructor)) {} + }; + + struct HandleElement + { + uint32_t rep = 0; + bool own = false; + LiftLowerContext *scope = nullptr; + uint32_t lend_count = 0; + }; + + class HandleTable + { + public: + static constexpr uint32_t MAX_LENGTH = 1u << 30; + + HandleElement &get(uint32_t index, const HostTrap &trap) + { + auto trap_cx = make_trap_context(trap); + trap_if(trap_cx, index >= entries_.size(), "resource index out of bounds"); + auto &slot = entries_[index]; + trap_if(trap_cx, !slot.has_value(), "resource slot empty"); + return slot.value(); + } + + const HandleElement &get(uint32_t index, const HostTrap &trap) const + { + auto trap_cx = make_trap_context(trap); + trap_if(trap_cx, index >= entries_.size(), "resource index out of bounds"); + const auto &slot = entries_[index]; + trap_if(trap_cx, !slot.has_value(), "resource slot empty"); + return slot.value(); + } + + uint32_t add(const HandleElement &element, const HostTrap &trap) + { + auto trap_cx = make_trap_context(trap); + uint32_t index; + if (!free_.empty()) + { + index = free_.back(); + free_.pop_back(); + entries_[index] = element; + } + else + { + trap_if(trap_cx, entries_.size() >= MAX_LENGTH, "resource table overflow"); + entries_.push_back(element); + index = static_cast(entries_.size() - 1); + } + return index; + } + + HandleElement remove(uint32_t index, const HostTrap &trap) + { + HandleElement element = get(index, trap); + entries_[index].reset(); + free_.push_back(index); + return element; + } + + const std::vector> &entries() const + { + return entries_; + } + + const std::vector &free_list() const + { + return free_; + } + + private: + std::vector> entries_{}; + std::vector free_; + HandleTable() { entries_.push_back(std::nullopt); } + }; + + class HandleTables { + public: + HandleElement &get(ResourceType &rt, uint32_t index, const HostTrap &trap) + { + return table(rt).get(index, trap); + } + + const HandleElement &get(const ResourceType &rt, uint32_t index, const HostTrap &trap) const + { + auto it = tables_.find(&rt); + auto trap_cx = make_trap_context(trap); + trap_if(trap_cx, it == tables_.end(), "resource table missing"); + return it->second.get(index, trap); + } + + uint32_t add(ResourceType &rt, const HandleElement &element, const HostTrap &trap) + { + return table(rt).add(element, trap); + } + + HandleElement remove(ResourceType &rt, uint32_t index, const HostTrap &trap) + { + return table(rt).remove(index, trap); + } + + HandleTable &table(ResourceType &rt) + { + return tables_[&rt]; + } + + private: + std::unordered_map tables_; }; struct Waitable @@ -72,16 +233,9 @@ namespace cmcpp struct ComponentInstance { - std::vector resources; - std::vector waitables; - std::vector waitable_sets; - std::vector error_contexts; bool may_leave = true; - bool backpressure = false; - bool calling_sync_export = false; - bool calling_sync_import = false; - // std::vector, Future>> pending_tasks; - bool starting_pending_tasks = false; + bool may_enter = true; + HandleTables handles; }; class ContextLocalStorage @@ -103,25 +257,6 @@ namespace cmcpp } }; - template - class Task - { - public: - using function_type = func_t; - - CanonicalOptions opts; - ComponentInstance inst; - function_type ft; - std::optional supertask; - std::optional> on_return; - std::function(std::future)> on_block; - int num_borrows = 0; - ContextLocalStorage context(); - - Task(CanonicalOptions &opts, ComponentInstance &inst, function_type &ft, std::optional &supertask = std::nullopt, std::optional> &on_return = std::nullopt, std::function(std::future)> &on_block = std::nullopt) - : opts(opts), inst(inst), ft(ft), supertask(supertask), on_return(on_return), on_block(on_block) {} - }; - struct Subtask : Waitable { }; @@ -130,21 +265,61 @@ namespace cmcpp { }; - // Lifting and Lowering Context --- - // template - class LiftLowerContext + inline void LiftLowerContext::track_owning_lend(HandleElement &lending_handle) { - public: - HostTrap trap; - HostUnicodeConversion convert; + trap_if(*this, !lending_handle.own, "lender must own resource"); + lending_handle.lend_count += 1; + lenders.push_back(&lending_handle); + } - LiftLowerOptions opts; - // ComponentInstance inst; - // std::optional, Subtask>> borrow_scope; + inline void LiftLowerContext::exit_call() + { + trap_if(*this, borrow_count != 0, "borrow count mismatch on exit"); + for (auto *handle : lenders) + { + if (handle && handle->lend_count > 0) + { + handle->lend_count -= 1; + } + } + lenders.clear(); + } - LiftLowerContext(const HostTrap &trap, const HostUnicodeConversion &convert, const LiftLowerOptions &opts) - : trap(trap), convert(convert), opts(opts) {} - }; + inline uint32_t canon_resource_new(ComponentInstance &inst, ResourceType &rt, uint32_t rep, const HostTrap &trap) + { + HandleElement element; + element.rep = rep; + element.own = true; + return inst.handles.add(rt, element, trap); + } + + inline void canon_resource_drop(ComponentInstance &inst, ResourceType &rt, uint32_t index, const HostTrap &trap) + { + HandleElement element = inst.handles.remove(rt, index, trap); + auto trap_cx = make_trap_context(trap); + if (element.own) + { + trap_if(trap_cx, element.scope != nullptr, "own handle cannot have borrow scope"); + trap_if(trap_cx, element.lend_count != 0, "resource has outstanding lends"); + trap_if(trap_cx, rt.impl != nullptr && (&inst != rt.impl) && !rt.impl->may_enter, "resource impl may not enter"); + if (rt.dtor) + { + rt.dtor(element.rep); + } + } + else + { + trap_if(trap_cx, element.scope == nullptr, "borrow scope missing"); + trap_if(trap_cx, element.scope->borrow_count == 0, "borrow scope underflow"); + element.scope->borrow_count -= 1; + } + } + + inline uint32_t canon_resource_rep(ComponentInstance &inst, ResourceType &rt, uint32_t index, const HostTrap &trap) + { + const HandleElement &element = inst.handles.get(rt, index, trap); + return element.rep; + } // ---------------------------- diff --git a/include/cmcpp/util.hpp b/include/cmcpp/util.hpp index 1df9b3c..e774ba2 100644 --- a/include/cmcpp/util.hpp +++ b/include/cmcpp/util.hpp @@ -7,14 +7,6 @@ namespace cmcpp { const bool DETERMINISTIC_PROFILE = false; - inline void trap_if(const LiftLowerContext &cx, bool condition, const char *message = nullptr) noexcept(false) - { - if (condition) - { - cx.trap(message == nullptr ? "Unknown trap" : message); - } - } - inline bool_t convert_int_to_bool(uint8_t i) { return i > 0; diff --git a/test/main.cpp b/test/main.cpp index 24fc42a..b39015b 100644 --- a/test/main.cpp +++ b/test/main.cpp @@ -14,6 +14,7 @@ using namespace cmcpp; #include #include #include +#include // #include #define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN @@ -204,6 +205,97 @@ TEST_CASE("Async runtime propagates cancellation") CHECK(thread->completed()); } +TEST_CASE("Resource handle lifecycle mirrors canonical definitions") +{ + ComponentInstance resource_impl; + ComponentInstance inst; + std::vector dtor_calls; + + HostTrap host_trap = [](const char *msg) + { + throw std::runtime_error(msg ? msg : "trap"); + }; + + ResourceType rt(resource_impl, [&](uint32_t rep) + { dtor_calls.push_back(rep); }); + + REQUIRE(inst.may_leave); + REQUIRE(inst.may_enter); + + uint32_t h1 = canon_resource_new(inst, rt, 42, host_trap); + uint32_t h2 = canon_resource_new(inst, rt, 43, host_trap); + + CHECK(h1 == 1); + CHECK(h2 == 2); + + LiftLowerOptions borrow_opts; + HostUnicodeConversion noop_convert = [](void *, uint32_t, const void *, uint32_t, Encoding, Encoding) + { + return std::pair{nullptr, 0}; + }; + LiftLowerContext borrow_scope(host_trap, noop_convert, borrow_opts, &inst); + borrow_scope.borrow_count = 1; + + HandleElement borrowed; + borrowed.rep = 44; + borrowed.own = false; + borrowed.scope = &borrow_scope; + uint32_t h3 = inst.handles.add(rt, borrowed, host_trap); + CHECK(h3 == 3); + + uint32_t h4 = canon_resource_new(inst, rt, 45, host_trap); + CHECK(h4 == 4); + + const auto &table_entries = inst.handles.table(rt).entries(); + CHECK(table_entries.size() == 5); + CHECK_FALSE(table_entries[0].has_value()); + CHECK(table_entries[1].has_value()); + CHECK(table_entries[2].has_value()); + CHECK(table_entries[3].has_value()); + CHECK(table_entries[4].has_value()); + + CHECK(canon_resource_rep(inst, rt, h1, host_trap) == 42); + CHECK(canon_resource_rep(inst, rt, h2, host_trap) == 43); + CHECK(canon_resource_rep(inst, rt, h3, host_trap) == 44); + CHECK(canon_resource_rep(inst, rt, h4, host_trap) == 45); + + dtor_calls.clear(); + canon_resource_drop(inst, rt, h1, host_trap); + CHECK(dtor_calls == std::vector{42}); + auto &table_after_drop = inst.handles.table(rt); + CHECK(table_after_drop.entries()[1].has_value() == false); + CHECK(table_after_drop.free_list().size() == 1); + + uint32_t h5 = canon_resource_new(inst, rt, 46, host_trap); + CHECK(h5 == 1); + CHECK(table_after_drop.entries().size() == 5); + CHECK(table_after_drop.entries()[1].has_value()); + CHECK(dtor_calls == std::vector{42}); + + borrow_scope.borrow_count = 1; + canon_resource_drop(inst, rt, h3, host_trap); + CHECK(dtor_calls == std::vector{42}); + CHECK(borrow_scope.borrow_count == 0); + CHECK(table_after_drop.entries()[3].has_value() == false); + CHECK(table_after_drop.free_list().size() == 1); + + canon_resource_drop(inst, rt, h2, host_trap); + CHECK(dtor_calls == std::vector{42, 43}); + + canon_resource_drop(inst, rt, h4, host_trap); + CHECK(dtor_calls == std::vector{42, 43, 45}); + + canon_resource_drop(inst, rt, h5, host_trap); + CHECK(dtor_calls == std::vector{42, 43, 45, 46}); + + auto &final_table = inst.handles.table(rt); + CHECK(final_table.free_list().size() == 4); + for (size_t i = 1; i < final_table.entries().size(); ++i) + { + CHECK_FALSE(final_table.entries()[i].has_value()); + } +} + TEST_CASE("Boolean") { Heap heap(1024 * 1024);