diff --git a/.github/instructions/core-karya-rbs.instructions.md b/.github/instructions/core-karya-rbs.instructions.md index f27b8d5a..8a8c520e 100644 --- a/.github/instructions/core-karya-rbs.instructions.md +++ b/.github/instructions/core-karya-rbs.instructions.md @@ -37,6 +37,18 @@ RBS in this repository is a correctness contract. `private`, mirror that visibility change in the same RBS patch. - Match argument names, keyword names, optionality, return types, and nested module/class structure. +- Match the full accepted input surface before normalization. If Ruby accepts + aliases, alternative casing, delimiter variants, or `nil` before rejecting or + normalizing, model that accepted input in the RBS instead of only the + normalized canonical value. +- Do not collapse shared contracts to one concrete implementation's keyword + surface. Shared interfaces should model only the common contract they truly + guarantee, not adapter-local boot options copied from the first + implementation. +- Do not use broad keyword maps to dodge exactness. If Ruby requires specific + keys, rejects unknown keys, or distinguishes one optional keyword from + another, encode that explicitly instead of using a generic catch-all keyword + shape. - Remove stale entries for deleted methods, constants, and modules. - Do not use `untyped`, `any`, or other generic escape hatches where a concrete type is knowable. diff --git a/.github/instructions/review.instructions.md b/.github/instructions/review.instructions.md index b098558b..38f66188 100644 --- a/.github/instructions/review.instructions.md +++ b/.github/instructions/review.instructions.md @@ -207,6 +207,16 @@ For any Ruby change that has a mirrored file under `sig/`: - **Required:** Treat spec-layout drift as a review concern when extracted owner-local files leave all direct behavior buried only in a monolithic owner spec. +- **Required:** Mirror the accepted input surface, not just the normalized + output surface. If Ruby accepts aliases, mixed casing, delimiter variants, or + broader nilability before normalization, the RBS input type must reflect + that same accepted set. +- **Required:** Flag signatures that overfit one implementation while claiming + to model a shared contract. If a shared base/module type is narrowed to one + concrete adapter's keyword set or return posture, treat that as contract + drift even when the current implementation still passes tests. +- **Required:** Flag generic keyword or catch-all type shapes that hide runtime + rules about required keys, rejected keys, or known option names. ## Architecture Guidelines Review diff --git a/core/karya/lib/karya.rb b/core/karya/lib/karya.rb index 76eb2cf5..0f42c668 100644 --- a/core/karya/lib/karya.rb +++ b/core/karya/lib/karya.rb @@ -20,6 +20,7 @@ require_relative 'karya/outbound_events' require_relative 'karya/reservation' require_relative 'karya/queue_store' +require_relative 'karya/backend' require_relative 'karya/workflow' require_relative 'karya/constant_resolver' require_relative 'karya/worker' diff --git a/core/karya/lib/karya/backend.rb b/core/karya/lib/karya/backend.rb new file mode 100644 index 00000000..dbb79ba2 --- /dev/null +++ b/core/karya/lib/karya/backend.rb @@ -0,0 +1,17 @@ +# frozen_string_literal: true + +# Copyright Codevedas Inc. 2025-present +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +module Karya + # Raised when backend configuration is invalid. + class InvalidBackendConfigurationError < Error; end + + # Namespace for backend interface and lifecycle contracts. + module Backend + autoload :Base, 'karya/backend/base' + autoload :InMemory, 'karya/backend/in_memory' + end +end diff --git a/core/karya/lib/karya/backend/base.rb b/core/karya/lib/karya/backend/base.rb new file mode 100644 index 00000000..06aa1cae --- /dev/null +++ b/core/karya/lib/karya/backend/base.rb @@ -0,0 +1,31 @@ +# frozen_string_literal: true + +# Copyright Codevedas Inc. 2025-present +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +module Karya + module Backend + # Shared backend contract above the queue-store persistence API. + module Base + def identifier + raise NotImplementedError, "#{self.class} must implement ##{__method__}" + end + + def build_queue_store + raise NotImplementedError, "#{self.class} must implement ##{__method__}" + end + + def before_start(queue_store:) + _queue_store = queue_store + nil + end + + def after_stop(queue_store:) + _queue_store = queue_store + nil + end + end + end +end diff --git a/core/karya/lib/karya/backend/in_memory.rb b/core/karya/lib/karya/backend/in_memory.rb new file mode 100644 index 00000000..7c633c4f --- /dev/null +++ b/core/karya/lib/karya/backend/in_memory.rb @@ -0,0 +1,56 @@ +# frozen_string_literal: true + +# Copyright Codevedas Inc. 2025-present +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +require_relative '../base' +require_relative '../backend' +require_relative '../queue_store/in_memory' + +module Karya + module Backend + # Quick-start backend wrapper around the single-process reference queue store. + class InMemory + include Base + + UNSET = Object.new.freeze + private_constant :UNSET + + def initialize(queue_store_class: QueueStore::InMemory) + @identifier = 'in_memory' + @queue_store_class = queue_store_class + end + + attr_reader :identifier + + def build_queue_store( + token_generator: UNSET, + expired_tombstone_limit: UNSET, + completed_batch_retention_limit: UNSET, + max_batch_size: UNSET, + policy_set: UNSET, + circuit_breaker_policy_set: UNSET, + fairness_policy: UNSET + ) + queue_store = queue_store_class.new(**{ + token_generator:, + expired_tombstone_limit:, + completed_batch_retention_limit:, + max_batch_size:, + policy_set:, + circuit_breaker_policy_set:, + fairness_policy: + }.reject { |_name, value| value.equal?(UNSET) }) + return queue_store if queue_store.is_a?(QueueStore::Base) + + raise InvalidBackendConfigurationError, 'queue_store_class must build a Karya::QueueStore::Base' + end + + private + + attr_reader :queue_store_class + end + end +end diff --git a/core/karya/lib/karya/worker_supervisor.rb b/core/karya/lib/karya/worker_supervisor.rb index 1f1f7a16..be4e2ec1 100644 --- a/core/karya/lib/karya/worker_supervisor.rb +++ b/core/karya/lib/karya/worker_supervisor.rb @@ -249,14 +249,19 @@ def with_signal_handlers(shutdown_controller) end def collect_signal_restorers(restorers, shutdown_controller) - register_signal_restorers(restorers, shutdown_controller) restorers << WakeupSignal.register_restorer(WAKEUP_SIGNAL) + register_signal_restorers(restorers, shutdown_controller) end def register_signal_restorers(restorers, shutdown_controller) SIGNALS.each do |signal| - restorers << runtime.subscribe_signal(signal, -> { shutdown_controller.advance }) + restorers << runtime.subscribe_signal(signal, proc do + shutdown_controller.advance + ensure + WakeupSignal.interrupt(WAKEUP_SIGNAL) + end) end + restorers end def cleanup_tracked_children(child_pids) diff --git a/core/karya/lib/karya/worker_supervisor/runtime.rb b/core/karya/lib/karya/worker_supervisor/runtime.rb index 9666d667..bcb59454 100644 --- a/core/karya/lib/karya/worker_supervisor/runtime.rb +++ b/core/karya/lib/karya/worker_supervisor/runtime.rb @@ -29,6 +29,16 @@ def self.default_killer ->(signal, pid) { Process.kill(signal, pid) } end + def self.default_signal_subscriber + lambda do |signal, handler| + previous_handler = Signal.trap(signal) { handler.call } + lambda do + Signal.trap(signal, previous_handler) + nil + end + end + end + def self.normalize_callable(name, value) Primitives::Callable.new(name, value, error_class: InvalidWorkerSupervisorConfigurationError).normalize end @@ -66,34 +76,34 @@ def initialize(**attributes) runtime_class = self.class @forker = runtime_class.normalize_forker( :forker, - runtime_class.resolve_option(attributes, :forker, default: method(:default_forker)) + resolve_runtime_option(attributes, :forker, default: method(:default_forker)) ) @instrumenter = runtime_class.normalize_optional_callable( :instrumenter, - runtime_class.resolve_option(attributes, :instrumenter, default: Karya.instrumenter) + resolve_runtime_option(attributes, :instrumenter, default: Karya.instrumenter) ) @killer = runtime_class.normalize_callable( :killer, - runtime_class.resolve_option(attributes, :killer, default: runtime_class.default_killer) + resolve_runtime_option(attributes, :killer, default: runtime_class.default_killer) ) @logger = validate_logger( - runtime_class.resolve_option(attributes, :logger, default: Karya.logger) + resolve_runtime_option(attributes, :logger, default: Karya.logger) ) @outbound_event_dispatcher = runtime_class.normalize_optional_outbound_event_dispatcher( :outbound_event_dispatcher, - runtime_class.resolve_option(attributes, :outbound_event_dispatcher, default: Karya.outbound_event_dispatcher) + resolve_runtime_option(attributes, :outbound_event_dispatcher, default: Karya.outbound_event_dispatcher) ) @poll_waiter = runtime_class.normalize_callable( :poll_waiter, - runtime_class.resolve_option(attributes, :poll_waiter, default: default_poll_waiter) + resolve_runtime_option(attributes, :poll_waiter, default: default_poll_waiter) ) @signal_subscriber = runtime_class.normalize_optional_callable( :signal_subscriber, - runtime_class.resolve_option(attributes, :signal_subscriber, default: nil) + resolve_runtime_option(attributes, :signal_subscriber, default: runtime_class.default_signal_subscriber) ) @waiter = runtime_class.normalize_callable( :waiter, - runtime_class.resolve_option(attributes, :waiter, default: default_waiter) + resolve_runtime_option(attributes, :waiter, default: default_waiter) ) end @@ -179,6 +189,10 @@ def default_waiter private + def resolve_runtime_option(attributes, key, default:) + self.class.resolve_option(attributes, key, default:) + end + def validate_logger(value) %i[debug info warn error].each do |level| value.public_method(level) diff --git a/core/karya/sig/karya/backend.rbs b/core/karya/sig/karya/backend.rbs new file mode 100644 index 00000000..fde37a3b --- /dev/null +++ b/core/karya/sig/karya/backend.rbs @@ -0,0 +1,12 @@ +# Copyright Codevedas Inc. 2025-present +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +module Karya + class InvalidBackendConfigurationError < Error + end + + module Backend + end +end diff --git a/core/karya/sig/karya/backend/base.rbs b/core/karya/sig/karya/backend/base.rbs new file mode 100644 index 00000000..130e8c26 --- /dev/null +++ b/core/karya/sig/karya/backend/base.rbs @@ -0,0 +1,10 @@ +module Karya + module Backend + module Base + def identifier: () -> String + def build_queue_store: () -> QueueStore::Base + def before_start: (queue_store: QueueStore::Base) -> nil + def after_stop: (queue_store: QueueStore::Base) -> nil + end + end +end diff --git a/core/karya/sig/karya/backend/in_memory.rbs b/core/karya/sig/karya/backend/in_memory.rbs new file mode 100644 index 00000000..31bd66c8 --- /dev/null +++ b/core/karya/sig/karya/backend/in_memory.rbs @@ -0,0 +1,31 @@ +module Karya + module Backend + interface _QueueStoreFactoryInput + def new: ( + ?token_generator: Karya::callable_value?, + ?expired_tombstone_limit: Integer?, + ?completed_batch_retention_limit: Integer?, + ?max_batch_size: Integer?, + ?policy_set: Backpressure::PolicySet?, + ?circuit_breaker_policy_set: CircuitBreaker::PolicySet?, + ?fairness_policy: Fairness::Policy? + ) -> QueueStore::Base + end + + class InMemory + include Base + + def initialize: (?queue_store_class: _QueueStoreFactoryInput) -> void + def identifier: () -> "in_memory" + def build_queue_store: ( + ?token_generator: Karya::callable_value?, + ?expired_tombstone_limit: Integer?, + ?completed_batch_retention_limit: Integer?, + ?max_batch_size: Integer?, + ?policy_set: Backpressure::PolicySet?, + ?circuit_breaker_policy_set: CircuitBreaker::PolicySet?, + ?fairness_policy: Fairness::Policy? + ) -> QueueStore::Base + end + end +end diff --git a/core/karya/sig/karya/internal/runtime_support/shutdown_state.rbs b/core/karya/sig/karya/internal/runtime_support/shutdown_state.rbs index ebe7c002..4573681c 100644 --- a/core/karya/sig/karya/internal/runtime_support/shutdown_state.rbs +++ b/core/karya/sig/karya/internal/runtime_support/shutdown_state.rbs @@ -20,7 +20,7 @@ module Karya def initialize: () -> void - def advance: () -> void + def advance: () -> (:draining | :force_stop | nil) def begin_drain: () -> bool diff --git a/core/karya/sig/karya/worker_supervisor.rbs b/core/karya/sig/karya/worker_supervisor.rbs index 457b07f7..ab6fc5a6 100644 --- a/core/karya/sig/karya/worker_supervisor.rbs +++ b/core/karya/sig/karya/worker_supervisor.rbs @@ -84,6 +84,8 @@ module Karya def with_signal_handlers: (WorkerSupervisor::ShutdownController shutdown_controller) { () -> Integer } -> Integer + def collect_signal_restorers: (Array[^() -> nil] restorers, WorkerSupervisor::ShutdownController shutdown_controller) -> Array[^() -> nil] + def register_signal_restorers: (Array[^() -> nil] restorers, WorkerSupervisor::ShutdownController shutdown_controller) -> Array[^() -> nil] def cleanup_tracked_children: (::Hash[Integer, bool] child_pids) -> nil diff --git a/core/karya/sig/karya/worker_supervisor/runtime.rbs b/core/karya/sig/karya/worker_supervisor/runtime.rbs index 5d2eb77b..d9a2e7de 100644 --- a/core/karya/sig/karya/worker_supervisor/runtime.rbs +++ b/core/karya/sig/karya/worker_supervisor/runtime.rbs @@ -38,6 +38,8 @@ module Karya def self.default_killer: () -> ^(String, Integer) -> Integer + def self.default_signal_subscriber: () -> ^(String, Karya::signal_handler) -> Karya::signal_restorer + def self.normalize_callable: (Symbol name, Karya::callable_value value) -> Karya::callable_value def self.normalize_forker: (Symbol name, Karya::forker value) -> Karya::forker diff --git a/core/karya/spec/karya/backend/base_spec.rb b/core/karya/spec/karya/backend/base_spec.rb new file mode 100644 index 00000000..6abd8804 --- /dev/null +++ b/core/karya/spec/karya/backend/base_spec.rb @@ -0,0 +1,26 @@ +# frozen_string_literal: true + +RSpec.describe Karya::Backend::Base do + subject(:backend) { implementation.new } + + let(:implementation) do + Class.new do + include Karya::Backend::Base + end + end + + it 'requires identifier to be implemented' do + expect { backend.identifier }.to raise_error(NotImplementedError, /implement #identifier/) + end + + it 'requires build_queue_store to be implemented' do + expect { backend.build_queue_store }.to raise_error(NotImplementedError, /implement #build_queue_store/) + end + + it 'provides no-op lifecycle hooks by default' do + queue_store = instance_double(Karya::QueueStore::Base) + + expect(backend.before_start(queue_store:)).to be_nil + expect(backend.after_stop(queue_store:)).to be_nil + end +end diff --git a/core/karya/spec/karya/backend/in_memory_spec.rb b/core/karya/spec/karya/backend/in_memory_spec.rb new file mode 100644 index 00000000..d20292ec --- /dev/null +++ b/core/karya/spec/karya/backend/in_memory_spec.rb @@ -0,0 +1,86 @@ +# frozen_string_literal: true + +require 'open3' +require 'rbconfig' + +RSpec.describe Karya::Backend::InMemory do + subject(:backend) { described_class.new } + + it 'loads as a standalone backend file' do + lib_path = File.expand_path('../../../lib', __dir__) + script = <<~RUBY + require 'karya/backend/in_memory' + puts Karya::Backend::InMemory.new.identifier + RUBY + + stdout, stderr, status = Open3.capture3(RbConfig.ruby, '-I', lib_path, '-e', script) + + expect(status.success?).to be(true), stderr + expect(stdout).to eq("in_memory\n") + end + + it 'exposes the normalized backend identifier' do + expect(backend.identifier).to eq('in_memory') + end + + it 'builds the queue store provider owned by the backend definition' do + queue_store = backend.build_queue_store + + expect(queue_store).to be_a(Karya::QueueStore::InMemory) + end + + it 'allows an injected queue store factory that returns a queue store base' do + queue_store = Karya::QueueStore::InMemory.new + queue_store_factory = Class.new do + define_singleton_method(:new) { |**| queue_store } + end + backend = described_class.new(queue_store_class: queue_store_factory) + + expect(backend.build_queue_store).to be(queue_store) + end + + it 'forwards provided queue-store builder keywords and preserves explicit nil values' do + queue_store = Karya::QueueStore::InMemory.new + captured_options = nil + queue_store_factory = Class.new do + define_singleton_method(:new) do |**options| + captured_options = options + queue_store + end + end + backend = described_class.new(queue_store_class: queue_store_factory) + token_generator = -> { 'token-1' } + + backend.build_queue_store( + token_generator:, + expired_tombstone_limit: 12, + completed_batch_retention_limit: nil, + max_batch_size: 50 + ) + + expect(captured_options).to eq( + token_generator:, + expired_tombstone_limit: 12, + completed_batch_retention_limit: nil, + max_batch_size: 50 + ) + end + + it 'rejects an injected queue store factory that returns a non queue store' do + queue_store_factory = Class.new do + define_singleton_method(:new) { |**| Object.new } + end + backend = described_class.new(queue_store_class: queue_store_factory) + + expect do + backend.build_queue_store + end.to raise_error(Karya::InvalidBackendConfigurationError, /queue_store_class must build a Karya::QueueStore::Base/) + end + + it 'declares no-op lifecycle hooks around queue-store usage' do + queue_store = backend.build_queue_store + + expect(backend.before_start(queue_store:)).to be_nil + expect(backend.after_stop(queue_store:)).to be_nil + end +end diff --git a/core/karya/spec/karya/internal/runtime_support/shutdown_state_spec.rb b/core/karya/spec/karya/internal/runtime_support/shutdown_state_spec.rb index 85e5b454..f3975b16 100644 --- a/core/karya/spec/karya/internal/runtime_support/shutdown_state_spec.rb +++ b/core/karya/spec/karya/internal/runtime_support/shutdown_state_spec.rb @@ -19,4 +19,12 @@ expect(state.force_stop).to be(true) expect(state.force_stop).to be(false) end + + it 'returns the resulting shutdown phase when advancing' do + state = described_class.new + + expect(state.advance).to eq(described_class::DRAINING) + expect(state.advance).to eq(described_class::FORCE_STOP) + expect(state.advance).to be_nil + end end diff --git a/core/karya/spec/karya/worker_supervisor/runtime_spec.rb b/core/karya/spec/karya/worker_supervisor/runtime_spec.rb index 5fb653ef..cf4bdf5d 100644 --- a/core/karya/spec/karya/worker_supervisor/runtime_spec.rb +++ b/core/karya/spec/karya/worker_supervisor/runtime_spec.rb @@ -125,13 +125,15 @@ end describe 'unit tests' do - it 'covers runtime defaults and no-op signal subscriptions' do + it 'covers runtime defaults and default signal subscriptions' do + allow(Signal).to receive(:trap).and_return('DEFAULT') runtime_instance = runtime_class.new - noop = runtime_instance.subscribe_signal('TERM', -> {}) + restorer = runtime_instance.subscribe_signal('TERM', -> {}) pid = runtime_instance.fork_child { 1 } waited_pid, waited_status = runtime_instance.wait_for_child - expect(noop).to respond_to(:call) + expect(restorer).to respond_to(:call) + restorer.call expect(pid).to be_a(Integer) expect(waited_pid).to eq(pid) expect(waited_status.success?).to be(true) @@ -358,6 +360,14 @@ expect(subscriptions.keys).to eq(['TERM']) end + it 'returns the noop subscription when signal subscriptions are explicitly disabled' do + runtime_instance = runtime_class.new(signal_subscriber: nil) + subscription = runtime_instance.subscribe_signal('TERM', -> {}) + + expect(subscription).to respond_to(:call) + expect(subscription.call).to be_nil + end + it 'rejects non-callable signal subscriber restorers' do runtime_instance = runtime_class.new(signal_subscriber: ->(_signal, _handler) { 'DEFAULT' }) diff --git a/core/karya/spec/karya/worker_supervisor_spec.rb b/core/karya/spec/karya/worker_supervisor_spec.rb index 6ff21e64..6ff227ac 100644 --- a/core/karya/spec/karya/worker_supervisor_spec.rb +++ b/core/karya/spec/karya/worker_supervisor_spec.rb @@ -250,6 +250,7 @@ def execute_child_block? expect(supervisor.run).to eq(1) expect(killed_processes).to eq([['TERM', 100], ['KILL', 100]]) + expect(supervisor.runtime_snapshot.phase).to eq('stopped') end it 'supports begin_drain through the public control API while running' do diff --git a/docs/index.md b/docs/index.md index 387c276c..f3958196 100644 --- a/docs/index.md +++ b/docs/index.md @@ -21,7 +21,7 @@ workflows, framework hosts, operator tooling, and governed production rollout. ## What The Platform Includes - a canonical runtime and CLI -- backend adapters with an explicit capability matrix +- backend adapters with a shared backend contract - first-class integrations for plain Ruby, Rails, Sinatra, Roda, and Hanami - ActiveJob compatibility and migration guidance - an optional dashboard addon with shared UI delivery, internal APIs, and Kaal @@ -44,9 +44,7 @@ workflows, framework hosts, operator tooling, and governed production rollout. | Area | Position | | ----------------------------- | ------------------------------------------------------------------------- | -| Default production backend | Postgres | -| Additional supported backends | Redis, MySQL, SQLite | -| Local/dev backend | `InMemory` | +| Backend model | shared backend interface with concrete backend classes | | First-class hosts | plain Ruby, Rails, Sinatra, Roda, Hanami | | Compatibility path | ActiveJob | | Scheduler | Kaal-backed recurring job and cron subsystem | @@ -57,7 +55,7 @@ workflows, framework hosts, operator tooling, and governed production rollout. ## Explore The Docs -- [Architecture](/architecture/): package map, capability model, and how +- [Architecture](/architecture/): package map, product layers, and how the platform fits together - [Getting Started](/getting-started/): setup path for repository work and initial platform evaluation @@ -69,8 +67,8 @@ workflows, framework hosts, operator tooling, and governed production rollout. governed recovery, and backpressure - [Workflows](/workflows/): orchestration, replay, signals, child workflows, and versioning -- [Backends](/backends/): selection guidance, tiers, and capability - matrix +- [Backends](/backends/): backend fit, production boundaries, and adapter + pairing - [Frameworks](/frameworks/): host integrations, ActiveJob, and parity notes - [Dashboard Hosting](/dashboard-hosting/): packaged asset contract and host diff --git a/docs/pages/adoption/goodjob.md b/docs/pages/adoption/goodjob.md index 5b0e6ad2..f063949e 100644 --- a/docs/pages/adoption/goodjob.md +++ b/docs/pages/adoption/goodjob.md @@ -7,7 +7,7 @@ permalink: /adoption/goodjob/ # GoodJob -GoodJob aligns naturally with Karya’s Postgres-first backend posture. +GoodJob aligns naturally with Karya when the target runtime stays SQL-backed. ## Guidance @@ -34,7 +34,8 @@ thinking into Karya’s workflow and operator surfaces. ## Related Concepts -- [Backends](/backends/): Postgres remains the default production path here +- [Backends](/backends/): choose the production backend that matches the + target SQL runtime - [Workflow Basics](/workflows/basics/): this is usually the biggest product expansion - [Cutover And Rollback](/adoption/cutover-rollback/): backend continuity does not diff --git a/docs/pages/architecture.md b/docs/pages/architecture.md index 8e30b99f..6138d023 100644 --- a/docs/pages/architecture.md +++ b/docs/pages/architecture.md @@ -25,7 +25,7 @@ and operator surfaces each have a clear role and are meant to compose cleanly. | `gems/karya-roda` | Roda host integration | | `gems/karya-sinatra` | Sinatra host integration | -## Capability Model +## Product Layers Karya’s product model is built from these layers: @@ -34,8 +34,8 @@ Karya’s product model is built from these layers: dead-letter isolation, and recovery automation. 3. Orchestration: workflows, batch execution, signals, child workflows, approval checkpoints, replay, and version evolution. -4. Persistence: backend adapters, capability reporting, parity guarantees, and - documented exceptions. +4. Persistence: backend adapters, shared backend contracts, and durable runtime + state. 5. Framework integration: host-native mounting, auth/session alignment, adapter pairing, and operator API exposure. 6. Operator experience: dashboard, APIs, CLI, search, drilldowns, activity, diff --git a/docs/pages/backends.md b/docs/pages/backends.md index 314c3a64..59a15306 100644 --- a/docs/pages/backends.md +++ b/docs/pages/backends.md @@ -6,74 +6,65 @@ permalink: /backends/ # Backends -Karya documents backend support through an explicit capability matrix instead of -implying parity from package names alone. - Backend choice shapes durability, operator workflows, scheduling behavior, failure recovery, and the overall fit between Karya and the rest of the stack. This page helps teams make that choice with intent. -## Recommended Default +Karya supports both local development backends and production backends, but +they are not interchangeable. The right choice depends on whether the goal is +fast local iteration, production-like local evaluation, or a durable +production deployment. -Postgres is the default production recommendation for teams that do not already -have a stronger operational preference. +## Backend Positions -For most teams, Postgres is the easiest recommendation to defend: it fits the -broader Karya product model, works well with framework integrations, and gives -operators one durable system of record for execution and orchestration state. +| Backend | Position | Typical Fit | +| ---------- | ----------------------------- | ------------------------------------------------- | +| `InMemory` | Local quick-start backend | Fast examples, tests, and local development | +| SQLite | Local production-like backend | Local SQL-backed evaluation | +| Redis | Production backend | Production use | +| Postgres | Production backend | Production use | +| MySQL | Production backend | Production use | -## Support Matrix +## Support Boundaries -| Backend | Position | Typical Fit | -| ---------- | ----------------------------- | ------------------------------------------------------ | -| Postgres | Default production backend | General-purpose production deployments | -| Redis | Supported production backend | Queue-centric, low-latency operational workloads | -| MySQL | Supported production backend | SQL environments standardized on MySQL | -| SQLite | Supported constrained backend | Embedded, single-node, or lightweight SQL deployments | -| `InMemory` | Local/dev/test backend | Examples, development, tests, and ephemeral evaluation | +- Use `InMemory` for local testing, examples, and development speed. +- Use SQLite when you want a local backend that feels closer to a real SQL + deployment. +- Use Redis, Postgres, or MySQL for production deployments. +- Do not use `InMemory` in production. +- Do not use SQLite in production. ## How To Choose -Choose Postgres when: - -- you want the default production path -- you expect workflows, schedules, audit history, and operator workflows to - matter from the beginning -- you want the broadest fit across hosts and future backlog capability - -Choose Redis when: - -- you are optimizing for queue-centric throughput and low-latency operational - behavior -- your team already runs Redis as a core infrastructure dependency -- you are comfortable reviewing backend-specific caveats as the product surface - expands - -Choose MySQL when: +Choose `Karya::Backend::InMemory` when: -- production standards already center on MySQL -- you want a supported SQL-backed path without introducing Postgres +- you want the fastest possible setup +- you are testing locally +- you are building examples +- you do not need durability across restarts or processes Choose SQLite when: -- the deployment is intentionally small, embedded, or single-node -- the operational tradeoffs are acceptable and clearly understood +- you want a local SQL-backed runtime +- you want a production-like local development path +- you are validating SQL-oriented behavior before moving to a production + backend +- the deployment is intentionally local or single-node + +Choose Redis when: -Choose `InMemory` when: +- you want a production backend +- Redis is the backend you intend to operate -- you are developing locally -- you need quick examples or tests -- durability and multi-process production behavior are not part of the goal +Choose Postgres when: -## Capability Expectations +- you want a production backend +- Postgres is the backend you intend to operate -The documented backend contract covers parity for: +Choose MySQL when: -- job and queue persistence -- workflow and batch state -- schedules and recurring-job state -- audit-relevant and operator-visible history -- capability reporting and intentional parity exceptions +- you want a production backend +- MySQL is the backend you intend to operate ## What Backends Influence @@ -81,15 +72,9 @@ Backend choice affects more than persistence: - how operators reason about queue depth, recovery, and history - how workflows and schedules remain durable across process or host failures -- what parity guarantees can be treated as universal versus backend-specific +- what adapter path and operational posture fit the deployment best - what troubleshooting guidance applies in production -## Unsupported Or Tiered Cases - -When a backend has different scale, durability, or concurrency tradeoffs, the -docs call that out explicitly. `InMemory` is documented as a non-primary backend -for local/dev/test usage rather than a peer production recommendation. - ## Common Scenarios ### General-Purpose Production Platform @@ -98,11 +83,11 @@ for local/dev/test usage rather than a peer production recommendation. host: rails backend: postgres goal: durable jobs, workflows, schedules, and operator visibility -recommendation: default production path +recommendation: production deployment ``` -This is the baseline recommendation for teams adopting Karya as a long-term -platform rather than a narrow queue runner. +This fits teams that want durable orchestration and a long-lived operator +surface. ### Existing Queue-Centric Runtime @@ -110,7 +95,7 @@ platform rather than a narrow queue runner. host: plain-ruby backend: redis goal: high-throughput queue execution with strong operational monitoring -recommendation: supported, with explicit review of parity caveats +recommendation: production deployment ``` This fits teams that already operate Redis heavily and want Karya to align with @@ -120,19 +105,31 @@ that environment. ```text host: plain-ruby -backend: InMemory +backend: Karya::Backend::InMemory goal: fast setup, tests, examples recommendation: local/dev/test only ``` This is for speed and simplicity, not as a production durability story. +### Production-Like Local SQL Evaluation + +```text +host: rails +backend: sqlite +goal: local SQL-backed evaluation before production rollout +recommendation: local only, production-like development path +``` + +This fits teams that want local SQL behavior without treating the local backend +as the production deployment target. + ## Adapter Pairings -- Active Record path: typically Rails with `core/karya-activerecord` -- Sequel path: Hanami, Roda, and Sinatra with `core/karya-sequel` -- plain Ruby: choose the adapter path that matches the selected backend and - persistence style +- Active Record path: choose a backend class that fits the SQL/runtime model +- Sequel path: choose a backend class that fits the host and persistence model +- plain Ruby: compose the backend class and persistence adapter that fit the + selected deployment model ## Related Concepts diff --git a/docs/pages/getting-started.md b/docs/pages/getting-started.md index 9b60a130..7c5edb23 100644 --- a/docs/pages/getting-started.md +++ b/docs/pages/getting-started.md @@ -28,8 +28,7 @@ run the shared verification flow. 1. Read the [Architecture](/architecture/) page for the package map and platform boundaries. -2. Choose a backend using [Backends](/backends/). Postgres is the default - production recommendation. +2. Choose a backend implementation using [Backends](/backends/). 3. Choose a host integration from [Frameworks](/frameworks/). 4. Review [Dashboard Hosting](/dashboard-hosting/) if you want a framework host to include the optional dashboard addon. @@ -54,7 +53,8 @@ bin/prepackage-build ## Recommended Starting Defaults - host: use the framework package that matches your application stack -- backend: start with Postgres unless a documented constraint points elsewhere +- backend: choose the backend implementation that matches the intended runtime + and durability model - scheduler: use the built-in Kaal-backed recurring-job subsystem - operator surface: treat the dashboard, API, and CLI as complementary rather than choosing only one diff --git a/docs/pages/troubleshooting.md b/docs/pages/troubleshooting.md index 122d5813..8f688ed9 100644 --- a/docs/pages/troubleshooting.md +++ b/docs/pages/troubleshooting.md @@ -83,7 +83,7 @@ When work is stuck, backlogged, or repeatedly failing, review: - dead-letter isolation snapshots, recovery action availability, and isolation reasons before retrying or replaying work - workflow replay, checkpoint, or approval state -- backend-specific caveats documented in the support matrix +- backend configuration, durability expectations, and host-adapter fit ### Common Scenario @@ -262,7 +262,7 @@ next move: confirm whether the block is intentional before treating it as a bug - [Dashboard Hosting](/dashboard-hosting/): asset delivery, mount path, and host rendering problems -- [Backends](/backends/): backend fit, parity caveats, and persistence tradeoffs +- [Backends](/backends/): backend fit, production boundaries, and persistence tradeoffs - [Runtime](/runtime/): job, worker, and control-surface behavior - [Reliability](/reliability/): retries, backpressure, uniqueness, and dead-letter flows