Skip to content

feat: rework inmem resource state implementation - #674

Open
smira wants to merge 1 commit into
cosi-project:mainfrom
smira:feat/shared-watch-buffer
Open

feat: rework inmem resource state implementation#674
smira wants to merge 1 commit into
cosi-project:mainfrom
smira:feat/shared-watch-buffer

Conversation

@smira

@smira smira commented Aug 4, 2026

Copy link
Copy Markdown
Member

This bring inmem closer to other state implementations:

  • inmem.NewState() is now namespaced inside, not an instance per namespace
  • one inmem state instance has a shared event buffer (stream) for all resource types and namespaces

This should provide better memory usage for any inmem user, and also it builds the foundation to build a shared, consistent watch API for the controller runtime (not in this PR).

Copilot AI lite review requested due to automatic review settings August 4, 2026 18:17
@github-project-automation github-project-automation Bot moved this to To Do in Planning Aug 4, 2026
@talos-bot talos-bot moved this from To Do to In Review in Planning Aug 4, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reworks the in-memory (inmem) state implementation to be a single shared state across namespaces/resource types, backed by a shared circular event buffer, bringing it closer to the behavior/shape of other state implementations and enabling more consistent watch semantics going forward.

Changes:

  • Switch inmem.NewState() to return a single state instance (no longer per-namespace builders) and introduce a shared event buffer across all collections.
  • Introduce new internal collection, eventbuffer, and errs packages to modularize storage/watch/error behavior.
  • Update BoltDB backing store and tests to support the new “store is multi-namespace” BackingStore interface and simplify state setup across the repo.

Reviewed changes

Copilot reviewed 41 out of 43 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
pkg/state/wrap_test.go Updates conformance test setup to use the new inmem.NewState() API.
pkg/state/teardowner_test.go Updates teardown wrapper tests to use inmem.NewState() directly.
pkg/state/teardown_and_destroyer_test.go Updates teardown+destroy wrapper tests to use inmem.NewState() directly.
pkg/state/registry/resource_test.go Updates registry test setup to use inmem.NewState().
pkg/state/registry/namespace_test.go Updates registry test setup to use inmem.NewState().
pkg/state/protobuf/protobuf_test.go Updates protobuf state server/client tests to use inmem.NewState().
pkg/state/protobuf/client/client_test.go Updates protobuf client test to use inmem.NewState().
pkg/state/owned/owned_test.go Updates owned-state tests to use inmem.NewState().
pkg/state/impl/store/bolt/namespaced.go Removes per-namespace Bolt backing store wrapper (no longer needed with multi-namespace store API).
pkg/state/impl/store/bolt/example_test.go Updates example to show mixing persistent/ephemeral via namespaced.NewState, using a shared persistent inmem.State.
pkg/state/impl/store/bolt/conformance_test.go Simplifies conformance setup by using a single inmem state with a multi-namespace Bolt backing store.
pkg/state/impl/store/bolt/bbolt.go Refactors Bolt backing store to implement the updated multi-namespace inmem.BackingStore interface.
pkg/state/impl/store/bolt/bbolt_test.go Updates Bolt backing store tests for multi-namespace load/put/destroy semantics.
pkg/state/impl/namespaced/namespaced_test.go Updates namespaced state conformance to construct per-namespace core state via a factory that returns inmem.NewState().
pkg/state/impl/inmem/options.go Updates history option docs and increases default shared buffer capacities for the new shared-buffer model.
pkg/state/impl/inmem/local_test.go Updates tests for new constructors and adds coverage for shared history buffer behavior.
pkg/state/impl/inmem/internal/eventbuffer/export_test.go Exposes watcher notify channel for testing feed isolation (test-only).
pkg/state/impl/inmem/internal/eventbuffer/eventbuffer.go Introduces shared circular buffer + per-feed subscription model for watches.
pkg/state/impl/inmem/internal/eventbuffer/eventbuffer_test.go Adds tests for feed isolation, overruns, bookmarks, tailing, and growth.
pkg/state/impl/inmem/internal/eventbuffer/bookmark.go Introduces bookmark encoding/validation scoped to inmem runs.
pkg/state/impl/inmem/internal/errs/errs.go Centralizes error types compatible with state error predicates.
pkg/state/impl/inmem/internal/collection/watch.go Implements per-collection watch/watch-kind semantics using the shared event buffer.
pkg/state/impl/inmem/internal/collection/collection.go Implements per-(namespace,type) collection storage and persistence hooks.
pkg/state/impl/inmem/internal/collection/collection_test.go Adds comprehensive unit tests for collection CRUD, watch semantics, and persistence interactions.
pkg/state/impl/inmem/inmem.go Reworks inmem state to be shared across namespaces/types and to use the new internal collection/eventbuffer packages.
pkg/state/impl/inmem/errors.go Switches public inmem error helpers to the internal errs implementations.
pkg/state/impl/inmem/collection.go Removes old monolithic per-type collection implementation (replaced by internal packages + shared buffer).
pkg/state/impl/inmem/build.go Removes old builder helper tied to per-namespace construction.
pkg/state/impl/inmem/backing_store.go Updates BackingStore interface to include namespace in Put/Destroy and Load callbacks.
pkg/state/impl/inmem/backing_store_test.go Updates backing store mock/tests for new multi-namespace backing store API.
pkg/state/filter_test.go Updates filter tests to use inmem.NewState() directly.
pkg/safe/state_test.go Updates safe state tests to use inmem.NewState() directly.
pkg/resource/handle/handle_test.go Updates handle tests to use inmem.NewState() directly.
pkg/controller/runtime/runtime_test.go Updates controller runtime tests for new inmem API and shared-buffer option usage.
pkg/controller/generic/transform/transform_test.go Updates transform controller tests to use inmem.NewState().
pkg/controller/generic/qtransform/qtransform_test.go Updates qtransform controller tests to use inmem.NewState().
pkg/controller/generic/destroy/destroy_test.go Updates destroy controller tests to use inmem.NewState().
pkg/controller/generic/cleanup/cleanup_test.go Updates cleanup controller tests to use inmem.NewState().
Makefile Regenerates Makefile and adds an image signing target/variable.
go.sum Updates dependency checksums to match module version changes.
go.mod Bumps a few module versions (e.g., grpc, compress, vtprotobuf).
cmd/runtime/main.go Updates runtime main to use inmem.NewState() directly.
.gitignore Regenerates .gitignore and adds additional ignored patterns.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/state/impl/inmem/internal/eventbuffer/eventbuffer.go
This bring inmem closer to other state implementations:

* `inmem.NewState()` is now namespaced inside, not an instance per
  namespace
* one inmem state instance has a shared event buffer (stream) for all
  resource types and namespaces

This should provide better memory usage for any inmem user, and also it
builds the foundation to build a shared, consistent watch API for the
controller runtime (not in this PR).

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
@smira
smira force-pushed the feat/shared-watch-buffer branch from d9d4070 to c3f9cbb Compare August 5, 2026 10:54
smira added a commit to smira/talos that referenced this pull request Aug 5, 2026
A testbed for the cosi-project/runtime#674

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
@smira
smira requested a lite review from Copilot August 5, 2026 11:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 47 out of 49 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants