feat: rework inmem resource state implementation - #674
Open
smira wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR reworks the in-memory (inmem) state implementation to be a single shared state across namespaces/resource types, backed by a shared circular event buffer, bringing it closer to the behavior/shape of other state implementations and enabling more consistent watch semantics going forward.
Changes:
- Switch
inmem.NewState()to return a single state instance (no longer per-namespace builders) and introduce a shared event buffer across all collections. - Introduce new internal
collection,eventbuffer, anderrspackages to modularize storage/watch/error behavior. - Update BoltDB backing store and tests to support the new “store is multi-namespace”
BackingStoreinterface and simplify state setup across the repo.
Reviewed changes
Copilot reviewed 41 out of 43 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pkg/state/wrap_test.go | Updates conformance test setup to use the new inmem.NewState() API. |
| pkg/state/teardowner_test.go | Updates teardown wrapper tests to use inmem.NewState() directly. |
| pkg/state/teardown_and_destroyer_test.go | Updates teardown+destroy wrapper tests to use inmem.NewState() directly. |
| pkg/state/registry/resource_test.go | Updates registry test setup to use inmem.NewState(). |
| pkg/state/registry/namespace_test.go | Updates registry test setup to use inmem.NewState(). |
| pkg/state/protobuf/protobuf_test.go | Updates protobuf state server/client tests to use inmem.NewState(). |
| pkg/state/protobuf/client/client_test.go | Updates protobuf client test to use inmem.NewState(). |
| pkg/state/owned/owned_test.go | Updates owned-state tests to use inmem.NewState(). |
| pkg/state/impl/store/bolt/namespaced.go | Removes per-namespace Bolt backing store wrapper (no longer needed with multi-namespace store API). |
| pkg/state/impl/store/bolt/example_test.go | Updates example to show mixing persistent/ephemeral via namespaced.NewState, using a shared persistent inmem.State. |
| pkg/state/impl/store/bolt/conformance_test.go | Simplifies conformance setup by using a single inmem state with a multi-namespace Bolt backing store. |
| pkg/state/impl/store/bolt/bbolt.go | Refactors Bolt backing store to implement the updated multi-namespace inmem.BackingStore interface. |
| pkg/state/impl/store/bolt/bbolt_test.go | Updates Bolt backing store tests for multi-namespace load/put/destroy semantics. |
| pkg/state/impl/namespaced/namespaced_test.go | Updates namespaced state conformance to construct per-namespace core state via a factory that returns inmem.NewState(). |
| pkg/state/impl/inmem/options.go | Updates history option docs and increases default shared buffer capacities for the new shared-buffer model. |
| pkg/state/impl/inmem/local_test.go | Updates tests for new constructors and adds coverage for shared history buffer behavior. |
| pkg/state/impl/inmem/internal/eventbuffer/export_test.go | Exposes watcher notify channel for testing feed isolation (test-only). |
| pkg/state/impl/inmem/internal/eventbuffer/eventbuffer.go | Introduces shared circular buffer + per-feed subscription model for watches. |
| pkg/state/impl/inmem/internal/eventbuffer/eventbuffer_test.go | Adds tests for feed isolation, overruns, bookmarks, tailing, and growth. |
| pkg/state/impl/inmem/internal/eventbuffer/bookmark.go | Introduces bookmark encoding/validation scoped to inmem runs. |
| pkg/state/impl/inmem/internal/errs/errs.go | Centralizes error types compatible with state error predicates. |
| pkg/state/impl/inmem/internal/collection/watch.go | Implements per-collection watch/watch-kind semantics using the shared event buffer. |
| pkg/state/impl/inmem/internal/collection/collection.go | Implements per-(namespace,type) collection storage and persistence hooks. |
| pkg/state/impl/inmem/internal/collection/collection_test.go | Adds comprehensive unit tests for collection CRUD, watch semantics, and persistence interactions. |
| pkg/state/impl/inmem/inmem.go | Reworks inmem state to be shared across namespaces/types and to use the new internal collection/eventbuffer packages. |
| pkg/state/impl/inmem/errors.go | Switches public inmem error helpers to the internal errs implementations. |
| pkg/state/impl/inmem/collection.go | Removes old monolithic per-type collection implementation (replaced by internal packages + shared buffer). |
| pkg/state/impl/inmem/build.go | Removes old builder helper tied to per-namespace construction. |
| pkg/state/impl/inmem/backing_store.go | Updates BackingStore interface to include namespace in Put/Destroy and Load callbacks. |
| pkg/state/impl/inmem/backing_store_test.go | Updates backing store mock/tests for new multi-namespace backing store API. |
| pkg/state/filter_test.go | Updates filter tests to use inmem.NewState() directly. |
| pkg/safe/state_test.go | Updates safe state tests to use inmem.NewState() directly. |
| pkg/resource/handle/handle_test.go | Updates handle tests to use inmem.NewState() directly. |
| pkg/controller/runtime/runtime_test.go | Updates controller runtime tests for new inmem API and shared-buffer option usage. |
| pkg/controller/generic/transform/transform_test.go | Updates transform controller tests to use inmem.NewState(). |
| pkg/controller/generic/qtransform/qtransform_test.go | Updates qtransform controller tests to use inmem.NewState(). |
| pkg/controller/generic/destroy/destroy_test.go | Updates destroy controller tests to use inmem.NewState(). |
| pkg/controller/generic/cleanup/cleanup_test.go | Updates cleanup controller tests to use inmem.NewState(). |
| Makefile | Regenerates Makefile and adds an image signing target/variable. |
| go.sum | Updates dependency checksums to match module version changes. |
| go.mod | Bumps a few module versions (e.g., grpc, compress, vtprotobuf). |
| cmd/runtime/main.go | Updates runtime main to use inmem.NewState() directly. |
| .gitignore | Regenerates .gitignore and adds additional ignored patterns. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This bring inmem closer to other state implementations: * `inmem.NewState()` is now namespaced inside, not an instance per namespace * one inmem state instance has a shared event buffer (stream) for all resource types and namespaces This should provide better memory usage for any inmem user, and also it builds the foundation to build a shared, consistent watch API for the controller runtime (not in this PR). Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
smira
force-pushed
the
feat/shared-watch-buffer
branch
from
August 5, 2026 10:54
d9d4070 to
c3f9cbb
Compare
smira
added a commit
to smira/talos
that referenced
this pull request
Aug 5, 2026
A testbed for the cosi-project/runtime#674 Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This bring inmem closer to other state implementations:
inmem.NewState()is now namespaced inside, not an instance per namespaceThis should provide better memory usage for any inmem user, and also it builds the foundation to build a shared, consistent watch API for the controller runtime (not in this PR).