feat: server-side data system #304
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR adds the initial set of interfaces defined in the Data System spec. The intention is to allow our internal components to interoperate using the same interfaces that a 3rd party would use to implement a custom persistent store.
This PR moves the contents of `src/data_sources` (polling + streaming implementations) into dedicated subdirectories of a new directory, `data_components`. The thousand-yard view is that we now have a larger abstraction (Data System) of which these components are a part of. The objects in `data_components` can be composed by SDK authors to create the Background Sync and Lazy Load data systems. Background Sync is the equivalent of streaming/polling, while Lazy Load operates against a database.
This introduces a set of configuration objects & builders for the server-side Data System. These are defined in `libs/server-sdk` rather than in common, since they don't apply to the client at the moment. In a future client major version, we could attempt to unify them.
This PR takes the existing components (like change notifier, expiration tracker, streaming/polling datasources) and refactors them to use the Data System interfaces. The broad strokes of the architecture are now visible in the server's directory structure.
…oses (#295) The original names were an attempting to retain some symmetry, while also showing that both modes are "sources". This doesn't need to be the case. These "sources" are conceptual, but have totally different modes of interaction. I've updated the names to `IDataSynchronizer` and `IDataReader`. These are names that I used in past revisions of the DS spec, and I think they are the better choice today.
This fixes a bunch of compiler errors on the `feat/data-system` branch, which existed because of the piecemeal manner in which I broke out chunks from my working Redis client branch.
This updates the server-side SDK architecture diagram to use Data System concepts. It visualizes the Background Sync and Lazy Load systems. Additionally, I've renamed `JsonReader` to `JsonDeserializer` to better reflect its purpose (figured this out when updating the diagram.) --------- Co-authored-by: Matthew M. Keeler <keelerm84@gmail.com>
Removes a verbose namespace prefix that isn't necessary.
This removes obsolete server-related config from the `common` lib.
This adds in some missing top-level Data System builder methods, as well as additional documentation. This re-adds the previously removed Offline mode configuration, which modifies Data System and Events configuration.
Removes some redundant namespace prefixes in `expiration_tracker` to make the code more readable.
#306) This refactors `IDataSynchronizer` to take an `IDestination` and bootstrap data in its `StartAsync` method, rather than having an independent `Init` function. The addition of `IDestination` makes the interface's purpose clearer. Before the destination was hidden - you'd only see it when constructing an implementation. Now it's clear that the `IDataSynchronizer` puts data into the `IDestination`; no hidden dependency. It also doesn't make sense to "Init" an `IDataSynchronizer` with an SDKDataSet. The purpose of that call is to allow the synchronizer to inspect any existing data in possession by the SDK. The result of that inspection might mean an optimized fetch of initial data is possible (e.g. if there was an etag present in the initial data.) --------- Co-authored-by: Matthew M. Keeler <keelerm84@gmail.com>
This adds a `Shutdown` method to `IDataSystem`, which allows the system to stop any long-running asynchronous operations (such as streaming/polling mode.) Currently for the LazyLoad system, this is a no-op. In the future if we have some sort of "active cache eviction" on an interval, this would be used.
This PR adds a JSON adapter (`JsonDestination`) which is capable of accepting SDK in-memory models, transforming them into JSON, and passing them down to an `ISerializedDataDestination`. In practice, this will be used to store serialized data in Redis.
Current usage of `assign` will be a no-op if the expiry for an existing item is added.
This implements the Lazy Load system, which is 1 of 2 possible Data Systems in the server-side SDK. It's a caching layer over an underlying `ISerializedDataReader`. When items are requested by the SDK, it first checks to see if the item is fresh or stale; if stale, it queries the reader.
This implements a Redis data source for the server-side SDK. When using this source, the SDK operates in what was previously known as "daemon mode". It treats Redis as read-only, loading data on-demand into an in-memory cache. There is no connection to LaunchDarkly when using a Redis data source. That'll come later, when Redis will be able to be used as a destination. --------- Co-authored-by: Molly <molly.jones@launchdarkly.com> Co-authored-by: Ryan Lamb <4955475+kinyoklion@users.noreply.github.com>
kinyoklion
approved these changes
Dec 4, 2023
Merged
cwaldren-ld
pushed a commit
that referenced
this pull request
Dec 4, 2023
🤖 I have created a release *beep* *boop* --- <details><summary>launchdarkly-cpp-client: 3.3.0</summary> ## [3.3.0](launchdarkly-cpp-client-v3.2.3...launchdarkly-cpp-client-v3.3.0) (2023-12-04) ### Features * server-side data system ([#304](#304)) ([9a3737d](9a3737d)) ### Dependencies * The following workspace dependencies were updated * dependencies * launchdarkly-cpp-internal bumped from 0.4.0 to 0.5.0 * launchdarkly-cpp-common bumped from 1.0.0 to 1.1.0 * launchdarkly-cpp-sse-client bumped from 0.2.0 to 0.3.0 </details> <details><summary>launchdarkly-cpp-common: 1.1.0</summary> ## [1.1.0](launchdarkly-cpp-common-v1.0.0...launchdarkly-cpp-common-v1.1.0) (2023-12-04) ### Features * server-side data system ([#304](#304)) ([9a3737d](9a3737d)) </details> <details><summary>launchdarkly-cpp-internal: 0.5.0</summary> ## [0.5.0](launchdarkly-cpp-internal-v0.4.0...launchdarkly-cpp-internal-v0.5.0) (2023-12-04) ### Features * server-side data system ([#304](#304)) ([9a3737d](9a3737d)) ### Dependencies * The following workspace dependencies were updated * dependencies * launchdarkly-cpp-common bumped from 1.0.0 to 1.1.0 </details> <details><summary>launchdarkly-cpp-server: 0.3.0</summary> ## [0.3.0](launchdarkly-cpp-server-v0.2.1...launchdarkly-cpp-server-v0.3.0) (2023-12-04) ### Features * server-side data system ([#304](#304)) ([9a3737d](9a3737d)) ### Dependencies * The following workspace dependencies were updated * dependencies * launchdarkly-cpp-internal bumped from 0.4.0 to 0.5.0 * launchdarkly-cpp-common bumped from 1.0.0 to 1.1.0 * launchdarkly-cpp-sse-client bumped from 0.2.0 to 0.3.0 </details> <details><summary>launchdarkly-cpp-sse-client: 0.3.0</summary> ## [0.3.0](launchdarkly-cpp-sse-client-v0.2.0...launchdarkly-cpp-sse-client-v0.3.0) (2023-12-04) ### Features * server-side data system ([#304](#304)) ([9a3737d](9a3737d)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Merged
cwaldren-ld
pushed a commit
that referenced
this pull request
Dec 4, 2023
🤖 I have created a release *beep* *boop* --- <details><summary>launchdarkly-cpp-server-redis-source: 0.1.0</summary> ## 0.1.0 (2023-12-04) ### Features * server-side data system ([#304](#304)) ([9a3737d](9a3737d)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This feature branch contains the server-side Data System implementation.
This system allows the server-side SDK to operate within two paradigms: that of a background connection that streams data from LaunchDarkly, or a connection to a database mediated by an in-memory caching algorithm.