data management

Data Management

Welcome back! In the previous chapters, we explored the Context object for individual visitors, the ExperienceManager for A/B tests, and the FeatureManager for feature flags. You might have noticed a pattern: when we asked the SDK to make a decision (like "which variation should user123 see?" or "is dark-mode enabled?"), the managers didn't just guess. They needed information!

But where does all this information live? How does the SDK know about your specific experiments, features, targeting rules, and which variation 'user123' was assigned to last time?

The Problem: Where Does Everything Live?

Imagine you're building a complex application. You have user settings, product information, application configuration, and maybe temporary data about the user's current session. Where do you keep all this data so different parts of your application can access it when needed? You probably use a database or some central configuration store.

The Convert SDK faces a similar challenge. It needs a central place to:

1. Store Project Setup: Keep track of all the experiences, features, audiences, goals, etc., defined in your Convert account for your specific project.
2. Fetch Updates: Get the latest version of this project setup from the Convert API or use data you provide directly.
3. Remember Visitor Decisions: If 'user123' is assigned to 'variation-B' of the headline test, the SDK needs to remember this so they see the same headline consistently.
4. Store Visitor Segments: Keep track of visitor properties or segments that might affect targeting.
5. Provide Data Access: Allow other managers (like ExperienceManager and FeatureManager) to easily retrieve the specific information they need.

How does the SDK manage this central pool of information?

What is DataManager? The Project's Librarian

Meet the DataManager! Think of it as the central librarian or database interface for your Convert SDK project. It's responsible for holding and managing all the essential information.

Its key responsibilities are:

• The Bookshelf (Project Configuration): It holds the entire configuration downloaded from Convert (or provided by you), including details about all your experiences, features, audiences, goals, etc.
• The Index Cards (Visitor State): It keeps track of visitor-specific information, most importantly:
  • Which variation a visitor has been assigned to for each experience (bucketing decisions).
  • Any relevant visitor segments or properties.
• Fetching New Books (Data Retrieval): It works with the ApiManager to fetch the latest project configuration from Convert's servers.
• Checking Books Out (Providing Data): It offers methods for other managers to look up specific pieces of information (e.g., "get me the details for the 'headline-test' experience" or "find the goal with ID '12345'").
• Using a Filing System (Data Persistence): It can optionally use a DataStoreManager (which you can configure) to save and retrieve visitor bucketing information so that decisions persist across sessions or requests.

In short, DataManager is the single source of truth within the SDK for both the overall project setup and the state of individual visitors.

How it's Used (Mostly Behind the Scenes)

As a developer using the SDK, you will very rarely interact directly with the DataManager. It's primarily an internal component used by other managers.

Remember how running an experience on a Context delegated to the ExperienceManager's variation selection? And how that immediately delegated further? That final delegation step often lands here, at the DataManager.

Let's revisit the flow when you ask for an experience variation:

1. You call the Context's run experience method with 'headline-test'.
2. The Context calls the ExperienceManager's select variation method.
3. The ExperienceManager calls the DataManager's getBucketing('user123', 'headline-test', ...) to handle the entire process.

Similarly, when the FeatureManager needs to know if a feature is enabled, it asks the DataManager to perform the necessary bucketing checks across relevant experiences (getBucketing).

Even fetching basic information relies on the DataManager. For example, inside the ExperienceManager:

• To get details of one experience, the ExperienceManager asks the DataManager to find the entity named by a given key in the 'experiences' list, using a method like getEntity(key, 'experiences').
• To get a list of all experiences, it asks the DataManager for the entire list of 'experiences', using a method like getEntitiesList('experiences').

The ExperienceManager doesn't store the experience data itself; it asks the DataManager for it using methods like getEntity and getEntitiesList.

So, while you don't call DataManager methods directly, understanding its role is key to understanding how the SDK manages data and makes decisions.

Under the Hood: How DataManager Works

Let's peek inside the library.

1. Storing Project Configuration:

• When the SDK initializes (Core), the ApiManager fetches the configuration data (a large JSON object) from Convert's servers (if using an sdkKey).
• This data is passed to the DataManager and stored internally.
• The DataManager provides helper methods to navigate this large structure efficiently:
  • getEntitiesList('experiences'): Returns the array of all experience objects.
  • getEntity('headline-test', 'experiences'): Finds and returns the specific experience with the key 'headline-test'.
  • getEntityById('100567', 'goals'): Finds and returns the goal with the ID '100567'.

2. Storing Visitor State:

• Decisions like "user123 gets variation-B" need to be remembered.
• The DataManager uses an internal, in-memory cache to store these decisions for the current session or request. The structure looks like: { 'account-project-user123': { bucketing: { 'exp1_id': 'varB_id' }, segments: {...} } }
• Persistence (Optional): To remember decisions across page loads, visits, or HTTP requests, the DataManager can use a DataStoreManager. This is a wrapper around a data store object that you might provide in the SDK configuration. The underlying store could be browser cookies, localStorage, sessionStorage, Redis, Memcached, a file-based cache, or even a custom backend storage system you implement.
  • When DataManager stores a decision (using putData), it updates its internal cache and tells the DataStoreManager to save it persistently.
  • When DataManager needs a decision (using getData), it checks its internal cache first, and if not found, asks the DataStoreManager to retrieve it from the persistent store.

graph LR
    DM[DataManager]
    cache["Internal Cache (In-Memory)"]
    dsm["DataStoreManager (Optional)"]
    Store[(Persistent Store <br/> e.g., Cookies, localStorage, Redis)]

    subgraph DataManager Scope
        DM -- Reads/Writes --> cache
        DM -- Reads/Writes --> dsm
    end

    subgraph External Persistence
        dsm -- Reads/Writes --> Store
    end

    style Store fill:#eee,stroke:#333,stroke-width:1px

Loading

3. Orchestrating Bucketing (getBucketing):

This is where the DataManager truly acts as the central coordinator. When the ExperienceManager's select variation method calls DataManager's getBucketing('user123', 'headline-test', attributes):

1. Check Cache/Store: DataManager first calls its getData('user123') method. This checks the internal in-memory cache. If not found there, it asks the DataStoreManager (if configured) to check the persistent store for a previously saved decision for 'headline-test' for 'user123'.
2. Return Cached Decision (If Found): If a valid, previous decision is found, DataManager retrieves the corresponding variation details and returns them immediately. Fast path!
3. Fetch Experience Details: If no decision is cached, DataManager retrieves the full definition of the 'headline-test' experience (variations, rules, traffic split) from its stored project configuration.
4. Check Targeting Rules: It uses the RuleManager to evaluate the experience's targeting rules (audiences, locations) against the visitor's properties (visitorProperties, locationProperties).
5. Rules Fail: If the visitor doesn't meet the targeting criteria, DataManager returns a RuleError.
6. Perform Bucketing: If the rules pass, DataManager calls the BucketingManager. It provides the visitor ID ('user123') and the traffic allocation defined for the variations of 'headline-test'. The BucketingManager calculates which variation ID the visitor falls into.
7. Store New Decision: DataManager receives the chosen variation ID from the BucketingManager. It then calls its putData('user123', { bucketing: { 'headline-test-id': 'chosen-variation-id' } }) method. This saves the decision to the internal cache and also tells the DataStoreManager (if present) to save it persistently.
8. Return New Decision: Finally, DataManager retrieves the full details of the chosen variation and returns the BucketedVariation object.

Here's a sequence diagram illustrating this flow:

sequenceDiagram
    participant EM as ExperienceManager
    participant DM as DataManager
    participant Cache as Internal Cache
    participant Store as DataStore (Optional)
    participant RM as RuleManager
    participant BM as BucketingManager

    EM->>+DM: getBucketing('user123', 'headline-test', ...)
    DM->>+Cache: Get decision for 'user123'/'headline-test'?
    Cache-->>-DM: Not found
    DM->>+Store: Get decision for 'user123'/'headline-test'?
    Store-->>-DM: Not found
    DM->>DM: Get 'headline-test' definition (rules, variations)
    DM->>+RM: Check rules match visitor properties?
    RM-->>-DM: Rules Pass
    DM->>+BM: Calculate variation for 'user123' (based on traffic split)
    BM-->>-DM: variationId = 'variation-B-id'
    DM->>+Cache: Store decision: 'user123'/'headline-test' -> 'variation-B-id'
    Cache-->>-DM: OK
    DM->>+Store: Store decision: 'user123'/'headline-test' -> 'variation-B-id'
    Store-->>-DM: OK
    DM->>DM: Get full details for 'variation-B-id'
    DM-->>-EM: Return BucketedVariation object for 'variation-B'

Loading

How It Works: DataManager Implementation

Let's look at simplified descriptions of the key internal operations.

1. Construction and Setup

When the DataManager is constructed, it:

• Stores references to the managers it needs to collaborate with (RuleManager, BucketingManager, EventManager, ApiManager, LoggerManager).
• Initializes the internal in-memory cache for bucketed visitors.
• Stores the main project configuration data if provided initially.
• Sets up the DataStoreManager if a data store is configured for persistence.

2. Accessing Project Data

The DataManager provides structured methods to access the project configuration:

• A setter/getter for the entire project configuration data, with validation.
• getEntitiesList(entityType) -- returns the full array for a given entity type (e.g., all experiences, all goals).
• getEntity(key, entityType) -- finds a single entity by its key within a given entity type.
• getEntityById(id, entityType) -- finds a single entity by its ID within a given entity type.

Internally, both getEntity and getEntityById use a shared helper that iterates through the list from getEntitiesList and matches on the specified field.

3. Managing Visitor Data

The DataManager manages visitor-specific decisions through two core methods:

• putData(visitorId, newData) -- Merges new data with existing data for the visitor, updates the internal in-memory cache, optionally enforces a cache size limit (evicting the oldest entry when exceeded), and if a DataStoreManager is configured, persists the updated data to the external store.
• getData(visitorId) -- Checks the in-memory cache first, then the persistent store (if configured), and merges the results. The in-memory cache takes priority.
• getStoreKey(visitorId) -- Builds a unique storage key by combining the account ID, project ID, and visitor ID.

4. Orchestrating Bucketing

The core getBucketing method coordinates the entire process:

• It delegates to an internal helper that accepts lookup by either key or ID.
• The helper first checks targeting rules (which also checks for cached bucketing decisions internally) via matchRulesByField.
• If a RuleError is returned, that error is propagated back to the caller.
• If the rules pass, it delegates to an internal _retrieveBucketing method.
• _retrieveBucketing checks whether a stored decision already exists. If so, it returns that. Otherwise, it builds variation buckets (mapping variation IDs to traffic allocations), asks the BucketingManager to pick a variation for the visitor, stores the new decision via putData, optionally enqueues a tracking event via the ApiManager, and returns the full BucketedVariation object.

Conclusion

The DataManager is the unsung hero working tirelessly behind the scenes. It acts as the central library and filing system for the Convert SDK, holding all project configuration (experiences, features, etc.) and managing visitor-specific state like bucketing decisions and segments.

You've learned:

1. Why a central data management component is necessary.
2. That DataManager stores both project-wide configuration and visitor-specific state.
3. How it uses an internal cache and optionally a persistent data store (via DataStoreManager) to remember visitor bucketing decisions.
4. That it orchestrates the bucketing process by coordinating with the RuleManager and BucketingManager.
5. That you typically interact with it indirectly through other managers like ExperienceManager and FeatureManager.

We saw that when rules pass, the DataManager relies on the BucketingManager to perform the crucial step of actually assigning a visitor to a specific variation based on traffic percentages. How does that mathematical assignment work?

Let's dive into the hashing and allocation logic next: BucketingManager!