DataManager

Chapter 5: DataManager

Welcome back! In the previous chapters, we explored the Context object for individual visitors, the ExperienceManager for A/B tests, and the FeatureManager for feature flags. You might have noticed a pattern: when we asked the SDK to make a decision (like "which variation should user123 see?" or "is dark-mode enabled?"), the managers didn't just guess. They needed information!

But where does all this information live? How does the SDK know about your specific experiments, features, targeting rules, and which variation 'user123' was assigned to last time?

The Problem: Where Does Everything Live?

Imagine you're building a complex application. You have user settings, product information, application configuration, and maybe temporary data about the user's current session. Where do you keep all this data so different parts of your application can access it when needed? You probably use a database or some central configuration store.

The Convert SDK faces a similar challenge. It needs a central place to:

1. Store Project Setup: Keep track of all the experiences, features, audiences, goals, etc., defined in your Convert account for your specific project.
2. Fetch Updates: Get the latest version of this project setup from the Convert API or use data you provide directly.
3. Remember Visitor Decisions: If 'user123' is assigned to 'variation-B' of the headline test, the SDK needs to remember this so they see the same headline consistently.
4. Store Visitor Segments: Keep track of visitor properties or segments that might affect targeting.
5. Provide Data Access: Allow other managers (like ExperienceManager and FeatureManager) to easily retrieve the specific information they need.

How does the SDK manage this central pool of information?

What is DataManager? The Project's Librarian

Meet the DataManager! Think of it as the central librarian or database interface for your Convert SDK project. It's responsible for holding and managing all the essential information.

Its key responsibilities are:

• The Bookshelf (Project Configuration): It holds the entire configuration downloaded from Convert (or provided by you), including details about all your experiences, features, audiences, goals, etc.
• The Index Cards (Visitor State): It keeps track of visitor-specific information, most importantly:
  • Which variation a visitor has been assigned to for each experience (bucketing decisions).
  • Any relevant visitor segments or properties.
• Fetching New Books (Data Retrieval): It works with the ApiManager to fetch the latest project configuration from Convert's servers.
• Checking Books Out (Providing Data): It offers methods for other managers to look up specific pieces of information (e.g., "get me the details for the 'headline-test' experience" or "find the goal with ID '12345'").
• Using a Filing System (Data Persistence): It can optionally use a DataStoreManager (which you can configure) to save and retrieve visitor bucketing information (e.g., using browser cookies or localStorage) so that decisions persist across visits.

In short, DataManager is the single source of truth within the SDK for both the overall project setup and the state of individual visitors.

How it's Used (Mostly Behind the Scenes)

As a developer using the SDK, you will very rarely interact directly with the DataManager. It's primarily an internal component used by other managers.

Remember how Context.runExperience() delegated to ExperienceManager.selectVariation()? And how ExperienceManager.selectVariation() immediately delegated further? That final delegation step often lands here, at the DataManager.

Let's revisit the flow when you ask for an experience variation:

1. You call visitorContext.runExperience('headline-test').
2. The Context calls experienceManager.selectVariation(...).
3. The ExperienceManager calls dataManager.getBucketing('user123', 'headline-test', ...) to handle the entire process.

Similarly, when the FeatureManager needs to know if a feature is enabled (runFeature), it asks the DataManager to perform the necessary bucketing checks across relevant experiences (getBucketing).

Even fetching basic information relies on the DataManager:

// Inside ExperienceManager (simplified conceptual code)

// Method to get details of one experience
getExperience(key) {
  // Asks DataManager to find the entity named 'key' in the 'experiences' list
  const experienceData = this._dataManager.getEntity(key, 'experiences');
  return experienceData;
}

// Method to get a list of all experiences
getList() {
  // Asks DataManager for the entire list of 'experiences'
  const allExperiences = this._dataManager.getEntitiesList('experiences');
  return allExperiences;
}

• The ExperienceManager doesn't store the experience data itself; it asks the DataManager for it using methods like getEntity and getEntitiesList.

So, while you don't call DataManager methods directly, understanding its role is key to understanding how the SDK manages data and makes decisions.

Under the Hood: How DataManager Works

Let's peek inside the library.

1. Storing Project Configuration:

• When the SDK initializes (Chapter 1: ConvertSDK / Core), the ApiManager fetches the configuration data (a large JSON object) from Convert's servers (if using an sdkKey).
• This data is passed to the DataManager and stored internally (often in a property like this._data).
• The DataManager provides helper methods to navigate this large JSON structure efficiently:
  • getEntitiesList('experiences'): Returns the array of all experience objects.
  • getEntity('headline-test', 'experiences'): Finds and returns the specific experience object with the key 'headline-test'.
  • getEntityById('100567', 'goals'): Finds and returns the goal object with the ID '100567'.

2. Storing Visitor State:

• Decisions like "user123 gets variation-B" need to be remembered.
• The DataManager uses an internal, temporary cache (like a Javascript Map called _bucketedVisitors) to store these decisions for the current session. Map({ 'account-project-user123': { bucketing: { 'exp1_id': 'varB_id' }, segments: {...} } })
• Persistence (Optional): To remember decisions across page loads or visits, the DataManager can use a DataStoreManager.
  • The DataStoreManager is a wrapper around a dataStore object that you might provide in the SDK configuration.
  • This dataStore could be configured to use browser cookies, localStorage, sessionStorage, or even a custom backend storage system you implement.
  • When DataManager stores a decision (using putData), it updates its internal cache and tells the DataStoreManager to save it persistently (e.g., set a cookie).
  • When DataManager needs a decision (using getData), it checks its internal cache first, and if not found, asks the DataStoreManager to retrieve it from the persistent store (e.g., read a cookie).

graph LR
    DM[DataManager]
    cache["Internal Cache (_bucketedVisitors Map)"]
    dsm["DataStoreManager (Optional)"]
    Store[(Persistent Store <br/> e.g., Cookies, localStorage)]

    subgraph DataManager Scope
        DM -- Reads/Writes --> cache
        DM -- Reads/Writes --> dsm
    end

    subgraph External Persistence
        dsm -- Reads/Writes --> Store
    end

    style Store fill:#eee,stroke:#333,stroke-width:1px

Loading

3. Orchestrating Bucketing (getBucketing):

This is where the DataManager truly acts as the central coordinator. When experienceManager.selectVariation calls dataManager.getBucketing('user123', 'headline-test', { ... }):

1. Check Cache/Store: DataManager first calls its getData('user123') method. This checks the internal _bucketedVisitors map. If not found there, it asks the DataStoreManager (if configured) to check the persistent store (e.g., cookies) for a previously saved decision for 'headline-test' for 'user123'.
2. Return Cached Decision (If Found): If a valid, previous decision is found, DataManager retrieves the corresponding variation details and returns them immediately. Fast path!
3. Fetch Experience Details: If no decision is cached, DataManager retrieves the full definition of the 'headline-test' experience (variations, rules, traffic split) from its stored project configuration (this._data).
4. Check Targeting Rules: It uses the RuleManager to evaluate the experience's targeting rules (audiences, locations) against the visitor's properties (visitorProperties, locationProperties).
5. Rules Fail: If the visitor doesn't meet the targeting criteria, DataManager returns a RuleError.
6. Perform Bucketing: If the rules pass, DataManager calls the BucketingManager. It provides the visitor ID ('user123') and the traffic allocation defined for the variations of 'headline-test'. The BucketingManager calculates which variation ID the visitor falls into.
7. Store New Decision: DataManager receives the chosen variation ID from the BucketingManager. It then calls its putData('user123', { bucketing: { 'headline-test-id': 'chosen-variation-id' } }) method. This saves the decision to the internal cache (_bucketedVisitors) and also tells the DataStoreManager (if present) to save it persistently (e.g., update the cookie).
8. Return New Decision: Finally, DataManager retrieves the full details of the chosen variation and returns the BucketedVariation object.

Here's a sequence diagram illustrating this flow:

sequenceDiagram
    participant EM as ExperienceManager
    participant DM as DataManager
    participant Cache as Internal Cache
    participant Store as DataStore (Optional)
    participant RM as RuleManager
    participant BM as BucketingManager

    EM->>+DM: getBucketing('user123', 'headline-test', ...)
    DM->>+Cache: Get decision for 'user123'/'headline-test'?
    Cache-->>-DM: Not found
    DM->>+Store: Get decision for 'user123'/'headline-test'?
    Store-->>-DM: Not found
    DM->>DM: Get 'headline-test' definition (rules, variations)
    DM->>+RM: Check rules match visitor properties?
    RM-->>-DM: Rules Pass
    DM->>+BM: Calculate variation for 'user123' (based on traffic split)
    BM-->>-DM: variationId = 'variation-B-id'
    DM->>+Cache: Store decision: 'user123'/'headline-test' -> 'variation-B-id'
    Cache-->>-DM: OK
    DM->>+Store: Store decision: 'user123'/'headline-test' -> 'variation-B-id'
    Store-->>-DM: OK
    DM->>DM: Get full details for 'variation-B-id'
    DM-->>-EM: Return BucketedVariation object for 'variation-B'

Loading

Code Dive: DataManager Implementation

Let's glance at simplified snippets from data-manager.ts.

1. The Constructor (data-manager.ts)

Sets up dependencies and the optional DataStore.

// File: packages/data/src/data-manager.ts (Simplified)

import {BucketingManagerInterface} from '@convertcom/js-sdk-bucketing';
import {RuleManagerInterface} from '@convertcom/js-sdk-rules';
import {DataStoreManager} from './data-store-manager';
// ... other imports

export class DataManager implements DataManagerInterface {
  private _data: ConfigResponseData; // Holds the main project config JSON
  private _bucketingManager: BucketingManagerInterface;
  private _ruleManager: RuleManagerInterface;
  private _dataStoreManager: DataStoreManagerInterface; // Wrapper for persistence
  private _bucketedVisitors = new Map(); // Internal cache

  constructor(
    config: Config,
    { // Dependencies injected by Core
      bucketingManager,
      ruleManager,
      eventManager,
      apiManager,
      loggerManager
    }: { /* ... */ }
  ) {
    // Store references to other managers
    this._bucketingManager = bucketingManager;
    this._ruleManager = ruleManager;
    // ... store others ...

    // Store the main config data if provided initially
    this._data = config?.data;

    // Initialize the DataStoreManager if a dataStore is provided in config
    this.dataStoreManager = config?.dataStore;
  }

  // dataStoreManager setter/getter allows initializing it
  set dataStoreManager(dataStore: any) {
    this._dataStoreManager = null;
    if (dataStore) {
      // Creates the wrapper if a valid store is provided
      this._dataStoreManager = new DataStoreManager(this._config, {
        dataStore: dataStore, /* ... */
      });
    }
  }
  get dataStoreManager(): DataStoreManagerInterface { /* ... */ }

  // ... other methods ...
}

• Stores references to the managers it needs to collaborate with (RuleManager, BucketingManager).
• Initializes the internal cache (_bucketedVisitors).
• Sets up the DataStoreManager if a dataStore (like cookies) is configured for persistence.

2. Accessing Project Data (data-manager.ts)

Methods to retrieve parts of the main configuration.

// File: packages/data/src/data-manager.ts (Simplified)

// Inside DataManager class:

  // Holds the big JSON object from Convert API or config.data
  set data(data: ConfigResponseData) {
    if (this.isValidConfigData(data)) {
      this._data = data;
      // Store project/account IDs too
    } else { /* Log error */ }
  }
  get data(): ConfigResponseData {
    return this._data;
  }

  // Get a whole list (e.g., all experiences)
  getEntitiesList(entityType: string): Array<Entity | string> {
    // Uses helper to safely access nested property in this._data
    // Example: entityType 'experiences' -> this._data['experiences']
    return objectDeepValue(this._data, entityType) || [];
  }

  // Find a single item by its key (e.g., find experience with key 'headline-test')
  getEntity(key: string, entityType: string): Entity {
    // Finds item in the list from getEntitiesList where item.key === key
    return this._getEntityByField(key, entityType, 'key');
  }

  // Find a single item by its ID (e.g., find goal with id '100567')
  getEntityById(id: string, entityType: string): Entity {
    // Finds item in the list where item.id === id
    return this._getEntityByField(id, entityType, 'id');
  }

  // Internal helper used by getEntity and getEntityById
  private _getEntityByField(identity: string, entityType: string, field: IdentityField): Entity {
      const list = this.getEntitiesList(entityType) as Array<Entity>;
      // Loop through the list and find the matching item
      for (let i = 0; i < list.length; i++) {
          if (list[i] && String(list[i]?.[field]) === String(identity)) {
              return list[i];
          }
      }
      return null; // Not found
  }

• These methods provide structured ways to access the potentially large this._data configuration object.

3. Managing Visitor Data (data-manager.ts)

Methods for storing and retrieving visitor-specific decisions.

// File: packages/data/src/data-manager.ts (Simplified)

// Inside DataManager class:

  // Stores or updates data for a visitor
  putData(visitorId: string, newData: StoreData = {}) {
    const storeKey = this.getStoreKey(visitorId); // e.g., "account-project-visitor123"
    const currentData = this.getData(visitorId) || {}; // Get existing data
    const updatedData = objectDeepMerge(currentData, newData); // Merge old and new

    // Update the internal cache
    this._bucketedVisitors.set(storeKey, updatedData);
    // Optional: Limit cache size (remove oldest if too large)

    // If DataStoreManager is configured, tell it to save persistently
    if (this.dataStoreManager) {
        this.dataStoreManager.enqueue(storeKey, updatedData); // Enqueue for saving
    }
  }

  // Retrieves data for a visitor
  getData(visitorId: string): StoreData {
    const storeKey = this.getStoreKey(visitorId);
    // 1. Check fast internal cache first
    const memoryData = this._bucketedVisitors.get(storeKey);

    // 2. If DataStoreManager exists, check persistent store
    let storeData = null;
    if (this.dataStoreManager) {
        storeData = this.dataStoreManager.get(storeKey); // e.g., read cookie
    }

    // Merge results (memory cache overrides persistent store if both exist)
    return objectDeepMerge(storeData || {}, memoryData || {});
  }

  // Helper to create a unique key for storage
  getStoreKey(visitorId: string): string {
    // Combines account, project, and visitor IDs for uniqueness
    return `${this._accountId}-${this._projectId}-${visitorId}`;
  }

• putData updates both the internal cache (_bucketedVisitors) and potentially queues the data for persistent storage via DataStoreManager.
• getData checks the cache first, then the persistent store, merging the results.

4. Orchestrating Bucketing (data-manager.ts)

The core getBucketing method coordinates the process.

// File: packages/data/src/data-manager.ts (Simplified)

// Inside DataManager class:

  getBucketing(
    visitorId: string,
    key: string, // e.g., 'headline-test'
    attributes: BucketingAttributes // contains visitorProperties, etc.
  ): BucketedVariation | RuleError | BucketingError {
    // This calls an internal helper that does the main work
    return this._getBucketingByField(visitorId, key, 'key', attributes);
  }

  private _getBucketingByField(
    visitorId: string,
    identity: string, // The key or ID of the experience
    identityField: IdentityField, // 'key' or 'id'
    attributes: BucketingAttributes
  ): BucketedVariation | RuleError | BucketingError {

    // 1. Check cache/store first (simplified view - happens inside matchRules/retrieveBucketing)
    const {bucketing: cachedBucketing} = this.getData(visitorId) || {};
    // ... logic to check if decision for this experience exists and is valid ...
    // if (cachedDecisionIsValid) return retrieveVariation(cachedDecision);

    // 2. If not cached, check rules
    const experience = this.matchRulesByField(visitorId, identity, identityField, attributes);
    // matchRulesByField uses this._ruleManager internally

    if (!experience) return null; // No match or archived/invalid experience
    if (/* experience is a RuleError */) return experience; // Rule check failed

    // 3. If rules pass, perform bucketing & store result
    return this._retrieveBucketing(
        visitorId,
        attributes.visitorProperties,
        attributes.updateVisitorProperties, // Should we store visitor props?
        experience as ConfigExperience, // The matched experience
        attributes.forceVariationId, // Optional override
        attributes.enableTracking // Should we send tracking event?
    );
  }

  // Helper that performs the actual bucketing and stores the result
  private _retrieveBucketing(
    visitorId: string, visitorProperties: any, updateVisitorProperties: boolean,
    experience: ConfigExperience, forceVariationId?: string, enableTracking = true
  ): BucketedVariation | BucketingError {

    // Check if already bucketed (re-checks cache/store specifically for this experience)
    const {bucketing} = this.getData(visitorId) || {};
    const storedVariationId = bucketing?.[experience.id.toString()];
    // ... logic to use storedVariationId if valid ...

    // If not stored or invalid:
    // Build variation buckets { variationId: trafficAllocation, ... }
    const buckets = /* ... get variations and traffic % from experience ... */;

    // Ask BucketingManager to pick one
    const decision = this._bucketingManager.getBucketForVisitor(
        buckets, visitorId, /* options */
    );
    const chosenVariationId = forceVariationId || decision?.variationId;

    if (!chosenVariationId) return BucketingError.VARIAION_NOT_DECIDED;

    // Store the new decision!
    this.putData(visitorId, {
        bucketing: { [experience.id.toString()]: chosenVariationId },
        // Optionally store visitorProperties too if updateVisitorProperties is true
    });

    // Enqueue tracking event via ApiManager (if enableTracking)
    if (enableTracking) { /* ... this._apiManager.enqueue(...) ... */ }

    // Get full variation details and return
    const variationDetails = this.retrieveVariation(experience.id, chosenVariationId);
    return { /* ... build BucketedVariation object ... */ };
  }

• This shows the high-level orchestration: check cache/rules first (matchRulesByField), then if needed, perform bucketing (_retrieveBucketing which uses _bucketingManager) and store the result (putData).

Conclusion

The DataManager is the unsung hero working tirelessly behind the scenes. It acts as the central library and filing system for the Convert SDK, holding all project configuration (experiences, features, etc.) and managing visitor-specific state like bucketing decisions and segments.

You've learned:

1. Why a central data management component is necessary.
2. That DataManager stores both project-wide configuration and visitor-specific state.
3. How it uses an internal cache and optionally a persistent DataStore (like cookies) to remember visitor bucketing.
4. That it orchestrates the bucketing process by coordinating with the RuleManager and BucketingManager.
5. That you typically interact with it indirectly through other managers like ExperienceManager and FeatureManager.

We saw that when rules pass, the DataManager relies on the BucketingManager to perform the crucial step of actually assigning a visitor to a specific variation based on traffic percentages. How does that mathematical assignment work?

Let's dive into the hashing and allocation logic in the next chapter: Chapter 6: BucketingManager!