Storage Architecture

Storage Architecture

Overview

DynamoDbLite stores all data in SQLite, using Dapper for data access. The library provides two storage implementations: in-memory (for tests and development) and file-based (for persistent storage).

SQLite Schema

tables — Table Metadata

CREATE TABLE tables (
    table_name                      TEXT NOT NULL PRIMARY KEY,
    key_schema_json                 TEXT NOT NULL,
    attribute_definitions_json       TEXT NOT NULL,
    provisioned_throughput_json      TEXT NOT NULL DEFAULT '{}',
    global_secondary_indexes_json   TEXT NOT NULL DEFAULT '[]',
    local_secondary_indexes_json    TEXT NOT NULL DEFAULT '[]',
    created_at                      TEXT NOT NULL,
    status                          TEXT NOT NULL DEFAULT 'ACTIVE',
    item_count                      INTEGER NOT NULL DEFAULT 0,
    table_size_bytes                INTEGER NOT NULL DEFAULT 0
);

Table metadata (key schema, attribute definitions, indexes) is stored as JSON. Item count and size are updated on every write.

items — DynamoDB Items

CREATE TABLE items (
    table_name  TEXT NOT NULL,
    pk          TEXT NOT NULL,
    sk          TEXT NOT NULL DEFAULT '',
    sk_num      REAL,
    ttl_epoch   REAL,
    item_json   TEXT NOT NULL,
    PRIMARY KEY (table_name, pk, sk)
);

Column	Description
table_name	Which DynamoDB table this item belongs to
pk	Partition key value (string representation)
sk	Sort key value (empty string if no sort key)
sk_num	Numeric sort key for proper numeric ordering (NULL for string sort keys)
ttl_epoch	Unix epoch for TTL expiration (NULL if no TTL)
item_json	Full item serialized as DynamoDB JSON

All DynamoDB tables share a single items SQLite table, partitioned by table_name.

ttl_config — TTL Settings

CREATE TABLE ttl_config (
    table_name      TEXT PRIMARY KEY,
    attribute_name  TEXT NOT NULL
);

table_tags — Resource Tags

CREATE TABLE table_tags (
    table_name  TEXT NOT NULL,
    tag_key     TEXT NOT NULL,
    tag_value   TEXT NOT NULL DEFAULT '',
    PRIMARY KEY (table_name, tag_key)
);

Index Tables — idx_{tableName}_{indexName}

Each secondary index (GSI or LSI) gets its own SQLite table:

CREATE TABLE "idx_{tableName}_{indexName}" (
    pk          TEXT NOT NULL,
    sk          TEXT NOT NULL DEFAULT '',
    sk_num      REAL,
    table_pk    TEXT NOT NULL,
    table_sk    TEXT NOT NULL DEFAULT '',
    ttl_epoch   REAL,
    item_json   TEXT NOT NULL,
    PRIMARY KEY (pk, sk, table_pk, table_sk)
);

The composite primary key (pk, sk, table_pk, table_sk) allows multiple base table items to map to the same index key (GSIs don't require unique keys).

exports / imports — Export and Import Tracking

CREATE TABLE exports (
    export_arn      TEXT PRIMARY KEY,
    table_name      TEXT NOT NULL,
    status          TEXT NOT NULL DEFAULT 'IN_PROGRESS',
    export_format   TEXT NOT NULL DEFAULT 'DYNAMODB_JSON',
    s3_bucket       TEXT NOT NULL,
    s3_prefix       TEXT NOT NULL DEFAULT '',
    export_manifest TEXT,
    item_count      INTEGER,
    billed_size     INTEGER,
    start_time      TEXT NOT NULL,
    end_time        TEXT,
    failure_code    TEXT,
    failure_message TEXT,
    client_token    TEXT
);

CREATE TABLE imports (
    import_arn          TEXT PRIMARY KEY,
    table_name          TEXT NOT NULL,
    status              TEXT NOT NULL DEFAULT 'IN_PROGRESS',
    input_format        TEXT NOT NULL DEFAULT 'DYNAMODB_JSON',
    input_compression   TEXT NOT NULL DEFAULT 'NONE',
    s3_bucket           TEXT NOT NULL,
    s3_key_prefix       TEXT NOT NULL DEFAULT '',
    table_creation_json TEXT NOT NULL,
    imported_count      INTEGER,
    processed_count     INTEGER,
    processed_bytes     INTEGER,
    error_count         INTEGER DEFAULT 0,
    start_time          TEXT NOT NULL,
    end_time            TEXT,
    failure_code        TEXT,
    failure_message     TEXT,
    client_token        TEXT
);

Storage Implementations

InMemorySqliteStore

• Connection string: contains :memory: or Mode=Memory
• Concurrency: AsyncReaderWriterLock (Stephen Toub pattern) — multiple concurrent readers, exclusive writer
• Sentinel connection: a persistent connection is held open to keep the in-memory database alive for the lifetime of the store
• Best for: unit tests, development

var client = new DynamoDbClient(); // defaults to in-memory
var client = new DynamoDbClient(new DynamoDbLiteOptions(
    "Data Source=MyDb;Mode=Memory;Cache=Shared"));

FileSqliteStore

• Connection string: any non-memory connection string
• Concurrency: SQLite WAL mode — concurrent reads and writes at the database level
• No additional locking: WAL mode handles concurrency natively
• Best for: persistent storage, mobile apps, integration tests

var client = new DynamoDbClient(new DynamoDbLiteOptions(
    "Data Source=myapp.db"));

Store Selection

The store type is automatically selected based on the connection string:

Contains ":memory:" or "Mode=Memory" → InMemorySqliteStore
Otherwise → FileSqliteStore

Serialization

AttributeValueSerializer handles conversion between DynamoDB's Dictionary<string, AttributeValue> and JSON:

• Serialize(Dictionary<string, AttributeValue>) → JSON string
• Deserialize(string json) → Dictionary<string, AttributeValue>

The JSON format matches DynamoDB's JSON representation (type descriptors like {"S": "value"}, {"N": "123"}).

Class Hierarchy

SqliteStore (abstract)
├── InMemorySqliteStore (sealed) — sentinel connection + AsyncReaderWriterLock
└── FileSqliteStore (sealed) — WAL mode, native SQLite concurrency

DynamoDbService (base class — IAmazonService)
└── DynamoDbClient (sealed, partial)
    ├── DynamoDbClient.cs — core, disposal, store creation
    ├── DynamoDbClient.TableManagement.cs — CreateTable, DeleteTable, etc.
    ├── DynamoDbClient.Crud.cs — PutItem, GetItem, DeleteItem, UpdateItem
    ├── DynamoDbClient.Query.cs — Query, Scan
    ├── DynamoDbClient.Batch.cs — BatchGetItem, BatchWriteItem
    ├── DynamoDbClient.Transactions.cs — TransactWriteItems, TransactGetItems
    ├── DynamoDbClient.Admin.cs — Tags, TTL, Endpoints, Limits, UpdateTable
    ├── DynamoDbClient.DataPipeline.cs — Export, Import
    ├── DynamoDbClient.Backup.cs — stubs (NotImplementedException)
    ├── DynamoDbClient.GlobalTables.cs — stubs
    └── DynamoDbClient.Streams.cs — stubs

Index Maintenance

On every write (put, delete, update, batch, transaction), DynamoDbLite:

1. Loads the table's index definitions
2. For deletes: removes old entries from all index tables
3. For puts/updates: extracts index key values from the new item
4. If the item has the required index key attributes, upserts into the index table
5. If the item is missing index key attributes (sparse index), no entry is created

All operations happen within the same SQLite transaction as the base table write.

Next Steps

• Expression Engine — tokenizer, parsers, AST, evaluators
• API Parity — compatibility matrix and behavioral differences