Skip to content

Storage Architecture

Jump to bottom
Mark Lauter edited this page Jun 2, 2026 · 10 revisions

Storage Architecture

Overview

DynamoDbLite stores all data in SQLite, using Dapper for data access. The library provides two storage implementations: in-memory (for tests and development) and file-based (for persistent storage).

SQLite schema

tables — Table metadata

CREATE TABLE IF NOT EXISTS tables (
    table_name                      TEXT NOT NULL PRIMARY KEY,
    key_schema_json                 TEXT NOT NULL,
    attribute_definitions_json       TEXT NOT NULL,
    provisioned_throughput_json      TEXT NOT NULL DEFAULT '{}',
    global_secondary_indexes_json   TEXT NOT NULL DEFAULT '[]',
    local_secondary_indexes_json    TEXT NOT NULL DEFAULT '[]',
    created_at                      TEXT NOT NULL,
    status                          TEXT NOT NULL DEFAULT 'ACTIVE',
    item_count                      INTEGER NOT NULL DEFAULT 0,
    table_size_bytes                INTEGER NOT NULL DEFAULT 0
);

Table metadata (key schema, attribute definitions, indexes) is stored as JSON. Item count and table_size_bytes are updated on every write — table_size_bytes is the sum of item_json byte lengths and does not match DynamoDB's item-size accounting.

items — DynamoDB items

CREATE TABLE IF NOT EXISTS items (
    table_name  TEXT NOT NULL,
    pk          TEXT NOT NULL,
    sk          TEXT NOT NULL DEFAULT '',
    sk_num      REAL,
    ttl_epoch   REAL,
    item_json   TEXT NOT NULL,
    PRIMARY KEY (table_name, pk, sk)
);
Column Description
table_name Which DynamoDB table this item belongs to
pk Partition key value as a string. Binary (B) keys are stored as Base64.
sk Sort key value (empty string if no sort key)
sk_num Numeric sort key for proper numeric ordering (NULL for string sort keys)
ttl_epoch Unix epoch for TTL expiration (NULL if no TTL)
item_json Full item serialized as DynamoDB JSON

All DynamoDB tables share a single items SQLite table, partitioned by table_name. The ttl_epoch column is added via ALTER TABLE on startup if missing, supporting databases created before TTL was introduced.

ttl_config — TTL settings

CREATE TABLE IF NOT EXISTS ttl_config (
    table_name      TEXT PRIMARY KEY,
    attribute_name  TEXT NOT NULL
);

table_tags — Resource tags

CREATE TABLE IF NOT EXISTS table_tags (
    table_name  TEXT NOT NULL,
    tag_key     TEXT NOT NULL,
    tag_value   TEXT NOT NULL DEFAULT '',
    PRIMARY KEY (table_name, tag_key)
);

Index tables — idx_{tableName}_{indexName}

Each secondary index (GSI or LSI) gets its own SQLite table:

CREATE TABLE IF NOT EXISTS "idx_{tableName}_{indexName}" (
    pk          TEXT NOT NULL,
    sk          TEXT NOT NULL DEFAULT '',
    sk_num      REAL,
    table_pk    TEXT NOT NULL,
    table_sk    TEXT NOT NULL DEFAULT '',
    ttl_epoch   REAL,
    item_json   TEXT NOT NULL,
    PRIMARY KEY (pk, sk, table_pk, table_sk)
);

The composite primary key (pk, sk, table_pk, table_sk) allows multiple base table items to map to the same index key (GSIs don't require unique keys).

exports / imports — Export and import tracking

CREATE TABLE IF NOT EXISTS exports (
    export_arn      TEXT PRIMARY KEY,
    table_name      TEXT NOT NULL,
    status          TEXT NOT NULL DEFAULT 'IN_PROGRESS',
    export_format   TEXT NOT NULL DEFAULT 'DYNAMODB_JSON',
    s3_bucket       TEXT NOT NULL,
    s3_prefix       TEXT NOT NULL DEFAULT '',
    export_manifest TEXT,
    item_count      INTEGER,
    billed_size     INTEGER,
    start_time      TEXT NOT NULL,
    end_time        TEXT,
    failure_code    TEXT,
    failure_message TEXT,
    client_token    TEXT
);

CREATE TABLE IF NOT EXISTS imports (
    import_arn          TEXT PRIMARY KEY,
    table_name          TEXT NOT NULL,
    status              TEXT NOT NULL DEFAULT 'IN_PROGRESS',
    input_format        TEXT NOT NULL DEFAULT 'DYNAMODB_JSON',
    input_compression   TEXT NOT NULL DEFAULT 'NONE',
    s3_bucket           TEXT NOT NULL,
    s3_key_prefix       TEXT NOT NULL DEFAULT '',
    table_creation_json TEXT NOT NULL,
    imported_count      INTEGER,
    processed_count     INTEGER,
    processed_bytes     INTEGER,
    error_count         INTEGER DEFAULT 0,
    start_time          TEXT NOT NULL,
    end_time            TEXT,
    failure_code        TEXT,
    failure_message     TEXT,
    client_token        TEXT
);

Storage implementations

All store connections share the same connection-string handling via SqliteConnectionStringBuilder: the library applies ForeignKeys=true when the caller leaves it unset and otherwise uses the string as written. Mode and Pooling are preserved — forcing Mode would rewrite a caller's Mode=Memory into a file-backed store, so the library leaves it alone.

The two stores differ in how concurrent writers are serialized and tuned — see Concurrency for the model and best practices; the essentials are summarized per store below.

InMemorySqliteStore

  • Connection string: contains :memory: or Mode=Memory
  • Concurrency: no application-level lock. Reads run concurrently through the shared cache; concurrent writers are serialized by Microsoft.Data.Sqlite's CommandTimeout-bounded retry — a shared-cache write conflict raises SQLITE_LOCKED_SHAREDCACHE, which the driver retries until CommandTimeout. busy_timeout is inert here, since the busy handler is not invoked for shared-cache locks.
  • Sentinel connection: a persistent connection is held open to keep the in-memory database alive for the lifetime of the store
  • Best for: unit tests, development
using var client = new DynamoDbClient(new DynamoDbLiteOptions(
    $"Data Source=app_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"));

The Data Source name keys SQLite's shared cache; two clients with the same name share one database. See Getting Started for the foot-gun and how to avoid it in tests.

FileSqliteStore

  • Connection string: any non-memory connection string
  • Concurrency (default): SQLite's default journal mode (delete) — exclusive locking; the writer blocks readers for the duration of the transaction. No application-level lock.
  • Concurrency (WAL): when constructed with UseWriteAheadLog = true (or via DynamoDbLiteOptionsBuilder.WithWriteAheadLog()), the constructor issues PRAGMA journal_mode=WAL and readers proceed concurrently with a writer. WAL is persistent on the file once enabled. See DI and Configuration.
  • Write-lock contention: even under WAL only one writer holds the lock at a time. To make a contending connection wait rather than fail immediately with SQLITE_BUSY, set PRAGMA busy_timeout via WithPragma (e.g. WithPragma("busy_timeout", "5000")). Pragmas apply per connection.
  • Write throughput: WAL is the main lever for file-backed write speed; see Performance for WAL, batching, and the checkpoint/synchronous/mmap_size knobs.
  • Best for: persistent storage, mobile apps, integration tests
var client = new DynamoDbClient(new DynamoDbLiteOptions(
    "Data Source=myapp.db"));

Store selection

The store type is automatically selected based on the connection string:

Contains ":memory:" or "Mode=Memory" → InMemorySqliteStore
Otherwise → FileSqliteStore

Serialization

AttributeValueSerializer handles conversion between DynamoDB's Dictionary<string, AttributeValue> and JSON:

  • Serialize(Dictionary<string, AttributeValue>) → JSON string
  • Deserialize(string json) → Dictionary<string, AttributeValue>

The JSON format matches DynamoDB's JSON representation (type descriptors like {"S": "value"}, {"N": "123"}).

Class hierarchy

DynamoDbClient : IAmazonDynamoDB, IAmazonService, IDisposable  (sealed, partial)
├── DynamoDbClient.cs — core, disposal, store creation
├── DynamoDbClient.TableManagement.cs — CreateTable, DeleteTable, DescribeTable, ListTables, UpdateTable
├── DynamoDbClient.Crud.cs — PutItem, GetItem, DeleteItem, UpdateItem
├── DynamoDbClient.Query.cs — Query
├── DynamoDbClient.Scan.cs — Scan (incl. parallel-scan segmentation)
├── DynamoDbClient.Batch.cs — BatchGetItem, BatchWriteItem
├── DynamoDbClient.Transactions.cs — TransactWriteItems, TransactGetItems
├── DynamoDbClient.Tags.cs — TagResource, UntagResource, ListTagsOfResource
├── DynamoDbClient.Ttl.cs — UpdateTimeToLive, DescribeTimeToLive
├── DynamoDbClient.Service.cs — DescribeEndpoints, DescribeLimits, DetermineServiceOperationEndpoint
├── DynamoDbClient.Export.cs — ExportTableToPointInTime, DescribeExport, ListExports
├── DynamoDbClient.Import.cs — ImportTable, DescribeImport, ListImports
└── DynamoDbClient.Unsupported.cs — out-of-scope stubs (throw `NotSupportedException`)

SqliteStore (internal abstract)
├── InMemorySqliteStore (sealed) — sentinel connection; no app lock (SQLite + driver retry)
└── FileSqliteStore (sealed) — opt-in WAL, native SQLite concurrency

DynamoDbClient has no base class; it implements three interfaces directly. SqliteStore is internal to the assembly.

Index maintenance

On every write (put, delete, update, batch, transaction), DynamoDbLite:

  1. Loads the table's index definitions
  2. For deletes: removes old entries from all index tables
  3. For puts/updates: extracts index key values from the new item
  4. If the item has the required index key attributes, upserts into the index table
  5. If the item is missing index key attributes (sparse index), no entry is created

All operations happen within the same SQLite transaction as the base table write.

Next steps

Repo · NuGet · API Parity

DynamoDbLite

Getting started

Reference

Recipes

Internals

Repo · NuGet

Clone this wiki locally