Expression Engine

Expression engine

DynamoDbLite implements a full expression engine for DynamoDB's expression language, using the Superpower parser combinator library. All expressions are parsed into an AST and then either evaluated in-memory or translated to SQL.

Pipeline

Expression string → DynamoDbTokenizer → Parser → AST → Evaluator/SQL Builder

1. Tokenize — DynamoDbTokenizer breaks the raw string into DynamoDbToken tokens, reclassifying identifiers that match DynamoDB keywords (e.g., AND, OR, BETWEEN, begins_with).
2. Parse — A type-specific parser converts the token stream into an AST (record types defined in Ast.cs).
3. Evaluate or build SQL — The AST is either evaluated in-memory against an item, or translated to SQL WHERE fragments for KeyConditionExpression.

Expression types

Expression	Used by	Evaluation
ConditionExpression	PutItem, DeleteItem, UpdateItem, Transactions	In-memory against existing item
UpdateExpression	UpdateItem, TransactWriteItems (Update)	In-memory — produces a new item
ProjectionExpression	GetItem, Query, Scan, BatchGet, TransactGet	In-memory attribute filtering
KeyConditionExpression	Query	Translated to SQL WHERE clause
FilterExpression	Query, Scan	Reuses ConditionExpression — in-memory post-retrieval filter

Components

Component	File	Purpose
DynamoDbTokenizer	Expressions/DynamoDbTokenizer.cs	Tokenizes expression strings; reclassifies keywords
DynamoDbToken	Expressions/DynamoDbToken.cs	Token type enum (identifier, number, string, operators, keywords)
Ast	Expressions/Ast.cs	AST record types for all expression nodes
ConditionExpressionParser	Expressions/ConditionExpressionParser.cs	Parses ConditionExpression into AST
ConditionExpressionEvaluator	Expressions/ConditionExpressionEvaluator.cs	Evaluates condition AST against an item
UpdateExpressionParser	Expressions/UpdateExpressionParser.cs	Parses UpdateExpression (SET, REMOVE, ADD, DELETE)
UpdateExpressionEvaluator	Expressions/UpdateExpressionEvaluator.cs	Applies update AST to produce a new item
ProjectionExpressionParser	Expressions/ProjectionExpressionParser.cs	Parses ProjectionExpression into attribute paths
ProjectionExpressionEvaluator	Expressions/ProjectionExpressionEvaluator.cs	Filters item attributes to projected set
KeyConditionExpressionParser	Expressions/KeyConditionExpressionParser.cs	Parses KeyConditionExpression
KeyConditionSqlBuilder	KeyConditionSqlBuilder.cs	Translates KeyCondition AST into SQL WHERE fragments

Expression name resolution

All expression types support ExpressionAttributeNames (placeholder #name → actual attribute name) and ExpressionAttributeValues (placeholder :val → AttributeValue). Resolution happens during evaluation, not during parsing — the AST stores the raw placeholders.

Key condition to SQL translation

KeyConditionSqlBuilder converts the KeyCondition AST into SQL fragments for efficient SQLite querying. Only KeyConditionExpression is translated to SQL; all other expressions are evaluated in-memory after retrieval.

KeyCondition	SQL
Partition key equality	pk = @pk
Sort key =, <, <=, >, >=	Comparison on sk (string) or sk_num (numeric)
begins_with(sk, :prefix)	sk >= @prefix AND sk < @prefixEnd
sk BETWEEN :low AND :high	sk >= @low AND sk <= @high (or sk_num for numeric)

Numeric sort keys use the sk_num column (stored as REAL) to preserve numeric ordering. String sort keys use the sk column with lexicographic ordering.

Condition expression operators

The condition expression parser and evaluator support:

• Comparisons: =, <>, <, <=, >, >=
• Logical: AND, OR, NOT
• Functions: attribute_exists, attribute_not_exists, attribute_type, begins_with, contains, size
• Between: value BETWEEN low AND high
• In: value IN (a, b, c)
• Parentheses: grouping with ( and )

Update expression clauses

The update expression parser supports all four DynamoDB clauses:

• SET — SET path = value, SET path = path + :val, SET path = if_not_exists(path, :val), SET path = list_append(list1, list2)
• REMOVE — REMOVE path
• ADD — ADD path :val (numeric addition or set union)
• DELETE — DELETE path :val (set subtraction)

Multiple clauses can appear in a single expression: SET #a = :v1 REMOVE #b ADD #c :v2.

Next steps

• Storage Architecture — SQLite schema, concurrency, class hierarchy
• API Parity — compatibility matrix and behavioral differences