Expression Engine

Expression engine

DynamoDbLite implements a full expression engine for DynamoDB's expression language, using the Superpower parser combinator library. All expressions are parsed into an AST and then either evaluated in-memory or translated to SQL.

Pipeline

Expression string → DynamoDbTokenizer → Parser → AST → Evaluator/SQL Builder

1. Tokenize — DynamoDbTokenizer breaks the raw string into DynamoDbToken tokens, reclassifying identifiers that match DynamoDB keywords (e.g., AND, OR, BETWEEN, begins_with).
2. Parse — A type-specific parser converts the token stream into an AST (record types defined in Ast.cs).
3. Evaluate or build SQL — The AST is either evaluated in-memory against an item, or translated to SQL WHERE fragments for KeyConditionExpression.

Expression types

Expression	Used by	Evaluation
ConditionExpression	PutItem, DeleteItem, UpdateItem, Transactions	In-memory against existing item
UpdateExpression	UpdateItem, TransactWriteItems (Update)	In-memory — produces a new item
ProjectionExpression	GetItem, Query, Scan, BatchGet, TransactGet	In-memory attribute filtering
KeyConditionExpression	Query	Translated to SQL WHERE clause
FilterExpression	Query, Scan	Reuses ConditionExpression — in-memory post-retrieval filter

Components

Component	File	Purpose
DynamoDbTokenizer	Expressions/DynamoDbTokenizer.cs	Tokenizes expression strings; reclassifies keywords
DynamoDbToken	Expressions/DynamoDbToken.cs	Token type enum (identifier, number, string, operators, keywords)
DynamoDbReservedWords	Expressions/DynamoDbReservedWords.cs	Reserved-word set the tokenizer consults during keyword reclassification
Ast	Expressions/Ast.cs	AST record types for all expression nodes
ConditionExpressionParser	Expressions/ConditionExpressionParser.cs	Parses ConditionExpression into AST
ConditionExpressionEvaluator	Expressions/ConditionExpressionEvaluator.cs	Evaluates condition AST against an item
UpdateExpressionParser	Expressions/UpdateExpressionParser.cs	Parses UpdateExpression (SET, REMOVE, ADD, DELETE)
UpdateExpressionEvaluator	Expressions/UpdateExpressionEvaluator.cs	Applies update AST to produce a new item
ProjectionExpressionParser	Expressions/ProjectionExpressionParser.cs	Parses ProjectionExpression into attribute paths
ProjectionExpressionEvaluator	Expressions/ProjectionExpressionEvaluator.cs	Filters item attributes to projected set
KeyConditionExpressionParser	Expressions/KeyConditionExpressionParser.cs	Parses KeyConditionExpression
KeyConditionSqlBuilder	KeyConditionSqlBuilder.cs	Translates KeyCondition AST into SQL WHERE fragments

Expression name resolution

All expression types support ExpressionAttributeNames (placeholder #name → actual attribute name) and ExpressionAttributeValues (placeholder :val → AttributeValue). Resolution happens during evaluation, not during parsing — the AST stores the raw placeholders.

Key condition to SQL translation

KeyConditionSqlBuilder converts the KeyCondition AST into SQL fragments for efficient SQLite querying. Only KeyConditionExpression is translated to SQL; all other expressions are evaluated in-memory after retrieval.

KeyCondition	SQL
Partition key equality	pk = @pk
Sort key =, <, <=, >, >=	Comparison on sk (string) or sk_num (numeric)
begins_with(sk, :prefix)	sk >= @skPrefix AND sk < @skPrefixEnd (with a SUBSTR(sk, 1, n) = @skPrefix fallback for prefixes that cannot be incremented, e.g. all-)
sk BETWEEN :low AND :high	sk BETWEEN @skLow AND @skHigh (or sk_num BETWEEN @skLow AND @skHigh for numeric)

Numeric sort keys use the sk_num column (stored as REAL) to preserve numeric ordering. String sort keys use the sk column with lexicographic ordering. begins_with applies only to string sort keys — numeric sort keys do not support it.

Condition expression operators

The condition expression parser and evaluator support:

• Comparisons: =, <>, <, <=, >, >=
• Logical: AND, OR, NOT
• Functions: attribute_exists, attribute_not_exists, attribute_type, begins_with, contains, size
• Between: value BETWEEN low AND high
• In: value IN (a, b, c)
• Parentheses: grouping with ( and )

size() returns a number suitable for comparison (e.g. size(Items) > :n). On a Number attribute it returns the character length of the string representation of the number — real DynamoDB rejects size() on N, so this is a deliberate deviation that lets size() work uniformly across types.

Update expression clauses

The update expression parser supports all four DynamoDB clauses:

• SET — SET path = value, SET path = path + :val, SET path = if_not_exists(path, :val), SET path = list_append(list1, list2)
• REMOVE — REMOVE path
• ADD — ADD path :val (numeric addition or set union)
• DELETE — DELETE path :val (set subtraction)

Multiple clauses can appear in a single expression and must appear in the order SET → REMOVE → ADD → DELETE (the parser is order-sensitive): SET #a = :v1 REMOVE #b ADD #c :v2.

Unlike SET, the ADD and DELETE clauses accept only an ExpressionAttributeValue reference (:val) as their second operand — a path-to-path form (ADD path1 path2) is not supported.

Error handling

• Tokenization or parse failures throw a Superpower ParseException describing the position and expected token.
• Unknown ExpressionAttributeName (#name) or ExpressionAttributeValue (:val) references throw ArgumentException during evaluation.
• Type-mismatched arithmetic (SET n = path + :val where one operand is non-numeric) and list_append with non-list operands throw ArgumentException.

Next steps

• Storage Architecture — SQLite schema, concurrency, class hierarchy
• API Parity — compatibility matrix and behavioral differences