-
Notifications
You must be signed in to change notification settings - Fork 2
NORM Querying
Read and write data with norm's typed repositories: filters, typed projections, relations, aggregates, and pagination. Every method on a repository resolves off your entity declaration, so column names, operators, projection shapes, and return types are checked at compile time. A typo is a TypeScript error rather than a runtime surprise.
- Setup
- The result envelope
- Reading: find, findOne, getByPK, count
- Filters
- Projections
- Relations
- Aggregates
- Pagination and totals
- Writes
- Escape hatches
- Related documentation
Every example below operates on a composed database handle db. You
obtain one by constructing a Norm from a database config and
composing one or more schemas with use:
import '@tundralibs/norm/engines/sqlite';
import { Norm } from '@tundralibs/norm';
import { Identity, Shortener } from './models/mod.ts';
const norm = new Norm({
database: { dialect: 'sqlite', path: './data' },
secret: process.env.SECRET,
});
const db = norm.use(Identity, Shortener);
// db.repo(key) returns a typed repository for a registered entity.
const users = db.repo('Users');
const links = db.repo('Links');db.repo(key) returns one of three accessor shapes, chosen by the
entity's kind:
-
TABLEentities return a fullRepo, with reads and writes. -
VIEWentities return aReadRepo, the read surface only. -
QUERYentities (stored SELECTs) return aQueryAccessor, a singlefind({ limit?, offset? })that re-issues the stored query.
The read methods (find, findOne, getByPK, count) live on every
Repo and ReadRepo. The write methods (insert, update, delete,
upsert, truncate) live on Repo only.
Every operation resolves to a single NormResult envelope. It rides on
the engine's result and adds correlation metadata:
| Field | Type | Meaning |
|---|---|---|
id |
string |
A ULID minted per operation; the same id is stamped on the call event. |
op |
string |
The executed operation: SELECT, INSERT, UPDATE, DELETE, COUNT, UPSERT, TRUNCATE, RAW. |
count |
number |
Rows in this result for reads (pagination applies), affected rows for writes, the answer for count(). |
time |
number |
Engine-reported duration in milliseconds. |
isSlow |
boolean |
Whether the engine flagged the call as slow. |
data? |
P |
Present on data-bearing ops; absent on count-only ops. |
total? |
number |
Only on find(filter, { total: true }): matching rows regardless of paging. |
scoped? |
Record<string,unknown> |
The equality scope filter that was applied (keyed by @column), for auditing. |
txId? |
string |
Present only when the op ran on a transaction-scoped handle. |
data is typed per method: Row[] for find, insert, and upsert;
Row | null for findOne and getByPK; and absent for count,
update, delete, and truncate, where the answer rides count. For
reads, count is the size of this page. Use total: true when you
need the full match count under pagination.
const r = await db.repo('Users').find({ '@role': 'admin' });
r.id; // '01J...' — correlate with the `call` event
r.op; // 'SELECT'
r.count; // rows in this page
r.data; // typed row array
r.isSlow; // engine slow-query flagThe filter is the first positional argument, not an option. The second
argument is FindOptions:
type FindOptions = {
orderBy?: Record<string, 'ASC' | 'DESC'>;
project?: ProjectionInput; // typed projection (see Projections)
aggregates?: AggregateInput; // grouped report queries (see Aggregates)
limit?: number;
offset?: number;
decrypt?: boolean; // default true; false leaves ciphertext
total?: boolean; // also run a matching COUNT → result.total
};const admins = await db.repo('Users').find({ '@role': 'admin' }, {
orderBy: { '@displayName': 'ASC' },
limit: 20,
project: { '@id': true, '@displayName': true },
total: true,
});
admins.data; // ProjectedRowOf<...>[] — { id, displayName }[]
admins.total; // total matching admins, ignoring the limitA limit-less read pages at the entity's defaultPageSize (10 unless
the entity declares otherwise). A limit of 0, passed or declared,
means unbounded and emits a warning event on every such read.
decrypt: false leaves encrypted columns as their stored ciphertext,
touches no secret, and skips afterRead transforms on those columns.
Use it for bulk exports that never need plaintext.
Returns the first matching row, even when several match, or null. It
takes the same options as find except limit, which is forced to 1:
const one = await db.repo('Users').findOne({ '@email': 'ada@shortly.dev' });
one.count; // 0 or 1
one.data?.role; // string | undefined
const withProfile = await db.repo('Users').findOne({ '@id': someId }, {
project: { '@id': true, '@Profile': { '@bio': true } },
});
withProfile.data?.Profile; // { bio: string } | nullFetch one row by primary key, supplying every key column for a
composite key. Options are { project?, decrypt? }:
const user = await db.repo('Users').getByPK({ id: someId });
user.data; // DefaultRowOf<...> | null
// Composite primary key — every key column is required:
const tag = await db.repo('PostTags').getByPK({ postId: 1, tagId: 2 });Counts matching rows. The answer rides count and the envelope has no
data. An empty {} filter counts all rows.
const n = await db.repo('Users').count({ '@role': 'viewer' });
n.count; // number
// Joined filter on a COUNT — a real SQL join under the hood:
const c = await db.repo('Visits').count({ '@Link.@slug': 'link-00' });Filters are the OQL filter language
typed to your columns. norm's FilterOf type is OQL's own
QueryFilter derived over your entity's column shape. A bare @column
key is equality shorthand, an operator bag applies one of the operators
below, and $and / $or compose sub-filters.
await db.repo('Users').find({ '@role': 'admin' }); // role = 'admin'
await db.repo('Links').find({ '@isActive': true }); // isActive = true| Operator | Meaning |
|---|---|
$eq |
Equal (the shorthand's long form). |
$ne |
Not equal. |
$in |
Value in a list. |
$nin |
Value not in a list. |
$like |
Pattern match (%, _). |
$ilike |
Case-insensitive pattern match. |
$between |
Inclusive range [low, high]. |
$null |
$null: true / false — IS (NOT) NULL. |
$or |
Any sub-filter matches. |
$and |
All sub-filters match. |
This guide's examples use the subset above. Because FilterOf rides
OQL's own operator typing, every OQL operator type-checks and runs on
the matching column kind: $gt, $gte, $lt, and $lte on number,
bigint, and date columns, and $nlike, $nilike, $startsWith,
$endsWith, and $contains on string columns. See OQL's
Comprehensive Filter System
for the complete operator grammar per column type.
// $in over a hashed column — plaintext equality, rewritten to digests:
await db.repo('Users').find({
'@email': { $in: ['bob@shortly.dev', 'eve@shortly.dev'] },
});
// $like / $between / $null on plain columns:
await db.repo('Links').count({ '@slug': { $like: 'link-1%' } });
await db.repo('Links').count({ '@id': { $between: [5, 8] } });
await db.repo('Links').count({ '@expiresAt': { $null: true } });
// $or composition:
await db.repo('Users').find({
$or: [{ '@email': 'ada@shortly.dev' }, { '@role': 'admin' }],
});A '@Alias.@col' key filters through a foreign-key alias or a reverse
relation. The relation name comes from your FK declaration:
// belongsTo: Links → Owner (a Users FK):
await db.repo('Links').find({ '@Owner.@role': 'viewer' });
// Cross-schema belongsTo on a COUNT:
await db.repo('Visits').count({ '@Link.@slug': 'link-00' });Filtering through a to-many relation that is not projected is lifted
into a correlated $exists subquery, so it never fans out. A base row
matching N related rows still comes back once, and count() does not
over-count:
// Every link with at least one visit from 'IN' — each link ONCE:
const inLinks = await db.repo('Links').find({ '@Visits.@country': 'IN' });
inLinks.count; // distinct links, no join fan-out
// count() agrees; total:true rides the same EXISTS plan:
await db.repo('Links').count({ '@Visits.@country': 'IN' });
await db.repo('Links').find({ '@Visits.@country': 'IN' }, {
orderBy: { '@slug': 'ASC' },
limit: 10,
total: true,
});A relation column may also appear in value position, as the right-hand
side of an operator, for a cross-column comparison. That plans the join
for a belongsTo or hasOne alias. An unprojected to-many cannot supply a
comparison value, because it runs as an EXISTS subquery with nothing
to compare an outer column against, so that spelling throws rather than
comparing against the literal text:
// belongsTo in value position — joins, compares column to column:
await db.repo('Links').find({ '@createdAt': { $gt: '@Owner.@createdAt' } });
// Unprojected to-many in value position — NormQueryError. Project the
// relation (it then joins), or write the condition key-position.
await db.repo('Owners').find({ '@name': { $gt: '@Items.@label' } });The lift above is norm generating an $exists node from an
'@Alias.@col' ref. Because FilterOf is OQL's own QueryFilter,
OQL's raw top-level
$exists / $nexists
correlated-subquery operators are part of the type too, and they reach
the SQL translator unchanged:
// Physical table name ('items'), NOT the entity registry key ('Items') —
// this bypasses norm's alias/entity abstraction entirely.
await db.repo('Owners').find({
$exists: {
table: 'items',
on: { '@ownerId': '@id' }, // subquery-local column : outer column
where: { '@label': 'x' },
},
});This is a raw escape hatch rather than a typed norm feature. table is
the physical table name, the columns inside on and where are not
validated against any entity, and none of norm's rewrites (hashed
column to digest, scope filters) apply inside it. Prefer the
'@Alias.@col' auto-lift, which is typed and validated and needs no
physical names. Reach for a hand-written $exists only for a
correlation OQL can express but norm's relation graph cannot, such as
correlating on a column that is not part of any declared foreign key.
A column declared .encrypt().hash() stores ciphertext but stays
filterable: equality-class operators against the plaintext are
rewritten to digest equality on the synthesized <col>_hash sibling,
with the same beforeWrite normalization the write path applies.
Digests support equality only: $eq, $ne, $in, $nin, $null.
// Filter by plaintext — rewritten to email_hash = sha256('ada@...').
// beforeWrite trims + lowercases, so this matches regardless of case:
const one = await db.repo('Users').findOne({
'@email': ' Ada@Shortly.Dev ',
});A standalone Column.hash(algo) digest column (a PIN, say) works the
same way: store and filter by the plaintext, and never see it again:
const byPin = await db.repo('Users').findOne({ '@pin': '4471' });References that cannot be filtered throw a NormQueryError before any
SQL runs:
- An encrypted column without
.hash(). Ciphertext is IV-randomized, so equality is meaningless; declare.hash()to enable plaintext equality. - A column marked
.unfilterable(). - Ordering or aggregating an encrypted column. Randomized ciphertext neither sorts nor groups.
JSON columns cannot be filtered by path. OQL supports @col.@key
JSON path filtering; norm
does not implement it. norm's relation resolver reads any two-segment
'@Alias.@col' key as a foreign-key or reverse-relation reference, JSON
columns included. So '@payload.@kind' on a Column.json<Shape>()
column type-checks, because the typed path key is inherited from OQL,
but throws NormQueryError (UNKNOWN_RELATION) at runtime, reporting
payload as an unknown relation alias. Filter a JSON column only as a
whole value ($eq, $ne, $in, $nin, $null). For path-level
filtering, model a VIEW with the extraction baked into its stored
SELECT, or drop to db.query() with a hand-built IR (see
Escape hatches).
project reshapes result rows. Every key is a @-prefixed name: a
local column, a rename, or a relation. The return type is derived from
the projection literal (ProjectedRowOf), and an invalid key is a
compile error at that key (ValidProjection) rather than a runtime
throw.
// Pick columns:
const slim = await db.repo('Users').find(undefined, {
project: { '@id': true, '@displayName': true },
});
slim.data[0]; // { id: string; displayName: string }
// Rename with a string value ('@title' → headline):
const renamed = await db.repo('Links').find(undefined, {
project: { '@slug': true, '@targetUrl': 'url' },
});
renamed.data[0]; // { slug: string; url: string }Columns marked hidden() are excluded from default reads but remain
projectable: name one explicitly to include it.
A relation key takes true (the whole relation in the target's
default-read shape), a string (the whole relation, renamed), or a
nested { '@col': true | 'rename' } sub-projection. Projections are
depth-1 by construction: a whole-relation target expands to its own
local default shape and never recurses further.
// belongsTo → object | null; sub-projection with a rename:
const links = await db.repo('Links').find({ '@Owner.@role': 'viewer' }, {
project: {
'@slug': true,
'@Owner': { '@displayName': 'owner' },
},
});
links.data[0]!.Owner; // { owner: string } | null
// hasMany reverse relation → array:
const adaView = await db.repo('Users').findOne({ '@id': adaId }, {
project: {
'@id': true,
'@Links': { '@slug': true }, // owns these
'@CreatedLinks': { '@slug': true }, // created these
},
});
adaView.data?.Links; // { slug: string }[]
adaView.data?.CreatedLinks; // { slug: string }[]A projection that names only relations is rejected, because the
JSON-aggregate SELECT needs at least one local grouping key. Include a
local column such as '@id': true alongside the relations.
Relations are declared as foreign keys that reference the target's registry key (see Schema definition). norm plans the joins through OQL's JOIN Support and resolves three cardinalities in query results:
-
belongsTo, the FK side: always
object | null(LEFT join). -
hasOne, a reverse relation where the FK columns equal the source's
primary key:
object | null. -
hasMany, any other reverse relation: an array (
[]when none).
Project a relation by naming its alias. A belongsTo is named by the FK
key; a reverse relation by the FK's reverseAs or the derived name:
// hasOne reverse, sub-projected:
const withProfile = await db.repo('Users').findOne({ '@id': adaId }, {
project: { '@id': true, '@Profile': { '@bio': true } },
});
withProfile.data?.Profile; // { bio: string } | nullA FK declared project: true (belongsTo) or reverseProject: true
(hasOne reverse) is eager: a projection-less read carries it as the
target's local default row. Eager never recurses:
// Users.Profile is an eager hasOne; a default read carries it:
const adaFull = await db.repo('Users').getByPK({ id: adaId });
adaFull.data?.Profile; // the profile row, or null when absentModel a many-to-many as a VIEW that joins the junction to the far
table and declares a logical foreign key back to the near entity. The
relation then reads like any other, in one call and one SELECT, with no
junction pivoting:
// A tags_of_posts VIEW carries a logical fk to Posts (reverseAs 'Tags'):
const posts = await db.repo('Posts').find(undefined, {
project: { '@id': true, '@title': true, '@Tags': { '@name': true } },
orderBy: { '@id': 'ASC' },
});
posts.data[0]!.Tags; // { name: string }[]
// Filtering THROUGH the M2M rides the EXISTS lift (relation not projected):
const denoPosts = await db.repo('Posts').find({ '@Tags.@name': 'deno' });find's aggregates option runs grouped report queries on the typed
surface, with no raw IR and no hand-written GROUP BY. Each entry
names a function and a local @column:
type AggregateInput = Record<
string,
{ fn: 'COUNT' | 'SUM' | 'AVG' | 'MIN' | 'MAX'; column: `@${string}` }
>;The projection keys become the GROUP BY; OQL groups every
non-aggregated projection key. Aggregate outputs land as the driver
returns them, numbers on SQLite and BIGINT/NUMERIC strings on Postgres
and MariaDB, so coerce with Number(...) at the call site.
// Grouped: visits per country.
const perCountry = await db.repo('Visits').find(undefined, {
project: { '@country': true }, // the GROUP BY key
aggregates: { visits: { fn: 'COUNT', column: '@id' } },
orderBy: { '@country': 'ASC' },
});
perCountry.data[0]; // { country: string; visits: number | string | bigint | null }
Number(perCountry.data[0]!.visits);
// Aggregate-only (no projection keys): a plain SELECT COUNT/MAX, one row.
const summary = await db.repo('Visits').find(undefined, {
aggregates: {
total: { fn: 'COUNT', column: '@id' },
latest: { fn: 'MAX', column: '@id' },
},
});
Number(summary.data[0]!.total);Aggregates are validated before any SQL and rejected when they would be meaningless or ambiguous:
- combined with
total: true(the total of a grouped query is its row count); - combined with relation projections (relation reads are separate queries; group over local columns);
- combined with mask columns (masks compute per row, not per group);
- targeting an encrypted column (randomized ciphertext never groups) or a non-physical or unfilterable column;
- an unknown function, a non-
@columntarget, or an alias that collides with a projected key.
A grouped find() is not exempt from the default page size. With no
explicit limit it is capped at the entity's defaultPageSize (10
unless declared), which for a report means the first ten groups, and a
truncated report looks exactly like a complete one. Ask for the page
you want:
// Every group. Know your cardinality first — this is an unbounded read.
const all = await db.repo('Visits').find(undefined, {
project: { '@country': true },
aggregates: { visits: { fn: 'COUNT', column: '@id' } },
limit: 0,
});
// Or page the report explicitly.
const top = await db.repo('Visits').find(undefined, {
project: { '@country': true },
aggregates: { visits: { fn: 'COUNT', column: '@id' } },
orderBy: { '@country': 'ASC' },
limit: 100,
});When a grouped read fills the default page exactly, norm emits a
warning event with the code grouped-page-cap:
norm.on('warning', (entity, op, code, message) => {
if (code === 'grouped-page-cap') logger.warn({ entity, op }, message);
});limit: 0 emits the usual unbounded-read warning instead, and an
explicit limit emits neither, because the page size was your
decision.
Page with limit and offset, and set total: true to also receive
the full match count. The total is a second COUNT sharing the same
rewritten filter and joins:
const page = await db.repo('Links').find(undefined, {
orderBy: { '@slug': 'ASC' },
limit: 10,
offset: 10,
total: true,
});
page.count; // 10 — rows on THIS page
page.total; // 25 — all matching rows
page.data[0]!.slug; // 'link-10'count reflects the current page and total every matching row.
Order by a stable key when paging so successive windows do not overlap.
Ordering by a to-many relation requires projecting it: an unprojected
to-many runs as an EXISTS subquery, which has no ordering scope.
Write methods live on Repo (TABLE entities). Validation, hooks,
encryption, and defaults are covered in
Schema definition; this section has the query-side
essentials.
Accepts a single row or an array (batch). Returns the inserted rows
(RETURNING), decrypted, with hidden() columns stripped and virtual
masks computed. count is the number of returned rows.
const created = await db.repo('Users').insert({
email: 'Ada@Shortly.dev',
apiKey: 'ak-ada-0001',
displayName: 'Ada',
passwordHash: 'bcrypt$ada',
});
created.data[0].email; // 'ada@shortly.dev' (decrypted, normalized)
const batch = await db.repo('Links').insert([
{
id: 1,
slug: 'link-01',
targetUrl: 'https://x.dev/1',
ownerId,
createdById,
},
{
id: 2,
slug: 'link-02',
targetUrl: 'https://x.dev/2',
ownerId,
createdById,
},
]);
batch.count; // 2update(data, filter?) updates matching rows and returns a count-only
envelope (count is the affected rows; no data). Omitting the filter
updates all rows and emits a warning event; pass an explicit {} to
silence it when you mean all rows. Hashed columns filter by plaintext
equality here too.
await db.repo('Users').update({ loginCount: 1 }, {
'@email': 'ada@shortly.dev',
});
await db.repo('Users').updateByPK({ displayName: 'Ada L.' }, { id: adaId });
await db.repo('Links').update({ isActive: false }, {}); // ALL rows, no warningSymmetric with update: delete(filter?) returns a count-only envelope,
a missing filter deletes all rows with a warning, and {} silences it.
await db.repo('PostTags').deleteByPK({ postId: 1, tagId: 2 });
await db.repo('PostTags').delete({}); // explicit all-rows, no warningupsert(data, { conflictKeys, updateOnConflict?, update?, decrypt? })
inserts each row or, when a row conflicts on conflictKeys, updates the
existing one, and returns the resulting rows. What a conflict writes
depends on the options:
| Options | On conflict |
|---|---|
| neither | The columns you passed, copied from the incoming row, plus the defaultOnUpdate columns |
updateOnConflict: [..] |
Exactly those columns, copied, plus the defaultOnUpdate columns |
update: {..} |
That payload, prepared like update() (validation, beforeUpdate, defaultOnUpdate); expressions see the existing row |
updateOnConflict: [] |
Nothing: the existing row is left as it is and returned |
A conflict never changes the primary key, the conflict keys, an
.insertOnly() or norm-owned column, or a column whose value came only
from an insert default (a generated id, CreatedOn, Status: 'ACTIVE').
Naming one of them in updateOnConflict or update throws
UPSERT_CONFLICT_KEY.
When a conflict has nothing to write (you passed only keys and insert-only
columns, or updateOnConflict: []), the row is untouched: no
defaultOnUpdate stamp and no audit version. count includes only the
rows actually written.
An update payload suits counters, since its expressions read the
existing row:
await db.repo('LinkStats').upsert({ linkId, day, clicks: 1 }, {
conflictKeys: ['linkId', 'day'],
update: { clicks: { $$_expression: 'ADD', args: ['@clicks', 1] } },
});Further rules:
- Every row of a batch must pass the same columns, or the upsert throws
UPSERT_BATCH_SHAPE. Passnullwhere you mean null, or upsert rows one at a time. - An entity with a
beforeUpdatehook cannot copy columns on conflict, because the hook cannot see them. Passupdate: {..}, which the hook runs on. -
updateapplies the same payload to every conflicting row of a batch, and adefaultOnUpdatefunction runs once per statement, as inupdate(). - Copied values come from the prepared insert row, after
beforeInsert.
An encrypted column can never be a conflict key, since its ciphertext is
nondeterministic; use its <col>_hash sibling. Updating an encrypted and
hashed column re-syncs its digest sibling so plaintext lookups keep
working.
On a db.scope(...) handle, upsert enforces the scope like insert
and update: the scope column is filled in, a contradicting payload is
rejected, and the write can never adopt or overwrite another scope's
row. See
Scoping
for the pre-flight probe that enforces this on every dialect and the
unique shape that turns it into a schema invariant.
Removes every row. Count-only envelope (op: 'TRUNCATE', count: 0);
pass { cascade: true } where the dialect supports it.
await db.repo('Visits').truncate();
await db.repo('Links').truncate({ cascade: true });truncate refuses on a db.scope(...) handle: TRUNCATE carries no
WHERE, so it would empty every scope's rows. Use delete({}) to
clear only the current scope. See Scoping.
When the typed surface cannot express a query, drop to one of two
escape hatches. Both bypass the typed pipeline (no scope, no
validation, no hashed-filter rewrite), and raw also skips
decryption.
Runs a hand-built OQL IR through the dialect translator. Rows come back
raw by default; bind to an entity to ride the read pipeline's decrypt,
decode, and afterRead column transforms:
// Entity-bound: email comes back decrypted.
const bound = await db.query({
type: 'SELECT',
table: 'users',
columns: ['id', 'email'],
projection: { '@id': true, '@email': true },
where: { '@id': adaId },
}, { entity: 'Users' });
bound.data[0]!.email; // 'ada@shortly.dev'
// Unbound: ciphertext stays ciphertext.
const raw = await db.query({
type: 'SELECT',
table: 'users',
columns: ['id', 'email'],
projection: { '@id': true, '@email': true },
where: { '@id': adaId },
});Binding does not apply hashed-filter rewrites (the IR already ran),
masks, or the afterRead row hook.
Runs a hand-written SQL string with named :param: placeholders, bound
to the connection or transaction. Rows come back exactly as the driver
returns them (no decrypt, no afterRead), and the call emits a
warning event so audits can see the escape hatch in use. SQL engines
only; MongoDB throws NormUnsupportedError.
Always pass values through params and never interpolate into the
string. Parameterized values are the injection-safe path.
const r = await db.raw<{ n: number | bigint }>(
'SELECT count(*) AS n FROM users WHERE role = :role:',
{ role: 'viewer' },
);
Number(r.data[0]!.n);
// Encrypted columns stay ciphertext through raw — decrypt by hand:
const rows = await db.raw<{ email: string }>(
'SELECT email FROM users WHERE id = :id:',
{ id: adaId },
);
const plain = await db.decrypt(rows.data[0]!.email);- Schema definition: columns, entities, relations, hooks, validators, encryption markers.
- Security: encryption, digest columns, masks, and the crypto override hooks.
- Scoping: tenant scoping and always-on default filters that ride every read and write.
-
Migrations: the
Migratorworkflow.