Import Export Format

Import / Export Format (portable bundle v1)

Export from a started project

nself db export --out <dir> [--schemas a,b] [--no-storage] [--json] reads a started project's database under one exported snapshot and writes this format. Objects come from the active store's loopback S3 port (SeaweedFS in v1.5). The manifest maps each bucket/key to its content-addressed member; all object writes use the portable writer. Config lists names only; Hasura source connection configuration is omitted. Plugin-owned tables and schemas are excluded. The scrubber fails closed on read/render errors or a secret in text/SQL (E526), removing staging output. Auth bcrypt hashes are allowed; user object bytes are not scanned. Nonempty output (E527) and insufficient disk (E525) are refused before writing. See Leaving-nSelf for restore instructions and limitations.

The portable bundle is the vendor-neutral directory that nself db export writes and every importer reads. It is contract cli.portable-export v1. The manifest schema is generated: schemas/portable-export.v1.schema.json.

Library: internal/portable (Writer, Reader, Manifest), internal/postgres/tablehash (count and hash per table), internal/objstore (S3 client for bucket contents).

Layout

<bundle>/
  manifest.json                 schema: schemas/portable-export.v1.schema.json
  db/schema.sql                 pg_dump --schema-only --no-owner --no-privileges (for humans)
  db/data.dump                  pg_dump -Fc --no-owner --no-privileges (schemas in manifest db.schemas)
  auth/users.jsonl              one JSON object per user, sorted by id
  storage/objects/<sha256 hex>  object bytes (see "Object names")
  hasura/metadata.json          export_metadata v3 subset for db.schemas (tables: definition, configuration, relationships, computed fields, permissions; event triggers, remote relationships, remote schemas, actions and connection configuration are not exported)
  config/keys.txt               sorted config key NAMES, never values (nself sources only)
  db/compat.sql                 optional source shims applied before pre-data

The bundle directory is mode 0700 and its files 0600. A bundle never holds a secret value: config/keys.txt lists key names only, and the exporter scrubs the text files before it finishes.

manifest.json

Field	Meaning
_format	always nself-portable-export
schema_version	"1"; readers accept 1 and 1.<n>
created_at	RFC 3339; the only time-dependent field
producer	tool, version, source (nself, postgres, nhost, hasura-cloud, supabase, firebase, appwrite)
postgres	server_version, major of the source
db	schemas, and tables[] with schema, name, rows, hash, pk; (schema, name) is unique and hash is canonical (0, or a decimal with no leading zero, no +, no -0)
auth	users, hash_algorithms (algorithm to count; keys bcrypt, unknown for a stored hash in another format, none for no hash), reset_required
storage	buckets[] (name, objects, bytes) and objects[] (bucket, key, member, sha256, bytes); (bucket, key) is unique and every object names a listed file with the same sha256 and bytes
hasura	counts of tables, relationships, permissions, functions, and skipped kinds
exemptions	items deliberately left out: kind (table, user, object, metadata), id, reason
source_counts	the source's own counts with the method used
compat	source shims the bundle needs
files	every member except manifest.json: path, sha256 (lowercase hex of the exact bytes), bytes

The manifest is deterministic: keys are sorted (Go marshals maps in key order), arrays are sorted by their natural id (db.tables by schema and name, files by path, and so on; a table's pk keeps the key's column order), two-space indent, one trailing newline. Two runs over the same input write the same bytes. Only created_at changes with time.

auth/users.jsonl

One JSON object per user: id, email, email_verified, phone, display_name, created_at, disabled, password_hash, hash_algorithm, metadata, roles and default_role. default_role is optional and additive inside contract v1 (schema_version stays "1"): a record without it, from an older bundle or a source with no such notion, loads as user. The loader registers the role in auth.roles before it loads the user. The verifier reads the target back and reports a user mismatch when a user's default role, email_verified, disabled or role set differs from the record (a target may add its own default role to the set).

Table hash

rows is count(*). hash is, as a decimal string:

SELECT count(*), coalesce(sum(hashtextextended(t::text, 0)::numeric), 0)::text FROM "schema"."table" AS t

The sum makes it independent of row order and of how the rows were inserted. An empty table hashes to "0". A changed value, an extra row or a duplicated row changes it.

t::text depends on session settings, so tablehash.Snapshot sets these for the session before it hashes, and restores the caller's values afterwards: TimeZone=UTC, DateStyle=ISO, IntervalStyle=postgres, extra_float_digits=3, bytea_output=hex, lc_monetary=C, statement_timeout=0, row_security=off and search_path=pg_catalog. The same fixture gives the same count and hash on Postgres 14, 15, 16 and 17 and under any session default of those settings.

• row_security=off is what pg_dump runs. A role that cannot read every row of a table gets the error "query would be affected by row-level security policy"; it never gets a hash of the rows it can see. Run the hash as a role that owns or bypasses the tables.
• search_path=pg_catalog makes regclass, regtype, regproc and the other reg* columns print schema-qualified, so the text form does not depend on the session's search_path.
• The database must be UTF8. The hash is over the server-encoding bytes, so the same text hashes differently in a LATIN1 database; Snapshot refuses a non-UTF8 database with ErrNotUTF8 before it queries anything.

Schema and table names must match ^[A-Za-z_][A-Za-z0-9_]{0,62}$. A table outside that shape is never queried; it comes back as unverifiable and the verify report lists it. Names are quoted with pgx.Identifier after the check.

Limits of the definition: the hash covers the row text in column order, so a target table must have the same columns in the same order as the source. It does not cover indexes, constraints or sequence values.

Round trip: import an exported bundle

nself db import nself <dir> [--plan] [--keep-stage] [--skip-storage] [--json] reads a bundle written by nself db export into a fresh started project. It is the shared importer (see Leaving-nSelf for the other direction) with one thin source:

1. The whole bundle is verified first (portable.Open): an unknown format or major is E515, a changed, missing, extra or unsafe file is E516 and names the member. This happens in --plan too, before the target is touched, and also on a target that already holds data.
2. The bundle is copied into the private stage .nself/import/<run>/stage byte for byte (the manifest is never rewritten; a hard link is not used because the reader refuses linked files). The stage is verified again and its manifest must equal the one verified in your directory, so a bundle that changes while it is staged is refused.
3. The shared loader leaves the platform's own public.uuidv7() out of the restore (every started project has it), restores the database, loads users (bcrypt hashes are kept, so users sign in with their original passwords), uploads the objects, restores constraints and applies the Hasura metadata, then the verifier compares tables (row count and hash), users, objects by SHA-256, tracked tables, relationships and permissions. The result is pass only with zero mismatches.

The integrity checks prove a bundle is intact, not that it is safe: the dump's SQL runs as the database owner, so import only bundles you trust. A bundle that carries db/compat.sql is refused (E516), because nself db export never writes one. Object keys are compared in the object store's canonical form (a//b and /a/b are a/b), so a key cannot slip past the overwrite check; a key with a . or .. segment is refused.

The target must be a fresh project with the same plugins installed: plugin-owned data (np_*) is not part of a bundle, and a target that already has the bundle's tables or users, or an object under one of the bundle's keys, is refused with E519 before any write (nothing is overwritten). The target's Postgres major must be at least the bundle's (E518). Run scripts/e2e/export-round-trip.sh for the end to end proof: a seeded project is exported, imported into a second project and checked for zero mismatches, a second import (E519), a flipped byte (E516) and schema_version 2 (E515).

Reading a bundle (untrusted input)

portable.Open(dir) treats the directory as untrusted and refuses it unless every check passes:

Failure	Code
no manifest.json, not JSON, wrong _format, schema_version not major 1 (for example "2")	E515
manifest does not match the v1 layout, has an unknown field (for schema_version 1 or 1.0), has data after the JSON object, repeats a key, has two keys that differ only in case, or spells a field in another case	E516
a table listed twice, an object listed twice, an object that names no listed file or a file with a different sha256 or bytes, a file under storage/objects/ that no object names, or a non-canonical table hash	E516
a listed file is missing, truncated, longer than listed, or its SHA-256 differs (the error names the file)	E516
a member path that is absolute, uses .., ., //, a backslash, a drive letter or :, a control character, a Windows device name, ends in a dot or space, or is not in Unicode NFC	E516
two members with the same path (compared after NFC normalisation and Unicode case folding), or a path that is both a file and a directory	E516
a symlink anywhere in a member's path, a symlink in the bundle, a hard link (counted from the file handle on Unix and on Windows), or a member that is not a regular file	E516
a member that is not the file that was verified when it is opened later (different device and inode, or Windows file ID, size or modification time)	E516
an entry in the directory that the manifest does not list	E516
more entries than MaxFiles, counting directories as well as files	E516
a declared size or count over the limits (defaults: manifest 64 MiB, 1,000,000 files, 1 TiB per file, 8 TiB total)	E516

Reading is streamed and capped at the declared length, so a file cannot inflate past what the manifest says. Reader.Open(rel) returns a reader that re-checks length and SHA-256 as it streams and returns E516 instead of io.EOF when the bytes changed since Open. Reader.OpenFile(rel) returns the open file for tools that need a descriptor (pg_restore can read standard input): it opens without following a symlink (O_NOFOLLOW where the platform has it) and refuses unless the file has the device and inode (Windows: volume and file index), size and modification time recorded while the bundle was verified. There is no path accessor, because a path can be swapped between the check and the use. Error messages name paths only; they never print bundle content.

Because the reader refuses unlisted entries, do not add files to a bundle directory (editor backups, .DS_Store). Copy the directory as is.

Object names

Object keys are arbitrary and case- and Unicode-sensitive; file systems are not. portable.StorageMember(bucket, key) returns storage/objects/<hex sha256(bucket + "\x00" + key)>, so every (bucket, key) pair has its own member on every platform: Readme.md and README.md, or the NFC and NFD spellings of café.jpg, export on a case-insensitive or normalising file system as they do on S3. The manifest objects[] holds the real bucket, key, member, sha256 and bytes; the importer maps key to file with StorageMember (or reads member) and never parses the file name. A key may not be empty or invalid UTF-8, and a bucket may not be empty or contain NUL. portable.NewObject(bucket, key, file) builds the entry from the File that Writer.WriteFile returned.

Object store client

internal/objstore talks S3 in path style with stdlib SigV4: EnsureBucket, Put, Get, List (continuation-token pagination, at most MaxListPages pages and MaxListObjects objects), ListPage. Listing sends encoding-type=url and decodes the keys, so a key with a character XML cannot carry still lists. The canonical request is built from the escaped path and the raw query without form decoding (a literal + signs as %2B). The default HTTP client has dial, TLS and response-header timeouts and no deadline on the whole request, so a large object can take as long as it needs; bound a call with its context. Client and Signer print with the secret key redacted. It is tested against SeaweedFS (chrislusf/seaweedfs) and the MinIO compatibility image (pgsty/minio, ADR 0028), over a direct and a loopback endpoint, with 1,100 keys that contain spaces, Unicode, + and %. A PUT streams with UNSIGNED-PAYLOAD only over plain http to a loopback endpoint; every other PUT signs the payload hash. Errors never contain the secret key (raw, or percent-encoded in any hex case and any mix of literal and encoded bytes) or a signature; redaction runs before an error message is truncated.

Evolution

Additive changes (a new optional field, a new producer.source) stay v1 and are released as a minor, schema_version 1.<n>. A reader refuses a field it does not know in a 1 or 1.0 manifest, and ignores one in a 1.<n> (n > 0) manifest. A rename or removal is v2: readers refuse an unknown major with E515 and nothing is read.

---

Home