feat: new vault format

.markdownlint.json

@@ -3,7 +3,7 @@
   "MD013": false,
   "MD024": false,
   "MD033": {
-    "allowed_elements": ["Tabs", "TabItem", "Steps", "Aside", "Card", "CardGrid", "LinkCard", "AsciinemaPlayer", "DownloadCommands", "VersionOutput", "VersionExample", "a", "details", "summary", "div", "img"]
+    "allowed_elements": ["Tabs", "TabItem", "Steps", "Aside", "Card", "CardGrid", "LinkCard", "AsciinemaPlayer", "DownloadCommands", "VersionOutput", "VersionExample", "a", "details", "summary", "div", "img", "Badge"]
   },
   "MD036": false,
   "MD041": false,

astro.config.mjs

@@ -98,6 +98,7 @@ export default defineConfig({
             { label: 'Philosophy', slug: 'concepts/philosophy' },
             { label: 'Security Model', slug: 'concepts/threat-model' },
             { label: 'Architecture', slug: 'concepts/architecture' },
+            { label: 'Vault Format', slug: 'concepts/vault-format' },
             { label: 'Standards Compliance', slug: 'concepts/compliance' },
             { label: 'Comparison', slug: 'concepts/comparison' },
           ],

src/content/docs/concepts/architecture.mdx

@@ -17,28 +17,34 @@ The vault is a JSONL (JSON Lines) file where each line is a self-contained JSON
 ### File Format
 
 ```text
-Line 1: Header with metadata and indexes
-Line 2+: Identity entries, secret definitions, and secret values
+# === VAULT HEADER v1 ===
+Line 2: Header JSON with metadata and indexes
+# === VAULT DATA ===
+Line 4+: Identity entries, secret definitions, and secret values
 ```
 
 Example structure:
 
 ```json
-{"version":1,"identities":[["ABC123",2],["DEF456",3]],"secrets":{"DATABASE_URL":{"secret":4,"values":[5]}}}
-{"type":"identity","data":{"fingerprint":"ABC123","uid":"alice@example.com","algorithm":"rsa4096"}}
-{"type":"identity","data":{"fingerprint":"DEF456","uid":"bob@example.com","algorithm":"ed25519"}}
-{"type":"secret","data":{"name":"DATABASE_URL","created_by":"ABC123","created_at":"2025-01-01T00:00:00Z"}}
+{"version":1,"identities":[["ABC123",4],["DEF456",5]],"secrets":{"DATABASE_URL":{"secret":6,"values":[7]}}}
+{"type":"identity","data":{"fingerprint":"ABC123","uid":"alice@example.com","algorithm":"RSA","algorithm_bits":4096}}
+{"type":"identity","data":{"fingerprint":"DEF456","uid":"bob@example.com","algorithm":"EdDSA","curve":"Ed25519"}}
+{"type":"secret","data":{"key":"DATABASE_URL","signed_by":"ABC123","added_at":"2025-01-01T00:00:00Z"}}
 {"type":"value","secret":"DATABASE_URL","data":{"available_to":["ABC123","DEF456"],"value":"encrypted..."}}
 ```
 
-### Header (Line 1)
+<Aside type="tip">
+  See [Vault Format](/concepts/vault-format) for details on format versioning.
+</Aside>
+
+### Header (Line 2)
 
 The header provides fast lookup without scanning the entire file:
 
 | Field | Purpose |
 |-------|---------|
 | `version` | Format version (currently 1) |
-| `identities` | Map of fingerprint to line number |
+| `identities` | Array of `[fingerprint, line]` pairs (sorted by line number) |
 | `secrets` | Map of secret name to definition line and value lines |
 
 ### Entry Types

src/content/docs/concepts/vault-format.mdx

@@ -0,0 +1,167 @@
+---
+title: Vault Format
+description: Understanding dotsecenv vault file format and structure
+---
+
+import { Aside, Badge } from '@astrojs/starlight/components';
+
+## Overview
+
+dotsecenv vaults use a versioned JSONL format. The format includes a version field in the header JSON, allowing the format to evolve over time while maintaining backward compatibility.
+
+## Current Format (v1) <Badge text="Current" variant="success" />
+
+The vault file consists of:
+
+1. **Header marker line** - Identifies the file as a dotsecenv vault
+2. **Header JSON** - Index for efficient lookups (identities, secrets)
+3. **Data marker line** - Separates header from data
+4. **Data entries** - One JSON object per line (identities, secrets, values)
+
+### Structure
+
+```text
+# === VAULT HEADER v1 ===
+{"version":1,"identities":[["fingerprint1",4],["fingerprint2",5]],"secrets":{...}}
+# === VAULT DATA ===
+{"type":"identity","data":{...}}
+{"type":"secret","data":{...}}
+{"type":"value","secret":"KEY","data":{...}}
+```
+
+### Header Format
+
+Identities are stored as an ordered array of `[fingerprint, line]` pairs, sorted by line number (order added):
+
+```json
+{
+  "version": 1,
+  "identities": [
+    ["ABC123DEF456", 4],
+    ["XYZ789GHI012", 5]
+  ],
+  "secrets": {
+    "DATABASE_URL": {
+      "secret": 6,
+      "values": [7, 8]
+    }
+  }
+}
+```
+
+| Field | Type | Purpose |
+|-------|------|---------|
+| `version` | `int` | Format version (currently 1) |
+| `identities` | `array` | Array of `[fingerprint, line]` pairs, sorted by line number |
+| `secrets` | `object` | Map of secret name to definition line and value lines |
+
+### Why Arrays for Identities?
+
+The array format preserves insertion order, which provides:
+
+- **Deterministic output** - Same vault always serializes identically
+- **Git-friendly** - Consistent ordering means cleaner diffs
+- **Audit trail** - Order reflects when identities were added
+
+## Entry Types
+
+### Identity Entry
+
+```json
+{
+  "type": "identity",
+  "data": {
+    "added_at": "2025-01-01T00:00:00Z",
+    "algorithm": "RSA",
+    "algorithm_bits": 4096,
+    "fingerprint": "ABC123DEF456789012345678901234567890ABCD",
+    "hash": "sha256:...",
+    "public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----...",
+    "signed_by": "ABC123DEF456789012345678901234567890ABCD",
+    "signature": "...",
+    "uid": "alice@example.com"
+  }
+}
+```
+
+### Secret Definition
+
+```json
+{
+  "type": "secret",
+  "data": {
+    "added_at": "2025-01-01T00:00:00Z",
+    "hash": "sha256:...",
+    "key": "DATABASE_URL",
+    "signature": "...",
+    "signed_by": "ABC123DEF456789012345678901234567890ABCD"
+  }
+}
+```
+
+### Secret Value
+
+```json
+{
+  "type": "value",
+  "secret": "DATABASE_URL",
+  "data": {
+    "added_at": "2025-01-01T00:00:00Z",
+    "available_to": ["ABC123DEF456...", "XYZ789GHI012..."],
+    "hash": "sha256:...",
+    "signature": "...",
+    "signed_by": "ABC123DEF456789012345678901234567890ABCD",
+    "value": "base64-encoded-encrypted-blob"
+  }
+}
+```
+
+## Version Detection
+
+dotsecenv detects the format version from the `version` field in the header JSON:
+
+```json
+{"version":1,"identities":[...],"secrets":{...}}
+```
+
+This allows efficient version detection by parsing just the header line.
+
+## Line Numbers
+
+Line numbers in the header are 1-indexed (first line = 1):
+
+| Line | Content |
+|------|---------|
+| 1 | Header marker (`# === VAULT HEADER v1 ===`) |
+| 2 | Header JSON |
+| 3 | Data marker (`# === VAULT DATA ===`) |
+| 4+ | Data entries (identities, secrets, values) |
+
+The header indexes point to line numbers where each entry can be found, enabling O(1) lookups.
+
+## Future Versioning
+
+The format version allows dotsecenv to evolve while maintaining compatibility:
+
+- **Backward compatibility** - Read older vaults without data loss
+- **Forward evolution** - Add new features without breaking existing vaults
+- **Automatic migration** - Future versions may include automatic upgrades
+
+<Aside type="note">
+  Currently only v1 format exists. Future format changes will be announced in release notes with migration guidance.
+</Aside>
+
+## Validation
+
+dotsecenv validates vault structure on read:
+
+1. Header marker must match expected format
+2. Header JSON must parse successfully
+3. Data marker must be present
+4. Line numbers in header must point to valid entries
+
+Use the validate command to check vault integrity:
+
+```bash
+dotsecenv validate
+```

src/content/docs/guides/shell-plugins.mdx

@@ -94,10 +94,10 @@ Use `dse` as a shorthand for the `dotsecenv` command:
 
 ```bash
 # Instead of:
-dotsecenv secret list
+dotsecenv vault list
 
 # Use:
-dse secret list
+dse vault list
 ```
 
 ### `secret` — Quick Secret Retrieval

src/content/docs/reference.mdx

@@ -598,12 +598,14 @@ dotsecenv version [flags]
 |------|---------|
 | `0` | Success |
 | `1` | General error |
-| `2` | Invalid arguments |
-| `3` | Config/vault not found |
+| `2` | Configuration error |
+| `3` | Vault error |
 | `4` | GPG error |
-| `5` | Secret not found |
-| `6` | Permission denied |
-| `7` | Validation failed |
+| `5` | Authentication error (not logged in) |
+| `6` | Validation failed |
+| `7` | Fingerprint required |
+| `8` | Access denied |
+| `9` | Algorithm not allowed |
 
 ---
 

src/content/docs/tutorials/revoke-access.mdx

@@ -148,20 +148,17 @@ The vault preserves full history. View who had access to each version:
 dotsecenv secret get DATABASE_PASSWORD --all --json | jq '.values[] | {timestamp, available_to}'
 ```
 
-### Remove identity from vault
+### Revoke all secrets from an identity
 
 If someone should no longer have any access:
 
 ```bash
-# Remove from vault identity list
-dotsecenv vault identity remove FINGERPRINT
-
-# Revoke from all existing secrets
+# Revoke from all existing secrets in all vaults
 dotsecenv secret revoke "*" FINGERPRINT --all
 ```
 
 <Aside type="note">
-Removing an identity doesn't delete their access to previously shared values. It only prevents sharing new secrets with them.
+The identity remains in the vault but cannot decrypt any current secret values. To fully remove an identity from the vault, use `vault defrag` after revoking all access.
 </Aside>
 
 ---
@@ -173,8 +170,7 @@ When a team member leaves:
 - Revoke their access to all secrets
 - Rotate ALL secrets they had access to
 - Update all systems with new credentials
-- Remove their identity from the vault
-- Consider defragmenting the vault (optional)
+- Defragment the vault to remove their identity
 - Document the change in your security log
 
 ```bash
@@ -184,14 +180,14 @@ FINGERPRINT="THEIR_FINGERPRINT"
 # 1. Revoke all access
 dotsecenv secret revoke "*" "$FINGERPRINT" --all
 
-# 2. Remove identity
-dotsecenv vault identity remove "$FINGERPRINT"
-
-# 3. Rotate secrets (do this for each secret)
+# 2. Rotate secrets (do this for each secret)
 echo "new-value" | dotsecenv secret put DATABASE_PASSWORD
 echo "new-value" | dotsecenv secret put API_KEY
 # ... repeat for all secrets
 
+# 3. Defragment to clean up (removes identity entries with no access)
+dotsecenv vault defrag --yes
+
 # 4. Commit
 git add vault
 git commit -m "Offboard: revoke access for $FINGERPRINT"