-
-
Notifications
You must be signed in to change notification settings - Fork 72
Bichon v2.x Migration Guide
v2.x replaces the Fjall blob storage engine with bichon-blob, a custom log-structured storage engine. Two migration paths are provided:
- v0.3.7 → v2.x — full migration: Tantivy indexes rebuilt, metadata migrated, blobs converted
- v1.x → v2.x — blob-only migration: Fjall keyspaces copied to bichon-blob, indexes and metadata left untouched
Both migrations are non-destructive — legacy files are never modified.
- Stop the bichon server completely. Migration must not run while the server is writing data.
- Use the latest v2.x release. Newer releases may include migration fixes and performance improvements.
- Ensure sufficient disk space. The new storage will coexist with legacy data during migration. For a 10 GB mailbox, expect ~10 GB of additional temporary space.
<bichon-root-dir>/
├── envelope/ # Tantivy — envelope metadata
├── eml/ # Tantivy — raw message bodies
├── meta.db # native_db — accounts, users, tokens
└── mailbox.db # native_db — mailboxes
<bichon-root-dir>/
├── bichon-indices/
│ ├── mail_metadata/ # Tantivy
│ └── attachment_metadata/ # Tantivy
├── bichon-storage/ # Fjall
│ ├── keyspaces/
│ ├── 0.jnl
│ ├── lock
│ └── version
└── memdb/ # Metadata
<bichon-root-dir>/
├── bichon-indices/
│ ├── mail_metadata/ # Tantivy
│ └── attachment_metadata/ # Tantivy
├── bichon-storage/
│ └── blobs/ # bichon-blob
│ ├── index.redb
│ ├── meta.bin
│ └── 00000001.seg
├── memdb/ # Metadata
└── STORAGE_VERSION # Contains "2"
If --bichon-index-dir or --bichon-data-dir were customized, bichon-indices/ and
bichon-storage/ are created under those directories respectively. memdb/ and
STORAGE_VERSION are always under bichon-root-dir.
Run bichon-admin and select:
Migrate Legacy v0.3.7 Storage to v2.x (bichon-blob)
The tool will prompt for:
| Prompt | Default | Description |
|---|---|---|
--bichon-root-dir |
(required) | Same value used by the old server |
--bichon-index-dir |
<root>/envelope |
Legacy envelope Tantivy index |
--bichon-data-dir |
<root>/eml |
Legacy EML Tantivy index |
| Batch size | 3000 | Controls memory usage per batch |
Migration proceeds in two steps:
Step 1 — Metadata migration (usually completes in seconds):
Migrates accounts, users, roles, tokens, mailboxes, and settings from meta.db /
mailbox.db (native_db) into memdb/.
Step 2 — Email index and blob migration:
Rebuilds Tantivy indexes under bichon-indices/ and writes email/attachment blobs
to bichon-storage/blobs/. Progress is shown per EML segment (Phase 1 scans envelope
metadata, Phase 2 ingests message bodies). Tantivy commits and segment merging happen
at the end of each batch and may take several minutes.
Memory usage depends on both batch size and the total size of your mailbox. Tantivy index writers each hold a 256 MB in-memory buffer, and the Phase 1 envelope lookup map scales with document count. Larger mailboxes will consume more RAM at the same batch size.
| Batch Size | Approx. RAM | Notes |
|---|---|---|
| 1000 | ~500 MB | Safe for low-memory servers |
| 3000 | ~1 GB | Recommended default |
| 5000 | ~2 GB | Faster on high-memory servers |
For very large mailboxes, the full migration can take several hours. Reducing batch size lowers memory but increases total time.
Run bichon-admin and select:
Migrate v1.x Storage to v2.x (Fjall → bichon-blob)
The tool will prompt for:
| Prompt | Default | Description |
|---|---|---|
--bichon-root-dir |
(required) | Same value used by the old server |
--bichon-data-dir |
<root> |
Directory containing bichon-storage/
|
| Batch size | 1000 | Controls memory usage per batch |
This migration copies email and attachment blobs from the Fjall keyspaces into a
new bichon-blob engine at bichon-storage/blobs/. Tantivy indexes and memdb are
not touched — the server will use them as-is after migration.
Unlike the v0.3.7 path, this migration does not involve Tantivy — it is a simple blob copy. Memory usage depends only on batch size and average blob size, not on total mailbox size. At 1000 blobs with an average email size of 75 KB, peak memory is ~75 MB. Adjust upward if you have ample RAM and want faster migration.
If migration needs to be re-run from scratch, delete the output directories first:
v0.3.7 → v2.x (default paths):
rm -rf <bichon-root-dir>/bichon-indices
rm -rf <bichon-root-dir>/bichon-storage
rm -rf <bichon-root-dir>/memdb
rm -f <bichon-root-dir>/STORAGE_VERSIONv1.x → v2.x:
rm -rf <bichon-data-dir>/bichon-storage/blobs
rm -f <bichon-root-dir>/STORAGE_VERSIONIf custom --bichon-index-dir or --bichon-data-dir were configured, adjust paths accordingly.
Warning: Only delete the new output directories. Do not delete legacy
envelope/,eml/,meta.db,mailbox.db, or the v1.xbichon-storage/Fjall files — they are the migration source and must remain intact.
-
Start the v2.x server — it detects
STORAGE_VERSION=2automatically. - Verify that emails and attachments are accessible and searchable.
- Clean up legacy files once the new server is stable:
Default paths:
rm -rf <bichon-root-dir>/envelope
rm -rf <bichon-root-dir>/eml
rm -f <bichon-root-dir>/meta.db
rm -f <bichon-root-dir>/mailbox.dbIf --bichon-index-dir was customized, the directory contains a mix of legacy Tantivy
segment files (.fast, .fieldnorm, .idx, .pos, .store, .term, meta.json,
.managed.json, .tantivy-*.lock) alongside the new bichon-indices/ subdirectory.
Remove all legacy Tantivy files — keep only bichon-indices/. Same applies to
--bichon-data-dir: remove legacy Tantivy files, keep bichon-storage/.
The new bichon-storage/blobs/ coexists with the old Fjall files. Remove the Fjall
files, keeping only the blobs/ directory:
rm -rf <bichon-data-dir>/bichon-storage/keyspaces
rm -f <bichon-data-dir>/bichon-storage/0.jnl
rm -f <bichon-data-dir>/bichon-storage/lock
rm -f <bichon-data-dir>/bichon-storage/versionAfter cleanup, bichon-storage/ should contain only the blobs/ subdirectory.
Warning: Double-check all paths before running
rm. Keep the legacy data for a few days after migration as a fallback.