Skip to content

Bichon Data Migration: v0.3.7 → v1.0

Jump to bottom
rustmailer edited this page May 22, 2026 · 2 revisions

Overview

Bichon v1.0.0 introduces a new storage architecture that separates email indexes (Tantivy) from raw message and attachment data (Fjall). Migration is required for all v0.3.7 installations. The migration is non-destructive — legacy files are left intact.

Directory Structure

Before (v0.3.7)

<bichon-root-dir>/
├── envelope/          # Tantivy index — envelope metadata
└── eml/               # Tantivy index — raw message bodies + attachments

After (v1.0.0)

<bichon-root-dir>/
├── envelope/          # (legacy, untouched)
├── eml/               # (legacy, untouched)
├── bichon-indices/
│   ├── mail_metadata/       # Tantivy index — email metadata + full-text
│   └── attachment_metadata/ # Tantivy index — attachment metadata
└── bichon-storage/
    ├── keyspaces/       # Fjall keyspaces — email bodies & attachment blobs
    ├── 0.jnl            # Fjall journal file
    ├── lock             # Fjall lock file
    └── version          # Fjall version file

If --bichon-index-dir or --bichon-data-dir were customized, bichon-indices/ and bichon-storage/ are created under those directories respectively.

Before Starting the Migration

Before running the migration tool:

  1. Stop the existing v0.3.7 bichon server completely

    • Ensure no bichon process is still running.
    • Migration must not be performed while the old server is actively writing data.

  2. Use the latest available v1.x.x release

    • It is strongly recommended to use the latest bichon and bichon-admin versions, as newer releases may include migration fixes, memory optimizations, or data consistency improvements.

  3. Run the migration using bichon-admin.

  4. After the migration finishes successfully:

    • Start the new v1.x.x bichon server.
    • It is recommended to keep the legacy data for a few days after migration. Only remove the old data once the new server has been running stably and the migration has been fully validated.

Warning: Do not start the v1.x.x server before migration has completed successfully.

Running the Migration

Run bichon-admin and select: Migrate Legacy v0.3.7 Storage to v1.0.0

Note: bichon-admin v1.0.0 has a known memory issue during migration of large mailboxes. Please use bichon-admin v1.0.1 which includes an emergency fix. The menu option still shows v1.0.0 in the label — this is a cosmetic issue, simply select it as normal.

The tool will prompt for:

  1. --bichon-root-dir — same value used by the old server
  2. --bichon-index-dir — leave blank to use default (<root>/envelope)
  3. --bichon-data-dir — leave blank to use default (<root>/eml)
  4. Batch size — controls memory usage per batch (see below)

Migration proceeds in two steps:

Step 1 — Metadata migration (usually completes in seconds):

Mail Settings done (0 items) Accounts done (1 items) OAuth2 Entities done (0 items) OAuth2 Pending done (0 items) OAuth2 Access Tokens done (0 items) Proxy Settings done (0 items) User Roles done (5 items) Users done (1 items) Access Tokens done (2 items) Mailboxes done (1 items) Metadata migration completed successfully.

Step 2 — Email index and blob migration: proceeds segment by segment. Progress is displayed for both the envelope scanning phase (Phase 1) and the ingestion phase (Phase 2).

Memory Usage

Batch size directly controls peak memory consumption. Choose based on your available RAM and average email size (larger attachments = higher memory per batch).

Batch Size Approx. RAM Notes
1000 ~500 MB Safe for low-memory servers
3000 ~1 GB Recommended default
5000 ~2 GB Faster on high-memory servers

After each batch is ingested, Tantivy performs a commit to flush the in-memory buffer to disk. This commit can take 2–3 minutes depending on batch size and disk speed — this is normal, please wait patiently. Segment merging is deferred to the end of migration to avoid mid-process stalls, and may also take several minutes once all batches are complete.

For large mailboxes (hundreds of GB), the full migration can take several hours. Reducing the batch size will lower memory usage but increase the number of commits, which may extend the total migration time.

Re-running the Migration

If migration needs to be re-run from scratch, delete the new output directories before running the tool again.

Default paths (when no custom dirs were configured):

rm -rf <bichon-root-dir>/bichon-indices
rm -rf <bichon-root-dir>/bichon-storage
rm -rf <bichon-root-dir>/memdb

Custom paths (when --bichon-index-dir or --bichon-data-dir were configured):

rm -rf <bichon-index-dir>/bichon-indices
rm -rf <bichon-data-dir>/bichon-storage
rm -rf <bichon-root-dir>/memdb   # memdb is always under bichon-root-dir

Warning: Only delete bichon-indices, bichon-storage and memdb. Do not delete or modify the legacy envelope/ and eml/ directories — they are the migration source and must remain intact. Deleting them will make re-migration impossible without restoring from backup.

After a Successful Migration

  1. Start the v1.0.x server — it will automatically detect the new storage layout and skip migration prompts.
  2. Verify that emails and attachments are accessible and searchable.
  3. Clean up legacy files once you have confirmed the migration is complete and the new server is stable:

Default paths:

rm -rf <bichon-root-dir>/envelope
rm -rf <bichon-root-dir>/eml
rm -f  <bichon-root-dir>/mailbox.db
rm -f  <bichon-root-dir>/meta.db

If --bichon-index-dir was configured, the directory contains a mix of legacy Tantivy segment files (.fast, .fieldnorm, .idx, .pos, .store, .term, meta.json, .managed.json, .tantivy-*.lock) alongside the new bichon-indices/ subdirectory. Remove all the legacy Tantivy files, but keep bichon-indices/ intact.

Same applies to --bichon-data-dir — remove the legacy Tantivy files, keep bichon-storage/.

<bichon-index-dir>/
├── bichon-indices/          ← KEEP — new index
├── .managed.json            ← delete
├── .tantivy-meta.lock       ← delete
├── .tantivy-writer.lock     ← delete
├── meta.json                ← delete
├── 0d4e7e91a66d44e58...fast ← delete
├── 0d4e7e91a66d44e58...store← delete
├── 0d4e7e91a66d44e58...term ← delete
└── ...                      ← delete all other segment files

<bichon-data-dir>/
├── bichon-storage/          ← KEEP — new storage
├── .managed.json            ← delete
├── meta.json                ← delete
└── ...                      ← delete all other segment files

Warning: Double-check paths before running any rm command. If --bichon-index-dir or --bichon-data-dir point to directories that also contain bichon-indices or bichon-storage as subdirectories, remove only the legacy contents, not the parent directory.

Clone this wiki locally