Proposal: Rust Sync Engine — Eliminating iOS Background Sync Failures at the Architecture Level

[santoshakil]

TL;DR: The iOS background sync bugs (#22850, #22522, #22101, #22320, #20331 — 267 comments combined, all open) share one root cause: Dart isolates can't share memory. A Rust shared library eliminates this by giving foreground and background the same Mutex<Connection> — no locks, no heartbeats, no 5-second timeout. I built a working proof-of-concept with 82 passing tests on real devices.

---

The Root Cause: One Picture

Current — fragile lock coordination:

Foreground Isolate <--heartbeat 3s--> Background Isolate
        |             5s timeout (iOS)        |
        |          IsolateNameServer          |
        v                                    v
    [Isar DB] <-- no coordination --> [Isar DB]
   (same file)    UNIQUE failures    (same file)

5s timeout  = no upload       (#22850)
Stale lock  = re-upload all   (#20331)
Lock stuck  = infinite loop   (#22101)
No state    = no communicate  (#22320)

Proposed — shared Rust engine:

Foreground Flutter          Background Flutter
        |                          |
        | FFI                      | FFI
        v                          v
+------------------------------------------+
|        Rust Sync Engine (static)         |
|        Shared across all callers         |
|                                          |
|  Dart (FFI) -----+                      |
|  Swift (dlopen) --+--> Mutex<Connection> |
|  Kotlin (JNI) ---+    (microseconds)    |
+--------------------+---------------------+
                     |
                     v
              [SQLite WAL mode]
              [single writer  ]

No lock. No heartbeat. No timeout.
Swift/Kotlin call Rust directly — no Flutter engine needed.

---

The Code Causing These Bugs

background.service.dart:

Bug 1: Background gives up after 5 seconds (#22850 — 42 comments)

// line 297-299
final waitForLock = Platform.isIOS
    ? acquireLock().timeout(const Duration(seconds: 5), onTimeout: () => false)
    : acquireLock();

if (!hasAccess) {
    return false;  // ← Background silently exits. No upload.
}

Foreground holds lock → background waits 5s → gives up → user's photos never upload.

Bug 2: False crash detection → concurrent writes (#22522 — 50 comments)

// line 228-237
other.send(tempSp);  // "are you alive?"
final answer = await bs.first.timeout(const Duration(seconds: 3), onTimeout: () => null);
if (answer == null) {
    // Assumes crash. But foreground might just be busy rendering.
    IsolateNameServer.removePortNameMapping(_portNameLock);
    break;  // ← Takes lock. Now BOTH isolates write to DB.
}

Foreground busy rendering → doesn't respond in 3s → background steals lock → both write → UNIQUE constraint failure.

Bug 3: Orphaned lock after kill (#22101, #20331 — 140 comments)

// line 261-269
void releaseLock() {
    IsolateNameServer.removePortNameMapping(_portNameLock);
    // ← Never runs if process is killed by iOS
}

iOS kills background task → releaseLock() never called → next wake finds stale lock → 3s heartbeat timeout → starts from scratch → re-uploads everything.

---

The Rust Solution

All of the above code disappears. Replaced by:

static STORE: RwLock<Option<Arc<dyn AssetStore>>> = RwLock::new(None);

pub struct Database {
    conn: Mutex<Connection>,  // compiler-enforced, not heartbeat-enforced
}

	Current (Dart)	Proposed (Rust)
Lock acquisition	5s timeout, heartbeat, crash detection	Mutex::lock() — microseconds
If foreground is busy	Background assumes crash, steals lock	Background waits on Mutex, gets it when foreground releases
If process killed	Orphaned lock in IsolateNameServer	Mutex dies with process. Next init creates fresh one.
DB writes	Two isolates, no coordination	One Mutex, compiler-enforced
Background cold start	~500ms (full Flutter engine)	<10ms (Rust static init)

---

Why Pure Rust (The Dropbox Approach)

This follows the same principle Dropbox applied when they rewrote their sync engine in Rust ("Nucleus"). Their team called Rust "a force multiplier" and "one of the best decisions they made."

One engine, every language:

Caller	How	When
Dart	dart:ffi	Flutter UI triggers sync
Swift	dlopen	iOS BGTaskScheduler — no Flutter engine needed
Kotlin	JNI	Android WorkManager — no Flutter engine needed
Python	ctypes	Future CLI/server tool

All call the same function, hit the same SQLite, protected by the same Mutex.

What it replaces:

• 4 Dart sync services (~14K lines, 82 files) → 1 Rust engine
• Isar (legacy) + Drift (new) mid-migration → 1 SQLite with WAL
• Platform channel → Kotlin SHA-1 + Platform channel → Swift SHA-1 → 1 Rust hash engine (mmap + SIMD on Android, streaming API for iOS)
• IsolateNameServer lock → gone

What stays unchanged:

• Flutter UI (Riverpod, all screens)
• Pigeon (PHAsset, MediaStore — OS APIs)
• ContentObserver / PHPhotoLibraryChangeObserver

---

Proof of Concept

Built and tested:

What	Status
Rust engine (8 crates, hexagonal architecture)	58 tests passing
Dart FFI plugin (typed errors, event streaming)	24 tests passing
iOS (PHPhotoLibraryChangeObserver + BGTaskScheduler + Rust dlopen)	Verified on iPad (iOS 18.7)
Android (ContentObserver + WorkManager ContentUriTrigger + Rust)	Verified on Samsung tablet (Android 16)
Data persistence across force-stop / app termination	10,020 assets survived kill, zero loss

Performance (real device, debug mode):

Operation	Current Immich	Rust PoC	Improvement
FFI round-trip	N/A	4μs/call	—
Hash 100 files	~500ms	4ms	125x
Sync 10K assets to DB	~3s	54ms	55x
Background cold start	~500ms	<10ms	50x

---

Migration Path

No rewrite needed. Drop-in Flutter plugin:

Phase 1: Add immich_core as dependency, feature-flagged. Run alongside existing sync.
Phase 2: Switch to Rust sync as primary. Dart as fallback.
Phase 3: Remove SyncService, SyncStreamService, BackupService, HashService,
         IsolateNameServer lock, Isar dependency.

---

Why This Can't Be Fixed in Dart

This isn't about Dart being slow. Dart is fine for sync logic. The problem is two Flutter engines sharing mutable state:

1. iOS background = separate Dart isolate
2. Dart isolates share nothing — no objects, no DB connections, no state
3. IsolateNameServer (name→SendPort registry) is the only coordination mechanism
4. Every solution in Dart is a variation of: message passing + timeouts + heuristic crash detection
5. You can tune timeouts and add retries, but you're simulating shared state through messages instead of actually sharing it

A Rust shared library's static is per-process, not per-isolate. Both engines see the same state. The compiler prevents data races. That's the fundamental difference.

---

Happy to share the code and discuss the architecture in detail. I believe this approach would resolve the most painful mobile sync issues while keeping the Flutter UI completely unchanged.

MD Shakil Mia
GitHub · LinkedIn

[github-actions[bot]]

This discussion has automatically been closed as it is likely a duplicate. We get a lot of duplicate threads each day, which is why we ask you in the template to confirm that you searched for duplicates before opening one. If you're sure this is not a duplicate, please leave a comment and we will reopen the thread if necessary.

[santoshakil]

This is not a duplicate. This is a new architectural proposal — not a bug report. The referenced issues (#22850, #22522, #22101, #22320, #20331) describe symptoms. This discussion proposes a Rust-based solution to their shared root cause (Dart isolate memory model). No existing discussion covers this approach. Please reopen.