rotate_bucket / rewrap_object_dek does not sync forest user_metadata with S3

[ehsan6sha]

Summary

EncryptedClient::rewrap_object_dek in crates/fula-client/src/encryption.rs:8822-8893 re-wraps an object's DEK under the current KEK and re-PUTs the object to S3 with an updated x-fula-encryption header (new wrapped_key + new kek_version). It does not update forest_entry.user_metadata["x-fula-encryption"] to match. Consequently, after rotate_bucket / rotate_bucket_with_journal succeeds, the local forest's AEAD-protected copy of the per-object encryption JSON is stale by design: forest carries the OLD wrapped DEK while S3 carries the NEW one.

This is a follow-up to #11 — the fix for #11 added get_object_encryption_metadata_with_fallback that falls back to forest_entry.user_metadata["x-fula-encryption"] when master is unreachable. After a key rotation, that fallback would return stale data, and any path that consumes it (offline share-token creation per #11, offline download via get_object_decrypted_inner's get_meta helper at encryption.rs:1453-1463) would attempt to unwrap a DEK that was wrapped under the previous KEK with the current KEK private — resulting in Failed to unwrap DEK.

Reproduction

// Pseudocode
let client = make_client(endpoint); // EncryptedClient with KEK_1
client.put_object_encrypted(bucket, key, data).await?; // upload, forest stores x-fula-encryption with KEK_1 wrap
let manager = client.create_rotation_manager(); // generates KEK_2
client.rotate_bucket(bucket, &manager).await?; // S3 now has KEK_2 wrap

// Forest entry still has KEK_1 wrap:
let storage_key = /* lookup */;
let forest_entry = /* fetch forest entry by storage_key */;
let s3_meta_str = client.inner().head_object(bucket, storage_key).await?
    .metadata.get("x-fula-encryption").unwrap();
let forest_meta_str = forest_entry.user_metadata.get("x-fula-encryption").unwrap();
assert_eq!(s3_meta_str, forest_meta_str); // FAILS — forest stale

Root Cause (file:line)

crates/fula-client/src/encryption.rs:8822-8893 (rewrap_object_dek):

pub async fn rewrap_object_dek(...) -> Result<u32> {
    let result = self.inner.get_object_with_metadata(bucket, storage_key).await?;
    let enc_metadata_str = result.metadata.get("x-fula-encryption").ok_or_else(...)?;
    let mut enc_metadata: serde_json::Value = serde_json::from_str(enc_metadata_str)?;
    // ... unwrap with old KEK, rewrap with new KEK ...
    enc_metadata["wrapped_key"] = ...; // new wrapped key
    enc_metadata["kek_version"] = ...; // new version

    self.inner.put_object_with_metadata(
        bucket,
        storage_key,
        result.data,
        Some(ObjectMetadata::new()
            .with_content_type(...)
            .with_metadata("x-fula-encrypted", "true")
            .with_metadata("x-fula-encryption", &enc_metadata.to_string())),
    ).await?;
    // ← S3 updated, forest_entry.user_metadata NOT touched

    Ok(rotation_manager.current_version())
}

rotate_bucket_inner (encryption.rs:8926+) calls rewrap_object_dek for each non-forest object. Forest itself is skipped, but the per-object forest entries' user_metadata are never synced with what rewrap_object_dek writes to S3.

Affected Read Paths

1. get_object_decrypted_inner (encryption.rs:1362+): the get_meta helper at lines 1453-1463 falls back from HTTP headers to forest_entry.user_metadata when the offline path returns cached/gateway bytes without HTTP metadata. After rotation, the cached metadata is stale → DEK unwrap fails.
2. get_object_encryption_metadata_with_fallback (introduced for Offline share-token creation fails: sharing.rs head_object bypasses offline-fallback #11): falls back to forest_entry.user_metadata["x-fula-encryption"] on transport-error. After rotation + master-unreachable, this returns the OLD wrapped DEK → share-token creation produces tokens with wrapped DEKs the recipient cannot decrypt.

Both paths are correct for an online S3 read (HEAD returns fresh metadata). The drift only surfaces when offline-fallback kicks in.

Expected vs Actual

• Expected: after rotate_bucket succeeds, forest_entry.user_metadata["x-fula-encryption"] for each rotated object reflects the same wrapped DEK + kek_version that head_object would return. Offline read paths remain correct.
• Actual: forest carries the pre-rotation metadata; offline reads after rotation fail to unwrap.

Suggested Fix

In rewrap_object_dek, after the S3 PUT succeeds, locate the forest entry by storage_key (forest entries for chunk objects don't exist; that's fine — silently skip). If found:

1. Update forest_entry.user_metadata.insert("x-fula-encryption", new_enc_json).
2. Upsert back into the forest (mirroring the pattern at encryption.rs:6095-6137 in the upload path).
3. Mark forest cache dirty so the next flush persists the change.

No FFI signature change. No new public surface needed (the helper can be private to EncryptedClient). Approximate diff: ~50 LOC.

Severity

Medium. Affects:

• Any consumer that uses rotate_bucket AND ever reads encrypted data offline post-rotation.
• The FxFiles offline scenario described in Offline share-token creation fails: sharing.rs head_object bypasses offline-fallback #11 specifically — share-token creation falls back to stale forest metadata after rotation.

Not exploited by current FxFiles flows because rotation is operator-initiated. But blocks the planned migration story for F-A1 (Argon2id KDF + user secret) where every existing user will rotate their KEK once on first sign-in after the upgrade — every user is post-rotation. After that migration, offline share-token creation AND offline download would silently break.

Verification Plan

A failing unit test will be added under tests/ that:

1. Spins up local in-memory gateway, uploads an encrypted file.
2. Calls rotate_bucket with a fresh KeyRotationManager.
3. Reads S3's x-fula-encryption via head_object (fresh, post-rotation).
4. Reads forest_entry.user_metadata["x-fula-encryption"] (currently stale).
5. Asserts they match → fails today, passes after fix.

After the fix lands, the same test should pass without modification.

Related: this affects the issue #11 fix's documented limitation ("Fully-offline flows after a rotation may use stale wrapped DEKs until a follow-up fix mirrors the rewrap into the forest entry"). That follow-up is this issue.