Proposal: Pluggable cache backends across Rust core and Python / Java bindings

[yanghua]

Background

Lance already has a typed cache framework and a CacheBackend abstraction:

• LanceCache: rust/lance-core/src/cache/mod.rs
• CacheBackend trait: rust/lance-core/src/cache/backend.rs
• Cache persistence envelope: rust/lance-core/src/cache/codec.rs
• Current production backend: MokaCacheBackend, defined in rust/lance-core/src/cache/moka.rs
• Rust currently has a custom backend injection point for the index cache: rust/lance/src/session.rs
• Python bindings currently expose cache size parameters, e.g. python/src/session.rs and python/src/dataset.rs

This means the lower-level abstraction is already extensible, but there is no unified mechanism yet for Session, language bindings, or third-party backends to select and register cache backends.

Motivation

The goal is to allow Lance users to choose a cache backend based on their deployment environment instead of always relying on a single default backend. Some concrete use cases:

1. Reduce cold-start cost If index cache / metadata cache can use a persistent or hybrid backend, a restarted process can reuse existing cached data instead of loading or rebuilding everything from scratch.
2. Fit different deployment environments A single-node workload may prefer an in-memory cache. A long-running service may prefer a disk or hybrid cache. Some production environments may already have shared cache infrastructure.
3. Allow third-party backends to evolve independently Not every backend implementation needs to live in the Lance repository. Third-party backends should be able to integrate as crates, and potentially as dynamic plugins in the future.
4. Expose the same capability through language bindings Python / Java users currently can tune cache sizes, but cannot explicitly select the backend. This limits configurability for users who primarily use Lance through bindings.

Goals

This proposal aims to add:

• A unified cache backend registration and construction mechanism in Rust core.
• Backend selection for both index cache and metadata cache.
• Stable backend configuration entry points in Python / Java bindings.
• Backward compatibility for existing cache size parameters.
• A design that can later support externally compiled dynamic .so plugins without changing the user-facing configuration model.

Non-goals

The following are out of scope for the initial implementation:

• Changing the core semantics of the existing CacheBackend trait.
• Adding a new concrete backend implementation as part of this proposal.
• Implementing a dynamic .so plugin loader or C ABI in the first step.
• Changing the current default backend behavior. If no backend is explicitly configured, Lance should behave as it does today.

Dynamic plugins are not part of the first implementation, but the design below should leave a clear path for them.

Proposed design

1. Add backend config and registry

Add a backend-independent config type in lance-core:

pub struct BackendConfig {
      pub kind: String,
      pub options: HashMap<String, String>,
  }

Where:

• kind identifies the backend type, e.g. "moka", "...".
• options contains backend-specific configuration.
• options uses HashMap<String, String> instead of a closed enum or Rust-only types, so it can be represented naturally across Python, Java, and future FFI boundaries.

The registry and construction API could look like:

pub type BackendBuildFn = fn(&BackendConfig) -> Result<Arc<dyn CacheBackend>>;

pub fn register_backend(kind: &'static str, build: BackendBuildFn) -> Result<()>;

pub fn build_from_config(config: &BackendConfig) -> Result<Arc<dyn CacheBackend>>;

register_backend should return Result<()> and reject duplicate kind registrations. This prevents different crates or future dynamic plugins from silently overriding each other.

MokaCacheBackend can also be registered as "moka" through the same mechanism, so the default backend and external backends share the same construction path.

2. URI as a binding-friendly configuration form

In addition to structured config, provide a URI form:

pub fn build_from_uri(uri: &str) -> Result<Arc<dyn CacheBackend>>;

Examples:

moka://?capacity=1073741824
foyer:///var/lance/cache?capacity=10G
redis://localhost:6379/0?prefix=lance

A URI would be parsed into BackendConfig, then passed to build_from_config.

The relationship is:

URI -> BackendConfig -> CacheBackend

This keeps Rust APIs structured while giving Python / Java / config files a compact string representation. It also avoids requiring each backend to implement its own incompatible URI parser.

3. Third-party crate integration

For a third-party backend that is compiled together with Lance, the integration path could be:

1. Implement CacheBackend.
2. Provide a BackendBuildFn.
3. Register it during initialization with register_backend("my_backend", build_my_backend).
4. Let users include it through a feature flag or dependency.

This does not require adding a new enum variant or hardcoded backend list in Lance core.

4. Symmetric backend selection for index cache and metadata cache

Rust currently has an injection point for the index cache, but metadata cache is not symmetric. This proposal suggests adding a unified construction path, for example:

pub fn with_caches(
        index_cache: LanceCache,
        metadata_cache: LanceCache,
        store_registry: Arc<ObjectStoreRegistry>,
    ) -> Self;

Alternatively, the current style can be preserved with separate methods:

pub fn with_index_cache_backend(...);

pub fn with_metadata_cache_backend(...);

The exact API shape can follow the existing Session design. The important part is that both index cache and metadata cache should be configurable through the same backend construction mechanism.

5. Python bindings

Python can accept a URI string:

session = lance.Session(
        index_cache_backend="moka://?capacity=1073741824",
        metadata_cache_backend="foyer:///var/lance/cache?capacity=10G",
    )

It can also accept a structured dict:

session = lance.Session(
        index_cache_backend={
            "kind": "foyer",
            "options": {
                "path": "/var/lance/cache",
                "capacity": "10G",
            },
        },
    )

The Rust binding layer can dispatch based on the Python value type and call either build_from_uri or build_from_config.

This avoids adding a Python class for every backend and avoids exposing Arc<dyn CacheBackend> directly to Python.

6. Java bindings

Java can use the same concepts:

• URI: String
• Structured config: kind + Map<String, String>

For example:

Session session = Session.builder()
        .indexCacheBackend("moka://?capacity=1073741824")
        .metadataCacheBackend("foyer:///var/lance/cache?capacity=10G")
        .build();

Or:

Session session = Session.builder()
        .indexCacheBackend("foyer", Map.of(
            "path", "/var/lance/cache",
            "capacity", "10G"
        ))
        .build();

Python and Java do not need to expose identical builder APIs, but the semantics should be consistent: each backend configuration should eventually map to a BackendConfig.

7. Compatibility with existing cache size parameters

Existing parameters should remain available, for example:

• index_cache_size_bytes
• metadata_cache_size_bytes

Suggested rules:

• If *_cache_backend is not set, keep the current behavior.
• If *_cache_backend is set, capacity and other backend-specific parameters should be supplied through the backend config.
• If both *_cache_backend and the corresponding *_cache_size_bytes are set, return a clear error instead of relying on implicit precedence.

For example:

lance.Session(
        index_cache_size_bytes=1024 * 1024 * 1024,
        index_cache_backend="moka://?capacity=1073741824",
    )

This should be rejected because the capacity is specified twice.

Compatibility with dynamic .so plugins

This proposal does not require implementing dynamic plugin loading in the first phase, but the API should be able to support externally compiled .so backends later.

A future dynamic plugin flow could look like:

1. Lance provides a plugin loading API, e.g. load_cache_backend_library(path).
2. The plugin exposes backend metadata and constructor functions through a stable ABI.
3. Lance loads the .so and registers the plugin backend into the same registry.
4. Users select the backend through the same URI or structured config mechanism.

In other words, dynamic plugins would add a loader and adapter under the registry:

    Python / Java / Rust config
          |
          v
  build_from_uri / build_from_config
          |
          v
  registry
          |
          +--> built-in backend
          |
          +--> third-party crate backend
          |
          +--> dynamically loaded .so backend

To keep this path possible without changing the public configuration model later, the current design should preserve these constraints:

1. Configuration must be FFI-friendly BackendConfig.options uses HashMap<String, String>, which can later be represented as string pairs, JSON, or another stable ABI-compatible form.
2. The registry must detect conflicts Dynamic loading can introduce duplicate backend names. register_backend should return an error instead of replacing an existing backend silently.
3. Construction errors should be reducible to code/message A dynamically loaded plugin cannot return Lance’s Rust error type directly across an ABI boundary. Backend construction errors should be representable as a stable error code and message.
4. Backend calls should not assume memory-only behavior Some backends may access disk, network, or go through an FFI adapter. Call sites should not assume in-memory cache latency characteristics.
5. Serialized backends need codecs. In-process Rust backends can cache Arc<dyn Any> directly. Dynamic plugins, remote backends, and persistent backends usually need to operate on bytes, so values should go through the existing CacheCodec / envelope mechanism.

This does not require ABI support today. It only means the factory / registry / config design should not conflict with future ABI-based plugins.

[yanghua]

cc @wjones127 @Xuanwo @westonpace @jackye1995

[Xuanwo]

This proposal does not seem to address how to add a cache backend that is not yet built into Lance. For example, Lance does not currently support Foyer. How would one load a Foyer backend?

[westonpace]

I think this is for index / metadata caching where foyer would be more for a data cache?

[yanghua]

Actually, we are trying to provide a commercial HybridLocalCacheBackend for index and metadata. For some indices, for example, IVF_Flat_SQ is also very large for a 1B-level+ dataset.

[yanghua]

This proposal does not seem to address how to add a cache backend that is not yet built into Lance.

Yes, currently, in the first phase, I only considered the register and switch cache backend, which is built into lance-core. So that we can reduce the complexity of this proposal.

However, if you'd prefer to have a third-party cache backend to register, I can refine it.

[yanghua]

I would split "add a backend that isn't built in" into two audiences, because they have very different answers:

• A. Users who build their own Rust binary can link a third-party crate and register it. This has a clean answer in phase 1 (below).
• B. Users on a prebuilt pylance / liblance cannot add a Rust dependency. For them "not built in" only becomes reachable via a .so loader, which I'm deliberately deferring to phase 2.

Conflating the two is what made the original text vague, so I'll address them separately.

The missing piece: when does registration happen? Rust has no automatic module init, so build_from_uri("foyer://...") only works if someone has already called register_backend("foyer", ...). Phase 1 will define this explicitly:

• Each backend crate exposes a register() that the application calls once at startup. It returns Result, detects duplicate kind conflicts, and is testable/predictable:

    // for example, if users implemented a new crate: lance-cache-foyer
    pub fn register() -> lance_core::Result<()> {
        lance_core::cache::register_backend("foyer", build_foyer_backend)
    }

    // application startup
    lance_cache_foyer::register()?;
    // now build_from_uri("foyer:///var/lance/cache?capacity=10G") resolves

• lance-core will auto-register only "moka". Everything else goes through the mechanism above.

For binding users, phase 1 is honest about its limits. The selectable set on a prebuilt distribution = the backends compiled in via cargo features (e.g. a backend-foyer feature behind pip install pylance[foyer]). The binding layer maps index_cache_backend="foyer://..." to build_from_uri; an unregistered kind returns a clear unknown backend: foyer error rather than silently falling back. True external loading for prebuilt binaries is phase 2's .so loader — and the forward-compat constraints already in the proposal (HashMap<String,String> config, conflict-detecting registry, error reducible to code/message, byte-oriented values via the codec) exist precisely to keep that door open.

[yanghua]

Loading a cache backend that is NOT compiled into Lance

Rust has no stable ABI, and CacheBackend is an async_trait whose futures cannot cross an FFI boundary. A backend that is not compiled together with Lance is therefore loaded as a C-ABI plugin (.so/.dylib/.dll): the boundary is a synchronous, byte-oriented vtable; async and type erasure stay on the Lance side. This reuses the five forward-compat constraints the proposal already reserved (string-map config, conflict-detecting registry, error as code+message, no in-memory latency assumption, values via CacheCodec).

LanceCacheBackendVtableV1 is the C-ABI projection of the CacheBackend trait; PluginBackend is its synchronous, byte-oriented simplification — async and Arc<dyn Any> stay on the Lance side and never cross the boundary.

The ABI (version 1)

Lance publishes a stable, #[repr(C)] vtable and a well-known entry symbol. Operations are blocking and exchange bytes; Lance bridges them to the async, typed CacheBackend.

pub const LANCE_CACHE_ABI_V1: u32 = 1;

#[repr(C)]
pub struct LanceKv { pub key: *const c_char, pub val: *const c_char }

#[repr(C)]
pub struct LanceKeyC {           // borrowed for the call's duration
    pub prefix: *const c_char,
    pub key:    *const c_char,
    pub type_name: *const c_char,
}

#[repr(C)]
pub struct LanceCacheBackendVtableV1 {
    pub abi_version: u32,                 // must equal LANCE_CACHE_ABI_V1
    pub kind: *const c_char,              // e.g. "foyer"

    // Construct from string options. NULL on failure (err_out filled).
    pub create: extern "C" fn(*const LanceKv, usize, *mut c_char, usize) -> *mut c_void,

    // Blocking, byte-oriented ops. 0 = ok; nonzero = stable error code.
    // On a get hit, plugin allocates *value_out; Lance returns it via free_value.
    pub get: extern "C" fn(*mut c_void, *const LanceKeyC, *mut *mut u8, *mut usize) -> i32,
    pub insert: extern "C" fn(*mut c_void, *const LanceKeyC, *const u8, usize, usize) -> i32,
    pub invalidate_prefix: extern "C" fn(*mut c_void, *const c_char) -> i32,
    pub clear: extern "C" fn(*mut c_void) -> i32,
    pub num_entries: extern "C" fn(*mut c_void) -> usize,
    pub size_bytes: extern "C" fn(*mut c_void) -> usize,

    pub free_value: extern "C" fn(*mut u8, usize),   // ownership handoff
    pub destroy: extern "C" fn(*mut c_void),
}

1. How a plugin author implements it

The author writes an ordinary Rust backend and links a small helper crate (lance-cache-plugin) whose macro generates the extern "C" shims and the entry symbol. Inside the .so everything is the author's own compiler and runtime; the C ABI only governs the Lance↔plugin edge.

// crate-type = ["cdylib"]
struct FoyerBackend { /* author's own foyer state */ }

impl lance_cache_plugin::PluginBackend for FoyerBackend {
    fn create(options: &BTreeMap<String, String>) -> Result<Self, String> { /* ... */ }
    fn get(&self, key: &PluginKey) -> Option<Vec<u8>> { /* ... */ }
    fn insert(&self, key: &PluginKey, value: &[u8], size_bytes: usize) { /* ... */ }
    fn invalidate_prefix(&self, prefix: &str) { /* ... */ }
    fn clear(&self) { /* ... */ }
    fn num_entries(&self) -> usize { /* ... */ }
    fn size_bytes(&self) -> usize { /* ... */ }
    pub approx_num_entries: extern "C" fn(*mut c_void) -> usize,
    pub approx_size_bytes:  extern "C" fn(*mut c_void) -> usize,
}

// Generates lance_cache_backend_plugin_v1() and all extern "C" shims.
lance_cache_plugin::export_cache_backend!(kind = "foyer", FoyerBackend);

cargo build --release        # produces libfoyer_cache.so

2. How Lance registers it

Lance exposes a loader. It dlopens the library, resolves the entry symbol, checks the ABI handshake, reads kind, wraps the vtable in an adapter that implements the async CacheBackend, and registers it into the same registry every other backend uses. The Library handle is kept alive for the process.

pub fn load_cache_backend_library(path: impl AsRef<Path>) -> Result<()> {
    // 1. libloading::Library::new(path)
    // 2. sym: fn() -> *const LanceCacheBackendVtableV1
    // 3. assert vtable.abi_version == LANCE_CACHE_ABI_V1  (else stable error)
    // 4. kind = CStr(vtable.kind)
    // 5. register_backend(kind, move |cfg| {
    //        let handle = (vtable.create)(cfg.options ...)?;   // -> *mut c_void
    //        Ok(Arc::new(DynPluginBackend { vtable, handle, lib }))
    //    })
}

The adapter turns blocking C calls into the async trait and does typed↔bytes via the codec:

#[async_trait]
impl CacheBackend for DynPluginBackend {
    async fn get(&self, key: &InternalCacheKey, codec: Option<CacheCodec>) -> Option<CacheEntry> {
        let bytes = spawn_blocking(move || call_get(vtable, handle, key_c)).await.ok()??;
        codec?.deserialize(&bytes).ok()          // bytes -> Arc<dyn Any>
    }
    // insert: codec.serialize -> spawn_blocking(insert)
    // get_or_insert: get(); on miss run the native Rust loader; insert().
    //                Concurrent-load dedup stays here, not in the plugin.
}

Because CacheBackend methods receive codec: Option<CacheCodec> and the trait requires that a None codec be stored in-memory (an entry type that cannot yet be serialized), the adapter carries its own L1 Moka layer. When codec is None, get / insert / get_or_insert operate entirely on that in-memory layer and never touch the plugin; only when codec is Some does the adapter serialize to bytes and hand them to the plugin. A bytes-only plugin cannot represent non-serializable entries, so this in-memory fallback is a correctness requirement, not an optimization.

3. How a user uses it

Identical to any built-in backend once loaded — the caller never sees the difference:

// Rust
lance::cache::load_cache_backend_library("/opt/lance/libfoyer_cache.so")?;
Session::builder().index_cache_backend("foyer:///var/lance/cache?capacity=10G").build()?;

# Python — no recompilation of pylance required
session = lance.Session(
    backend_plugins=["/opt/lance/libfoyer_cache.so"],
    index_cache_backend="foyer:///var/lance/cache?capacity=10G",
)

Constraints this relies on

• ABI handshake: abi_version mismatch is rejected with a stable error; bumping the vtable defines ...VtableV2 alongside V1.
• Compiler coupling is confined to the plugin: the .so is internally self-consistent; only the C ABI crosses the boundary, so plugin and Lance need not share a rustc version.
• Ownership: values allocated by the plugin are returned to it via free_value; the handle is torn down via destroy on backend drop.

[westonpace]

Looks like a good approach to me. This is somewhat similar to how the scalar index plugin registry is setup. Instead of just using a factory function we went ahead and made a ...Plugin trait (e.g. CacheBackendPlugin or CachePlugin that has the build function. Still, that might not be needed here. With index plugins you need to select the plugin based on manifest information. With a cache the user is always specifying what cache they want to use (we don't save cache backend preference as part of the table) so the simple build function might be fine.

[wjones127]

FYI, I'm working with another contributor on improving the cache implementations. Two changes that will likely come soon: we will change the keys to a u128 instead of a string, and we will drop invalidate_prefix, which is hard to implement in most backends anyways.

Serialized backends need codecs. In-process Rust backends can cache Arc directly. Dynamic plugins, remote backends, and persistent backends usually need to operate on bytes, so values should go through the existing CacheCodec / envelope mechanism.

Some work that's not complete yet is I'd like to deprecate and remove the use of Arc<dyn Any> and make sure we are always supporting CacheCodec. So I agree with this direction.