Speedup recall using Hugging Face local cache#42
Merged
Conversation
Add FetchStore, the read mirror of CaptureStore: an ObjectStore decorator that serves every get from a whole local file supplied by a FileFetcher (sliced to the requested range) and delegates every other method to the inner store. It owns no cache — the fetcher decides where files come from and whether they persist — so the module stays backend-agnostic, with no reference to HF, exactly like capture_store. The HF-specific pieces live in hf_dataset beside CaptureWrapper: HubFetcher (hf-hub download_file pinned to a commit SHA, for zero-network warm reads over an immutable file set) and FetchWrapper (the WrappingObjectStore that installs FetchStore). hf_dataset depends on fetch_store, never the reverse. This is Phase 1 of caching remote recall reads; the mechanism is inert until FetchWrapper is wired into Store::open (Phase 2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the FetchWrapper into Store::open. A remote open now resolves the head commit, pins lance to it (hf_revision), and installs the wrapper, so lance's reads go through hf-hub's download_file instead of opendal's per-range fetches over hf://. hf-hub serving repeat reads from its local cache — what makes warm recalls fast — is the bonus that justifies it over opendal; a cold read still downloads. - dataset::open_wrapped installs a WrappingObjectStore before load, so it covers the load-time reads too, and stays backend-agnostic. - hf_dataset::fetch_wrapper builds the repo handle, resolves the head, and returns the wrapper pinned to that SHA. - Freshness: every command re-opens and re-resolves the head, so a push is seen on the next call. If the head can't be resolved (offline/transient), degrade to a live, uncached open rather than fail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a gated live test (HF_FUNES_TEST_TOKEN, like the other remote tests) that isolates HF_HUB_CACHE to a temp dir and asserts a warm recall over the test fixture downloads nothing: the cache footprint is byte-for-byte unchanged across a cold then warm recall, and both surface the marker. This is the regression guard for the read-through cache's core guarantee — a change that broke caching (e.g. dropping the token so reads fall back to the live store) would otherwise pass CI silently. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The read-through cache change made hf-hub's file cache (HF_HUB_CACHE) the operative one for remote reads, but bench_recall's --cold still isolated HF_XET_CACHE — which the read path no longer uses — so a populated file cache served the "cold" leg and it reported a warm-as-cold (cold ≈ warm). Point --cold at HF_HUB_CACHE so the remote cold call is a genuine download again: cold shows the one-time download premium, warm collapses to ≈ local. Also drop the shell-out to the `hf` CLI for the local-leg download — funes already depends on hf-hub, so fetch via snapshot_download with local_dir (which writes straight to the temp dir, bypassing the hub cache, so it can't pre-warm the remote leg) and let HFClient read the token from the environment. No external CLI needed. Refresh benchmark/README.md to the new behavior (warm remote ≈ local, not ~10×) and note the absolute numbers are rerank-bound and host-dependent — compare the remote/local ratio, not the floor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI's format check runs nightly rustfmt, which wraps the long assert! that stable rustfmt (used locally) left on one line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The default lance
opendal[huggingface]backend used forrecalldoes not perform any caching.We wrap the
lanceobject store used when reading to replaceopendaldownload calls byhf-hubdownload calls that benefit from the local cache.Details:
FetchStoreis a genericObjectStoredecorator (the read mirror ofCaptureStore); the HF-specificHubFetcher/FetchWrapperlive inhf_dataset, so the generic layer stays backend-agnostic.hf://round-trips are gone); the first touch still pays a one-time download. Guarded by a gated live test (tests/remote_cache.rs).bench_recall --coldto isolate the hf-hub file cache (it was isolating the Xet cache, which this path no longer uses) and drops itshf-CLI dependency in favour of thehf-hubcrate.