A workbench for reading LanceDB datasets — schema, versions, indices, fragments and rows, with the byte cost of every read shown as you go.
A Lance table can hold 2.65 GB of video while a search over it reads none of it. The bytes a search touches and the bytes a table holds live in different files. This measures both.
Website · Live console · Docs · Guide · Releases
Six things, precisely.
| Reads a database, exactly | Schema, versions, indices, fragments and rows, with the byte cost of each read shown as you go. Nothing here writes to a dataset. |
| Answers "why is this slow" | Run a scalar, full-text, vector or hybrid query; see which access path Lance chose, what it read, and the Python that reproduces it elsewhere. Compare two versions of a table and run the same query against both. |
| Checks the data, and charges for it out loud | Duplicates, rows missing their content, class imbalance, a split that leaks, dead embeddings. These read your columns rather than a manifest, so every one is priced from the file footers before it runs, cancelling stops the work, and each result carries the bytes it cost. Asking whether 162 videos are all actually there reads 43.4 KB and none of the 2.65 GB. |
| Lets the answer leave | Take the whole diagnosis away as one document — the findings with their evidence, the plan, the exact cost, the reader underneath — as markdown to paste into an issue or JSON another console opens. No rows, no credentials, and the database root redacted unless you ask for it. |
| Says what it already knows | Ten rules over metadata — an unindexed vector column, small-file counts that would mislead, tombstone debt, a manifest that understates the size of the thing it describes — each carrying the numbers it was derived from. No model is involved in any of them. |
| Adds language, optionally | With a local model or an API key it translates a question into a filter and describes a table in a few sentences. Every response reports the tokens and dollars it spent beside the bytes it read. |
Ten rules run over the same manifests the other tabs read. Each finding carries its evidence on the same row — fragments, rows, pass bytes, bytes per vector — and each also appears under the panel holding the numbers it came from. Nothing here costs a token, because no model is asked.
Everything above is derived from manifests and costs kilobytes. The Data tab is the one place that reads your columns, so nothing on it runs until you press a button — and what it offers first is the bill:
missing-content reads 11.9 KB–43.4 KB, and none of the 2.65 GB of blob payload
those descriptors point at
exact-duplicates reads 6.8 KB–43.4 KB
class-balance name the one column that holds the label
near-duplicates 'vector' has no vector index. Without one this would be a full pass
over every vector for each row sampled
A Blob V2 column projects to its descriptor rather than its bytes, so asking whether every video is actually there costs kilobytes. A check that cannot run says which sentence applies rather than being greyed out. And cancelling stops the work — the one place in the console where it does, because that loop is ours and a Lance query's is not.
It reports the data and not a verdict: it cannot tell you whether a label is right, cannot find a duplicate the embedding does not encode, and its near-duplicates are approximate twice over. Every one of those is written on the panel.
The same rules narrowed to what a training run actually pays for: how many bytes an epoch reads, how many loader workers the fragment split can feed, and the version to record in the run config. It reports the layout and nothing about the data — it cannot tell you whether your labels are right, and says so.
The query panel opens by reading the first page, so looking at a table costs no clicks. Ask in English and the draft predicate lands in the filter box — the same box that completes column names and their values — for you to read before anything runs.
Heavy columns — vectors, images, and Blob V2 columns holding the large data — are described from the schema rather than materialised. Browsing a table that holds gigabytes of video costs kilobytes, and that is the point rather than a limitation.
Three ways in. All of them read your data where it already is.
git clone https://github.com/mrlynn/lancescope && cd lancescope
make setup
make devThen open localhost:3000/console and point it at any directory holding .lance
tables — or paste a Hugging Face URI and read someone else's over the network, without
downloading it. Needs uv and Node 22.
make local is the other one worth knowing: it serves the exported interface and the
API on one origin, which is what actually ships. A static export has no rewrites
and no dev server, and that difference has broken things make dev could not see.
The way in on Linux, on a server, or beside a pipeline already writing Lance tables:
docker run --rm -p 8088:8080 \
-v /path/to/your/lance:/data:ro \
ghcr.io/mrlynn/lancescope:pylance-11.0.0There is one image per Lance reader, because a Lance reader is not universal: a
dataset written by one version may need that version to read it. The tag names the
reader, and docker/compose.yaml is the version worth having
when something else in the stack owns the volume.
make appBuilds LanceScope.app — a window with its own title bar, running a server it starts
and stops itself. Nothing to install on the machine it lands on: no Python, no Node, no
Lance. 160 MB as a DMG.
It carries the console and leaves out torch, so the demo's semantic search is
unavailable in a packaged build and says so. ./desktop/sign.sh signs and notarises it
with a Developer ID — see the guide. A signed, notarised
DMG is built by the tagged release workflow rather than by hand; until one is published,
building it yourself is the whole thing and nothing is missing from it.
demo.lancescope.mlynn.dev — the console
itself, on a public LanceDB dataset, with nothing to install. Read-only, rate limited,
and pinned to one dataset it does not host: pylance opens hf:// lazily, so the
deployment stores nothing and reads only what you look at.
The console is not wired to the demo. It reads whatever connection is active, and you
manage those at /console/settings — paste a path, check what is there, switch. The
catalog is repointed in place; nothing restarts.
Where the root comes from, in order:
| rung | when |
|---|---|
LANCE_ROOT |
set in the environment — wins, and the settings page says so |
| the active connection | saved in ~/.config/lancescope/settings.json |
data/lance |
first run with nothing configured, and only if it holds tables |
| nothing | the console says so and points at settings |
LANCESCOPE_CONFIG moves the settings file. Connections may be local directories,
hf://datasets/… roots, object-store URIs (s3://, gs://, az://, abfss://),
or db:// for LanceDB Cloud.
The datasets LanceDB publishes on Hugging Face open directly — the settings page offers five of them, so a fresh install has something to look at without building anything first:
hf://datasets/lance-format/openvid-lance/data
937,957 rows carrying the MP4s themselves beside their embeddings. Opening it costs 24,568 bytes; browsing five rows costs about 73 KB and reads no video at all. The same claim this repo makes about its own corpus, checked against someone else's.
s3://, gs://, az:// and abfss:// are browsed the same way, with credentials
from the ordinary AWS_*, GOOGLE_* and AZURE_* environment names — listed through
Lance's own object store, so a bucket that lists is a bucket that opens.
db://my-database reads LanceDB Cloud with LANCEDB_API_KEY set.
The byte figures for those are reported as unverified rather than claimed: the mechanism is the one that works locally and on the Hub, and nothing here has yet measured it against a live bucket. A scheme with no adapter at all is saved and said to be unbrowsable, rather than shown as an empty database — and adding one is an installable package, not a wait.
It will build you a database. It will never edit one.
-
Browsing. Reading a table cannot change it. That is checked rather than asserted: a test drives the entire read API and every MCP tool over a real corpus and then checks that not one byte on disk moved.
-
Building. The ingest wizard creates a new table from your own files and that is the only thing in the project that writes one. It is create-only by construction: it refuses a destination that already exists, only ever appends into a table it made during that same run, and has no reachable path to an overwrite.
Images and PDFs anywhere; video and audio from a checkout. Those two need
ffmpeg, which is a binary rather than a Python package — an order of magnitude larger than the wheels beside it and carrying a licensing decision nobody has made — so the signed app does not ship it. The wizard says which build you are on before you choose a folder rather than after it has scanned one, and files it cannot decode are counted and left alone rather than failing partway through a run.brew install ffmpegand a checkout does all four.
The whole write surface is one module, and CI fails if a dataset mutation appears anywhere else. Deleting only ever happens when you ask for it: the button that clears a finished job and the button that deletes the table it produced are deliberately two different buttons.
The console's read surface is also an MCP server, so Claude Code — or any agent host that speaks MCP — can inspect a LanceDB database directly:
Open Settings → Agents and copy the command, or write it out yourself:
claude mcp add lancescope -- lancescope mcp # installed, or the packaged app
claude mcp add lancescope -- uv --directory /path/to/lancescope run python -m ingest.cli mcpThe settings pane also carries the JSON block for Claude Desktop, Cursor, Windsurf and VS Code, and the path each one reads it from.
Which database? Whichever connection the console is pointed at — the same ladder,
resolved on every call, so switching connections in the console switches what the agent
sees. list_tables reports the root it resolved, so the agent can say which database it
is describing. To pin it to one instead, independently of the console:
claude mcp add lancescope -- lancescope mcp --root /path/to/tablesThe read surface, as tools — catalog, schema, versions, indices, fragments, findings,
estimates, run configs, diagnosis bundles, operation plans, and the query surface as
explain_query, query_capabilities, validate_filter and compare_versions. Every
one of them is read-only and declared as such, and none can materialise a blob column,
because the routes underneath them cannot. Nothing here runs a query: explain_query
returns the plan and a Python reproduction, and spending the read budget stays a button
in the console. The full list is in the reference, which
is generated from the code. It needs no key of its own: the intelligence is the agent's,
and the tools are the same routes the console calls.
Ask it "what's in this database and what's wrong with it" and it comes back with the unindexed vector column and what a search therefore costs — with the numbers those conclusions were derived from, not a summary of them.
Optional, and the console is useful without it. The settings page configures which provider powers the language layer — Claude, a local Ollama model, or any OpenAI-compatible endpoint — and reports what is actually available on the machine right now, including the models Ollama has pulled.
ollama pull qwen3:8b # local, free, offline, no account
export ANTHROPIC_API_KEY=sk-… # or Claude, with the cost of every call shownNeither is required, and neither has to be configured: with a key set the Claude path comes up on its own, and with Ollama running the local one does. An explicit choice in settings beats both. A key pasted into the settings page is stored at mode 0600 — the environment variable is the safer path and always wins over it.
Test it before trusting it. Test the model spends one real call on the configured provider and reports what came back, how long it took and what it cost, so a stale key or a deleted model is a sentence on screen rather than a mystery three features later:
answered, and honoured the schema
gemma3:27b · 11.8s · 72 in / 41 out · no cost — this ran on your machine
GET /intel/capabilities is the same answer without spending anything. Both report
cost_usd: null rather than a guess for a model that isn't in the price registry —
prices are cached data, and carry the date they were read.
Measured rather than assumed. scripts/compat/probe.py
installs each candidate into its own environment and reads a real blob table with it,
and the floor it establishes is pylance 3.0.0:
| below the floor | what breaks |
|---|---|
2.0.1 |
sees a Blob V2 column and cannot open it against storage format 2.2 |
1.0.4 |
does not recognise a Blob V2 column at all |
0.38 |
has no io_stats_incremental, so every byte figure in the console is missing |
That last one was this project's declared floor until the probe was written — a number nobody had checked.
CI runs the contract tests against eight readers on every push — 3.0.0, 4.0.2,
6.0.1, 7.1.0, 8.0.1, 9.0.1, 10.0.0, 11.0.0 — and a row goes red if the reader is missing
a capability the console names, not merely if a test fails. The same eight are published
to ghcr.io, and the settings page has a The reader section naming the versions
underneath and anything they cannot do.
Lance versions is the matrix.
Everything above in more depth, and served by the app itself at /docs — a tutorial,
how-to guides, generated reference, and the reasoning behind the decisions that look odd:
| What LanceScope is | the shape of the thing |
| Getting started | clone to a real finding, about five minutes |
| Connect a database | local, switching, pinning, remote |
| Diagnose a slow query | access paths, costs, before and after |
| Enable the language layer | local and free, or a key |
| Point an agent at it | MCP, and which database it reads |
| Run it in a container | Docker and Compose, pinned to the reader your data needs |
| Build a desktop app | the sidecar, signing, notarisation |
| Reference | config, query modes, findings, models, tools, routes |
| Lance versions | which readers are supported, and how that was measured |
| Why it works this way | cost as the unit, Blob V2, evidence before advice |
The six reference pages are generated from the code by make docs, and make test
fails if they drift from it.
make checkEverything CI runs, in the order that fails cheapest first: ruff on the same pin the
workflow uses, the contract tests, then the interface's typecheck, lint and build. It
ends by building the container and booting it to ask which reader is inside, because a
container that starts is not a container that serves.
Docker Desktop is found where it actually installs itself rather than only on PATH —
its CLI lives in ~/.docker/bin, which plenty of shells never see. Where there is
genuinely no Docker, or its daemon is not answering, the target says which of the two
and moves on, so a skipped check never reads as a passed one.
make test stays the fast inner loop and covers one of CI's five jobs; make check
covers four, and tests/test_check.py fails if a job is added to the workflow that the
target neither runs nor explicitly excuses. The fifth is the eight-version reader
matrix, which is minutes rather than seconds and is excused by name.
make verify is a different gate: the real corpus rather than synthetic fixtures, and
the one that has to pass before a demo.
A ~10 minute conference demo of LanceDB, built around one claim:
The video and its index are the same table.
You type "a diagram with boxes and arrows", get back actual frames from a corpus of conference talks, click one, and the video plays at that exact second — while an instrument along the bottom of the screen shows how few bytes moved to make it happen.
No OCR ran. Nobody captioned those frames. And searching the whole corpus reads zero bytes of video — not "very little", zero, because the video bytes are not in the files a search opens.
The interesting part of LanceDB in 2026 isn't vector search — it's that a Lance table can hold the MP4 itself in a Blob V2 column, in a side file, behind lazy handles. Search and filter cannot touch it. That is hard to assemble out of S3 + Postgres + a vector DB, and it makes for a claim you can prove live rather than assert on a slide.
Every figure comes from Lance's own IO accounting (Dataset.io_stats_incremental()), not
from anything measured off to the side. Run make verify to reproduce.
Measured on a 16-talk corpus: 1,114 moments, 162 segments, 2.65 GB of video.
| operation | index bytes | video bytes |
|---|---|---|
| semantic search over every moment | 3.45 MB | 0 |
| full-text search over transcripts | 0.11 MB | 0 |
| the same search, filtered to one devroom | 3.45 MB | 0 |
| open a blob handle | — | 2,722 |
| start playback (cold segment) | — | ~17 MB, one segment |
| seek again inside it (warm) | — | 262,144 — byte-exact |
On disk that table is 2.65 GB of video in .blob side files against 20.1 MB of
everything a search reads — a ratio of 132 to 1, which the S view shows live.
See FINDINGS.md for the measurements that shaped the design, including the one that forced videos to be stored as ~16 MB segments rather than whole files.
make setup # uv sync + npm install
make ingest LIMIT=36 # download, transcode, segment, embed, build, verify
make demo # API on :8000, UI on :3000Once set up, double-click LanceScope.command in Finder instead — it runs the same
thing, checks the handful of conditions that otherwise fail silently, and opens the
console for you. Closing the window stops both servers.
Requires ffmpeg on PATH. Everything else lives in the venv. Once ingested the demo runs
fully offline — no network, no services, no containers. It needs a built corpus and a
local embedding model, so it is not in the packaged app: the DMG deliberately ships
without torch, and the demo screen says so rather than failing quietly.
The interface is driven from the keyboard so you never hunt for a mouse:
| key | does |
|---|---|
1–4 |
run the four rehearsed queries |
/ |
focus the search box to type something the audience suggests |
↵ |
open the first result |
S |
the schema, read live off disk — this is the Act 3 slide |
R |
reset the byte instrument |
T |
cycles light / dark — light is what it ships on, and what these captures are |
Esc |
close whatever is open |
- The architecture isn't a triangle. Every video search system is S3 + Postgres + a vector DB + a queue. Replace that slide with one box.
- Search.
1finds diagrams by what they look like.2and3show it holds up. Switch to full text, then hybrid. Then pick a devroom — the SQL predicate runs inside the vector search, not as a filter afterwards. - The schema (
S).video_blobsits in the same table as the embeddings. The panel at the bottom is the point: gigabytes of video in.blobside files, and a few megabytes of everything a search actually reads. - The instrument. Press
Rto zero it. Run a search: the playing it needle stays pinned at NONE. Then open a result and watch it move. That's the close. - The doors not walked through. Branching and shallow clone, Geneva backfills, DuckDB SQL over the same files.
ingest/download.py builds it from the
FOSDEM video archive: direct MP4s over plain HTTP, official
.vtt subtitles for every talk, and a schedule feed that supplies real titles, speakers
and devrooms. Talks are transcoded to 720p on the way in, which keeps slide text sharp at
about a third of the bytes.
FOSDEM recordings are published under CC-BY. The videos are gitignored regardless — the repo ships the pipeline, not the corpus.
A YouTube path survives in ingest/download_youtube.py, but YouTube rate-limits scraping
hard and answers with bot checks. It is a bad thing to depend on the night before a talk,
which is why it is not the default.
Next.js (:3000) ──/api/*──> FastAPI (:8000) ──> a source adapter ──> local disk,
hf://, an object store, or db://
│ │
SigLIP (MPS) MCP server (stdio)
ingest/ download + transcode -> segment + keyframes -> SigLIP -> Lance tables
server/ FastAPI: catalog, query, findings, intelligence, ingest jobs, MCP
web/ Next.js: home, console, docs, settings, the demo
docker/ one image per Lance reader
desktop/ Tauri shell, PyInstaller sidecar, signing and notarisation
scripts/ blob_bench.py (the evidence), verify.py (green-room preflight), compat/probe.py
docs/ the guide, generated reference, sprint plans
Two tables in the demo corpus, one format, one store, zero services:
moments— one row per keyframe: SigLIP embedding, transcript window, thumbnail, speaker, devroom. This is all a search ever touches.segments— one row per ~16 MB playable MP4 chunk in a Blob V2 column, streamed to the browser over HTTP Range straight out of the column.
The data layer is Python because take_blobs and multivector search are Python-only
today; the TypeScript SDK covers vector, FTS and hybrid but not blobs.
Licensed under Apache-2.0. See CONTRIBUTING.md for how the work is planned and landed.
An independent tool. Not affiliated with, endorsed by, or supported by LanceDB. The dot lattice in the mark is derived from theirs, with thanks; the glass is ours. LanceDB and Lance, and their logos, are the property of their owner.