Skip to content

feat(duckvfs): install the extension from the community repository - #433

Merged
jrosskopf merged 1 commit into
mainfrom
feat/duckvfs-community-extension
Sep 1, 2026
Merged

feat(duckvfs): install the extension from the community repository#433
jrosskopf merged 1 commit into
mainfrom
feat/duckvfs-community-extension

Conversation

@jrosskopf

@jrosskopf jrosskopf commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

The duckvfs lane store could not be deployed to a container. This makes it deployable.

The problem

extension_path names a locally built gdrive.duckdb_extension. It is the only way to register write_blob / remove_file / file_size, which live in the extension rather than DuckDB core and are needed for every root, not only gdrive://. A pod has no such file.

Leaving it unset is the worse failure, not an escape: None skips the load and assumes the functions are present. The store then opens cleanly and the first write fails with Catalog Error: Scalar Function with name write_blob does not exist!. A boot that succeeds and a write path that cannot work is the shape that reaches production looking healthy.

The change

gdrive now ships in the DuckDB community repository — which duckvfs.rs already anticipated in a comment ("Once gdrive is in the community repository this becomes INSTALL gdrive FROM community; LOAD gdrive;"). extension_repo is that, as ESCUREL_STORAGE_DUCKVFS_EXTENSION_REPO: community, or a repository URL.

A new field, not a redefinition of extension_path: None. Repurposing None would change behaviour for existing file:// roots and force a network install where none happens today. extension_path keeps its exact meaning and still takes precedence, so an operator pointing at a local build always gets that build.

One detail encoded in the code: community is a keyword in DuckDB's grammar and must not be quoted, while a URL must be. Quoting the keyword makes DuckDB look for a repository literally named "community" and fail with a message that says nothing about quoting.

Verification

Red → green, plus two checks — the second is the one that mattered.

  1. Load-bearing: mutating extension_repo to None fails the test with the missing-write_blob error. It asserts a write round trip, not that the constructor returns Oknew performs no I/O against the root, so a store that never loaded the extension constructs perfectly.
  2. Not just my cache: it first passed in 0.65s, too fast for a download — gdrive.duckdb_extension was already in ~/.duckdb/extensions/v1.5.5/linux_amd64/ from local use. Re-run with a cold HOME it genuinely downloads, populates the fresh cache and passes. That is the container case; the deployment already sets HOME to a writable dir. Without this check the test would have been green here and broken everywhere it is actually needed.

The new test deliberately takes no ESCUREL_TEST_GDRIVE_EXTENSION — needing a prebuilt file is the thing being removed. It does need network access to the community repository.

1377 passed, 0 failed, clippy clean. API contract unchanged.

Why now

This unblocks running an Escurel in the cluster against the existing gdrive-backed lake (the datazoo-loops corpus) rather than moving that data. The remaining pieces are a WIF credential for the Drive, a catalog DSN secret, and the substrate app.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

The duckvfs lane store could not be deployed. `extension_path` names a
LOCALLY BUILT gdrive.duckdb_extension — fine on a developer's machine,
impossible in a container — and it is the only way to register
write_blob/remove_file/file_size, which live in the extension rather than
DuckDB core and are needed for EVERY root, not only `gdrive://`.

Leaving it unset does not help, and is the worse failure: `None` SKIPS the
load and assumes the functions are present. The store then opens cleanly and
the first WRITE fails with "Scalar Function with name write_blob does not
exist". A boot that succeeds and a write path that cannot work is exactly the
shape that reaches production looking healthy.

gdrive now ships in the DuckDB community repository, which duckvfs.rs already
anticipated in a comment. `extension_repo` is that: `community`, or a
repository URL. Wired through as ESCUREL_STORAGE_DUCKVFS_EXTENSION_REPO.

A NEW field rather than repurposing `extension_path: None`. Redefining None
would change behaviour for existing `file://` roots and force a network
install where none happens today; `extension_path` keeps its exact meaning
and still wins, so an operator pointing at a local build gets that build.

The test asserts a WRITE round trip, not that the constructor returns Ok:
`new` performs no I/O against the root, so a store that never loaded the
extension constructs perfectly. It takes no ESCUREL_TEST_GDRIVE_EXTENSION,
because needing a prebuilt file is the thing being removed.

Two checks, and the second is the one that mattered. Mutating extension_repo
to None fails it with the missing-write_blob error, so it is load-bearing.
And it first passed in 0.65s — too fast for a download, because
gdrive.duckdb_extension was already in ~/.duckdb from local use. Re-run with
a cold HOME it downloads, populates the fresh cache and passes, which is the
container case; the deployment already sets HOME to a writable dir. Without
that check this would have been green here and broken everywhere else.

1377 passed, 0 failed. clippy clean.
@jrosskopf
jrosskopf merged commit aa0fead into main Sep 1, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant