What's the purpose of haybarn?

[vergenzt]

I skimmed through the readme, main site, etc., and it's become pretty clear what this project is / what it does -- but I could not find any explanation of why. I don't see a technical difference, and the absence of explanation otherwise seems conspicious; the only thing I can think of is maybe there was an extension moderation / "governance" disagreement?

Care to explain? I'm eyeing this project a bit suspiciously without a clearer purpose. 🤨 What does this project do which the main DuckDB project itself doesn't?

[rustyconover]

Hey — fair question, and I'm glad you asked directly rather than just staying suspicious. 🙂

Not to disappoint you, but there's no manifesto of grievances behind Haybarn. There was no governance blow-up, no extension-moderation falling-out, no drama. If you came looking for a juicy fork story, this isn't one.

Query Farm's interests don't always line up with DuckDB Labs', but they're mostly complementary — which is true of basically any two businesses working near each other. Haybarn isn't a protest and it isn't a competitor to upstream DuckDB. It's a derived distribution: same engine, ABI-compatible, rebuilt and repackaged under its own name so it can stand on its own infrastructure, signing keys, and release cadence — and so we can push the parts of the stack we care about faster than waiting on upstream.

The "why" in one line: it's about owning the distribution — branding, a verifiable extension supply chain, and release timing — and having a place to ship engine/transport work that goes beyond what upstream does today. Which brings me to the part where I'd push back on "no technical difference."

What's actually different — technically

The engine stays deliberately ABI-compatible with upstream DuckDB: same duckdb:: namespace, same .duckdb_extension format, same platform string. That's a feature — it keeps forward-porting cheap. But the patch stack on top of the v1.5.x base is not just a rename, and it's all public, curated one-concern-per-commit. Some highlights:

Core engine

• HTTP request cancellation as a first-class engine capability. We added a cancellation hook on BaseRequest (a non-owning pointer to the query's interrupted flag) and a cancelled response state, and taught the retry logic to treat cancellation as terminal — so a cancelled query can abort in-flight remote reads instead of draining them. (The actual wire-level abort lives in httpfs; this is the engine-side interface.)
• Loading extensions straight from node_modules. A bare LOAD foo that misses the local extension dirs now resolves @haybarn/ext-foo-… packages installed via npm and loads them in place — still gated by the RSA signature check, nothing written to disk. It's the first step toward distributing extensions through npm/PyPI rather than only a binary repository.
• pg_catalog correctness for PostgreSQL wire clients. Strict PG-protocol clients now actually work: pg_attribute.atttypid maps through real PG type OIDs (instead of colliding with DuckDB's internal type ids), pg_type emits standard array rows and proper NOT-NULL columns so Npgsql's type loader doesn't crash on connect, and we added a batch of catalog stubs/functions (pg_roles, pg_extension, pg_relation_size, pg_get_indexdef, …) that SQLAlchemy and Npgsql probe.
• A single signing trust root + independent extension channel. Upstream's core and community public keys are replaced by one embedded Haybarn key, and the repositories point at our own R2-backed host. DuckDB-signed extensions won't load; Haybarn-signed ones will. This is the security model that makes the distribution genuinely self-contained.
• Plus real stability fixes picked up along the way (a dangling-reference SIGBUS on the extension-install retry path; a %n format-string bug that masked HTTP errors during exception construction).

httpfs — where we've moved furthest beyond upstream

• HTTP/2 with real cross-thread stream multiplexing. A new process-global dispatcher owns a CURLM* on a dedicated thread with CURLPIPE_MULTIPLEX; worker threads submit handles and wait. This is the only way to get true h2 multiplexing — CURLOPT_PIPEWAIT is a no-op under curl_easy_perform. The payoff: N parallel range reads collapse from N TCP+TLS connections to one connection with N streams. On by default.
• Server-arbitrated consistency via HTTP preconditions. We replaced the old client-side ETag string comparison (which had weak/strong/quoting representation bugs) with If-Match / If-Unmodified-Since preconditions and proper RFC 7232 strong/weak detection. If the remote object changes mid-scan you get a clear "remote file appears to have changed" error instead of stale bytes fed into the Parquet/CSV reader.
• Per-method Accept-Encoding negotiation + transparent decompression (gzip/deflate/brotli/zstd), with range reads and HEADs forced to identity so byte offsets stay correct — useful for the API-shaped payloads httpfs increasingly touches (S3 LIST XML, multipart responses, HuggingFace JSON).
• End-to-end cancellation that reaches the wire: cancelling a query aborts in-flight range reads in ~50ms instead of draining them — the consumer of the engine-side hook above.

wasm — extensions that load and authenticate in the browser

• The headline is OAuth from inside a wasm extension. We added duckdb_wasm_open_auth_url / duckdb_wasm_get_auth_error: the first opens an OAuth popup on the main thread and blocks the Worker on Atomics.wait against a SharedArrayBuffer until the auth code returns. That lets a browser-resident extension authenticate a user against a remote API — a surface upstream duckdb-wasm doesn't have.
• To make that work we rewrote the wasm HTTP layer so it returns response bodies on non-2xx statuses (upstream short-circuits ≥400 to null, which makes 302/401 OAuth flows impossible), passes Authorization through unencoded, and handles range 206s — over a unified binary wire format. Plus worker-reachable SHA-256/random crypto stubs for dynamically-linked side modules, a non-busy-spin worker sleep, and SAB-based sub-message-latency query cancellation.
• And the part you can see in the repo screenshots: INSTALL lindel FROM community; LOAD lindel; then running a Hilbert-curve query — in the browser. The signed-wasm-extension machinery is upstream's, but it only works when the engine, the trust root, and the extension channel all agree, and controlling all three as one unit is exactly what the fork buys. Stock duckdb-wasm can only load against DuckDB's trust root and extensions.duckdb.org.

So, the honest summary

Same engine semantics and file formats, by design — but a meaningfully different network stack (h2 multiplexing, conditional reads, end-to-end cancellation), a browser story that can load and authenticate community extensions, and a fully self-hosted signed distribution underneath it all. No conspiracy; just a place to own the distribution and ship transport/extension work at our own pace.

Today we just wrote a report about how Haybarn's CI can consistently build the entire community extension universe much faster than DuckLab's where users are still waiting for extensions to build more than 24 hours after a release: report.pdf

Everything above is in the public patch stack, curated one-concern-per-commit, so none of it requires taking my word for it — dig in and verify. Happy to go deeper on any of these.

[vergenzt]

Why not attempt to do propose those changes upstream? If doing so isn't worth the effort right for you at the moment... why publish a branded "fork" of DuckDB?

This seems like it could potentially negatively impact DuckDB's community by fracturing it.