Everything C API v2!

[evertlammerts]

Hi there, beloved community :-)

We are proud to announce the 2nd version of our C API, completely rewritten with the goal to provide you with a single API to build your extensions and clients with. This API is available in experimental mode starting with v2.0.0, and we need you to help us work towards stabilizing it. We'd love to do this over the course of DuckDB's v2 line, in small pieces. With stabilization of the API come guarantees: extensions and clients that are built on stabilized symbols only will not have to recompile across minor and patch releases. DuckDB guarantees a stable ABI and your binaries will Just Keep Working ™️.

In this discussion we want to ask you for any and all feedback you have. What works? What doesn't? Which features do you need us to add? Also tell us about your experience with the C++ API we built on top of it (duckdb::cxx / duckdb_cpp.hpp), or about the new Rust bindings.

A few notes: It's totally fine to link your bug reports and issues here as well if they contribute to the discussion. But please do not report them here, or use this forum as a place to make them visible. Also note our community discord channel, which is most suitable for getting community support around specific questions you may have.

[Maxxen]

There are some thing we know we already want/need to change that could be good to be aware of

Remove duckdb_v2_str

The duckdb_v2_str is really just a simple struct { char* ptr, int64_t len }; helper that was originally introduced to ensure all strings passing through the C-API had an associated length. However, this turned out to be problematic in some FFI environments which don't implement the full C calling convention and cant pass c-structs by value (see the discussion in #24991). To support these environments better we changed the API so that functions that used to take duckdb_v2_str now take a const duckdb_v2_str *, which adds an extra bit of indirection, but is not really a big deal in the grand scale of things.

Unfortunately this turns out to also be problematic, but this time when interfacing with the API from languages in garbage collected runtimes (e.g. golang, C#, and java?) as these runtimes require that e.g. no pointers passed to C are pointing to memory that contain other pointers managed by the runtime.

There are usually solutions for this that involve either "telling" the runtime to not move these pointers during garbage collection (e.g. Go's pinner), or making a copy of the data into the C-heap through some "unsafe" native memory allocation call. While these workarounds don't add a catastrophic amount of runtime overhead, they do add extra implementation and maintenance complexity.

So with all this in mind, keeping the duckdb_v2_str around adds a bunch of (albeit small) runtime cost and ergonomic friction, which kind of go against the whole reason we added it to begin with.

We therefore plan to (before stabilization) just remove the duckdb_v2_str and change the functions that currently take it into taking separate const char* ptr and idx_t len parameters instead. This is a large, but almost entirely mechanical change, and should not have any impact on the user-facing surface of the "higher level" wrappers like duckdb::cxx and the duckdb_neo rust crate.

[hello-world-bfree]

+1 to removing duckdb_v2_str. I use the capi with luajit for a few things and it passes a lua string as const *char with no copy. the struct pointer forces a ffi.new every call

[Maxxen]

Rework/fix expressions and table-filter-pushdown

The V2-C-API adds basic read-only introspection of some of DuckDBs bound expressions, which is currently only used to support filter-pushdown for table functions.

These are currently very much borked with multiple known defects, e.g. #25391, #27475.

Part of the complexity here is trying to make a simpler and "generic" interface that can handle all of DuckDBs complex expression types without completely blowing up the API surface - while we at the same time are actively trying to rework and simplify how we represent different expressions in core (#24008, #22490) . There are also other small ergonomic issues in this part of the API, e.g. getting the qualified name from a function expression performs an allocation/requires destroying the qname handle, not all function properties can be inspected, etc.

I don't really have anything concrete to say here more than: we know filter-pushdown is important, we want to expose a small but powerful expression API for both inspecting/creating expressions, we're actively working on making changes in core DuckDB to streamline this, this part of the API will likely change significantly before being stabilized. One way I kind of envision this to eventually look like is having a set of "leaf" expression types (constants, column refs), and a small set of "special forms", e.g. CASE/CONJUNCTION/TRY that have non-standard evaluation semantics, while everything else falls under a FUNCTION expression.

[thomas-0816]

Which features do you need us to add?

PHP-PDO has an inTransaction method (see https://www.php.net/manual/en/pdo.intransaction.php), currently I'm implementing:

extern "C" int duckdb_has_active_transaction(duckdb_connection conn) {
	try {
		auto *conn_ptr = reinterpret_cast<duckdb::Connection *>(conn);
		return conn_ptr->context->transaction.HasActiveTransaction() ? 1 : 0;
	} catch (...) {
		return 0;
	}
}

[Maxxen]

Yeah having something like this make sense, we should also consider adding the other explicit transaction lifecycle methods like conn.Begin()/Commit()/Rollback()/SetTransactionMode().

[thomas-0816]

Which features do you need us to add?

For unwrapping variants, I'm currently using VariantValue::GetValue().
duckdb_v2_vector_get_value() seems not returning the inner type of variant.

extern "C" int duckdb_variant_to_vector(duckdb_connection conn, duckdb_vector vec, idx_t row,
                                         duckdb_vector *out_vec, duckdb_logical_type *out_type) {
	(void)conn;
	try {
		auto *vec_ptr = reinterpret_cast<duckdb::Vector *>(vec);

		// GetValue on a VARIANT vector already returns the inner value, unwrapped to
		// its concrete type (INTEGER, DECIMAL(width,scale), TIMESTAMP_NS, BIGNUM, ...).
		// Unwrap once more defensively; it is a no-op for already-typed values.
		auto value = VariantValue::GetValue(vec_ptr->GetValue(row));
		if (value.IsNull()) {
			return 0;
		}

		auto *tmp_vec = new duckdb::Vector(value.type(), 1);
		tmp_vec->SetValue(0, value);
		*out_vec = reinterpret_cast<duckdb_vector>(tmp_vec);
		*out_type = reinterpret_cast<duckdb_logical_type>(new duckdb::LogicalType(value.type()));
		return 1;
	} catch (...) {
		return 0;
	}
}

[Maxxen]

I think duckdb_v2_vector_get_value() on a variant vector returns a variant value? which can then be cast to any value with a value-cast. There is also a duckdb_v2_vector_cast which will cast the whole vector to a target type.

[thomas-0816]

from ds 4.1 flash, I'm not sure it is correct:

duckdb_v2_vector_get_value() returns an owned value. For a VARIANT cell that value is still boxed as VARIANT — it does not return the inner type (capi_v2_vector.cpp:428-441). 

duckdb_v2_value_cast() can cast it to a type you already know, or to VARCHAR, but there is no API to discover the carried type or unwrap it generically.

duckdb_v2_vector_cast() That does not fit a VARIANT cell, because the type carried by a VARIANT is a runtime, per-row property. The inner type can differ from row to row, and duckdb_variant_to_vector returns that row's actual type in out_type. A single target vector can only have one type, so vector_cast cannot produce the same result. 
duckdb_v2_vector_cast only helps when you already know the single destination type.

So it's the functionality of VariantValue::GetValue() which I'm currently using to unwrap variants to normal types.

[Maxxen]

Right, there is no way to know what a variant contains, thats true. Im not 100% we want to provide this, generally the user needs to make a decision of what format they want to unwrap a variant into, because the variants internal type-system is not the same as duckdbs general type system. But ok, we could consider adding something like VariantValue::GetValue.

[mskyttner]

Could be related? #27570

[mskyttner]

I have experimented with using zig 0.17 to build an extension using comptime wrappers for the C API v 1 and v 2 (preview) using a fork of the "zig-duckdb-ext" repo and wonder how to use "table in-out functions" with the C API v 2?

[Maxxen]

"table in-out" functions are not currently supported! We may look into adding them in the future, but I think they should/need to be separated from regular table-functions first as they actually share very little logic.

[mskyttner]

Curious about a high level picture of what other things are not yet/now supported with the C API v2 but possible when building C++ extensions?

[Maxxen]

At the top of my head - statistics, optimizer extensions, custom operators, custom client/instance state, secrets.
The goal is to more or less provide everything eventually, but some things are not currently as extendable/stable as we would like them to be internally, and so require more work in core until we are comfortable comitting them to the C-API.

[rustyconover]

Thanks for putting this together. A stable ABI would make a huge difference for us, so I'm happy to give detailed feedback.

Some background: at Query.Farm we maintain around 45 community extensions (airport, adbc_scanner, datasketches, crypto, httpserver, tributary, radio, redis, shellfs, quickjs, a5, geosilo and a bunch more). Right now every one of them gets rebuilt for every DuckDB release on v1.4, v1.5 and v2.0. I went through each extension and checked what it uses from DuckDB internals against duckdb_v2.h as of 94abd4e, to figure out what we could port today and what's blocked.

The good news first. The simple scalar/aggregate extensions (crypto, hashfuncs, bitfilters, datasketches, rapidfuzz, tsid, marisa, lindel, etc.) look like a fairly mechanical port. A few things I really like:

• The per-thread init callback on scalar functions. quickjs, jsonata, evalexpr_rhai and minijinja all rebuild their interpreter per chunk (or per row in quickjs's case) today because there wasn't a good place to keep per-thread state. This fixes that.
• Function docs per overload. We register descriptions and examples for everything, so having this built in is nice.
• Ownership being spelled out on every function. That makes FFI work a lot less guessy.
• Parameter kinds with named args and defaults, plus resolve_types, cover all of our bind-time return type logic.

Now the stuff that blocks us, roughly in order of how many extensions it hits.

Running SQL / getting an instance from extension code. This is the big one. Callbacks get a duckdb_v2_context_handle, but the only things you can do with it are get_config, get_factory, get_file_system and log. To execute anything you need a connection, which needs an instance handle, and there's no way to get one from the extension or the context. The entrypoint doc actually says the context is one "to read and run queries through" (around line 2627 of the header), so I think something is missing here or the comment is out of date.

This blocks cronjob (scheduler thread runs queries), httpserver (runs queries on server threads), webmacro, quackscale and radio. tributary also logs from librdkafka callback threads, and duckdb_v2_instance_log would work for that if we could get an instance handle.

Related to this, we'd need:

• some way to get the instance (duckdb_v2_context_get_instance or similar)
• lifecycle callbacks: instance shutdown so we can stop background threads cleanly, plus connection open and query begin/end, which we currently use via ClientContextState and OnConnectionOpened
• per-database state, something like ObjectCache. Function user data is per-function and process globals break when there are multiple databases open.

Secrets. There's no secrets API at all. redis, tributary, openprompt, airport and adbc_scanner all register secret types and look secrets up. We could switch to settings or named parameters, but users already have secrets set up so that would be a step backwards for them.

HTTP. No access to HTTPUtil. A lot of our extensions make HTTP requests and currently go through DuckDB's HTTP stack so they pick up the user's proxy and CA settings. Without it each extension has to bundle its own HTTP client and TLS and ignore those settings. quackscale goes further and replaces the instance's HTTPUtil to route traffic over Tailscale.

Storage/catalog extensions. airport and adbc_scanner are built on StorageExtension with custom catalogs, schemas, tables and transaction managers, plus INSERT/UPDATE/DELETE/CTAS/ALTER and rowid. I get that this is a big surface and still moving in core. It would just be useful to know if it's planned for the v2 line at all, since that decides whether those two stay on the C++ API for the long haul.

Table functions.

• In-out table functions (agree with @mskyttner). inflector and airport use them.
• Column statistics, virtual columns/rowid, and a cardinality callback instead of only a bind-time estimate.
• A way to check for interruption, or return "blocked, try again later", from exec. radio and tributary have scans that block waiting on messages.
• Filter pushdown: @Maxxen's plan sounds right to me. For airport the main thing is being able to walk the whole expression (LIKE, struct extract, ORs, etc.) because we send filters to a remote server. A stable serialized form would be even better.

Types.

• geosilo is a BLOB with type modifiers (CRS to scale) and a modifier bind. v2 custom types are just a name and a base type. The doc for factory_create_type_with_alias mentions custom type bind callbacks but I couldn't find any API for them.
• Casts can't see the target type's modifiers, since there's no cast bind.
• GEOMETRY can only be read or written per cell with vector_get_value/set_value, which goes through a cast every row. a5 writes geometry in bulk so that would be slow. We'd also need a way to read the CRS, not just whether one is set.
• Registering Arrow extension types (geosilo has queryfarm.geosilo) and a hook for mapping Arrow types on import (adbc_scanner, airport).
• Minor: a union value constructor, and a push-back style helper for list children.

Missing extension points.

• Filesystem registration. shellfs is entirely a FileSystem subclass.
• Parser, operator, optimizer and planner extensions (dialectic, airport).
• An autocomplete hook and catalog enumeration for fuzzycomplete. tokenize_sql and describe_table help, but we can't list what's in the catalog, and we can't run duckdb_tables() either because of the SQL problem above.

Smaller things.

• context_log only takes a type string. tributary, airport and events register structured LogTypes.
• stochastic uses the per-client RandomEngine so setseed() works. Without it results aren't reproducible.
• openprompt reads and writes user variables from inside a function.
• jsonata autoloads the json extension.
• Aggregates have no window callback or state serialization, and unlike scalar/table callbacks they don't get a context handle.

VGI. Our biggest extension is vgi, which exposes scalar, table, aggregate, table in-out and COPY functions implemented by remote workers (subprocess, HTTP, unix sockets, containers, Iroh, and in the browser under WASM), with data moving over Arrow IPC. It also ATTACHes remote catalogs. It's a good test of how far v2 can go, and right now only around 10-15% of it could move: the diagnostic table functions, global settings, logging, file I/O for the result cache, and Arrow conversion for the basic types. Beyond everything above, it would need:

• Registering functions when a catalog is ATTACHed, not just at load. VGI discovers its scalar, table, aggregate, in-out and COPY functions from the worker and creates them per schema. It also creates secret types and settings per ATTACH. v2 only has extension_register_* at load or connection_register_* from a client. We'd want either a catalog-scoped lookup (schema + name returning a function handle) or registration tied to an attached catalog, and an unregister on DETACH. There's no way to remove a function, setting or secret type today, so things registered at ATTACH outlive the catalog.
• Async table function exec. VGI prefetches the next batch from the worker while DuckDB processes the current one, and under WASM everything has to be cooperative on one thread. We'd need exec to be able to say "blocked, call me back", plus a way to check for interruption.
• A cancellation callback. When a query is cancelled we forward that to the remote worker, which today means a background thread with its own Connection. An on-interrupt callback per query would be much simpler.
• Window aggregates where we get the whole frame. VGI sends frames to the worker in batches, so it needs window init with access to the partition and the frame bounds for each output row.
• Arrow extension types, and zero-copy. We register GeoArrow today. On the zero-copy side, VGI has a shared memory transport, and the v2 importer always copies into data chunks. An option to wrap Arrow buffers for fixed-width and string view columns, and to export straight into caller-owned buffers, would help a lot.
• Expression pushdown with "required" filters (some worker functions refuse to run without a filter on certain columns, and we check that after optimization), scan statistics from the worker, and bind data serialization.
• A way to add our own metrics to EXPLAIN ANALYZE. We report cache hits and RPC timings there.
• Structured extra info on errors (worker, RPC method, etc.), not just a code and a message.

A few questions:

1. The header says the aggregate destroy callback "runs on states that are discarded without being finalized" (around line 5272), but the implementation hooks it up to SetStateDestructorCallback, which runs on every state. Which one is right? It matters for whether finalize should free the state. If we guess wrong it's either a leak or a double free.
2. When a call succeeds, *err is left alone, so if you reuse an error slot it can still have an old error in it. Is the expectation to always check the return code and never the slot? Might be worth saying in the docs.
3. Everything is marked unstable right now, and duckdb_extension_v2.h won't compile without DUCKDB_V2_API_ALLOW_UNSTABLE, so a v2 extension is still tied to one exact DuckDB version. Could the function registration parts (scalar, aggregate, table, cast, types, vectors, docs) be stabilized first? That alone would let a lot of community extensions stop rebuilding every release.
4. Is the Rust crate for v2 out yet? Our Rust extension is still on the v1 C API through the duckdb crate and I didn't see v2 bindings in the repo.

On duckdb::cxx: it's nice to use. RAII handles and exceptions mapping to the error slot make registration feel close to the current C++ API. Some notes:

• ScalarExecutor doesn't handle string or blob args/results, which is most of what we write, so you end up back on raw vectors.
• There's no aggregate helper like UnaryAggregate, so every aggregate is a hand-written loop over the states.
• It's a header plus a ~7k line .cpp you have to compile in. Header-only, or a prebuilt lib in the extension template, would make builds simpler.
• tools/cpp/example/README.md still talks about "the stable C++ API" and points at a cpp_api/example path that doesn't exist.

And +1 on dropping duckdb_v2_str for ptr + len.

If it helps, we're happy to port a few of the simpler extensions (crypto, hashfuncs, and datasketches, which is code-generated so it's a decent stress test) to v2 + duckdb::cxx and report back. Also happy to try out any proposals for instance access, secrets or lifecycle hooks against real extensions.

[Maxxen]

Theres a lot to unpack here.

Running SQL / getting an instance from extension code

This is intentionally left out. Running SQL from inside a callback doesn't make sense as the connection is already busy executing the query which itself executed the callback. This is also why the new extension entrypoint doesnt receive a connection either.

In theory, if you had access to the instance, you could open a new connection and execute whatever there, but then you lose transactionality, and this newly opened connection will not have the same internal context as your callback has, so settings, state, etc will not propagate.

This is why extensions have (and will need more) dedicated C functions to modify the state of the database through the context/catalog, as these modifications run in the same transaction as the callback. Granted, most things you can change from a callback currently aren't truly transactional (like settings), but anything we add in the future that modifies the catalog (creating tables, schemas, functions etc) should be.

If you really need to run SQL specifically, you can define a PRAGMA function (which basically expands into SQL), and have the user invoke it. But you probably should not run SQL from a table/scalar/copy function, or other callback. The extension-load callback is debatable I guess, but like I said we will most likely expand the C-API to be able to make catalog modifications imperatively from C.

lifecycle callbacks: instance shutdown so we can stop background threads cleanly, plus connection open and query begin/end, which we currently use via ClientContextState and OnConnectionOpened
per-database state, something like ObjectCache. Function user data is per-function and process globals break when there are multiple databases open.

We'll add something like this eventually for sure! I've been sketching a something like a generic scoped "service" interface, but we might also just start with something simpler.

Secrets.
HTTP

These are indeed planned to be added, but both are undergoing or have lots of planned changes ATM so we're gonna wait a bit with them.

Storage/catalog extensions

For sure, I've sketched this too but its a massive API surface. But indeed we also have a lot of catalog extensions (sqlite, mysql, postgres) that are blocked on this.

Table functions

In-out functions need some rethinking in general. As far as interruption goes, yes, I expect us to eventually expose some primitives for this as the async-io work continues to progress.

Types

I have tons of ideas for custom types and made a bunch of foundational changes in core to eventually make them more powerful (like having dedicated constructors and function-signatures), but again we are blocked on core work here (including some pretty messy changes binding, and to storage to make them persistent) to really make it work.

As far as geometry is concerned, you can cast geometry values to/from BLOB (WKB) or VARCHAR (WKT). You can also get the CRS through the type_get_child accessor. (type parameters and nested types are both represented as values in the new API).

Missing extension points.

Filesystem: #25916

Parser, operator, optimizer and planner extensions.

Sure, but again, this is enormous API surface, and we are not ready to commit to stabilizing optimizer or planner extensions any time soon as there are way too many things we want to change in our internal plan/expression representation and optimizer framework.

As far as the parser goes I dont know as I'm not really involved in that work, but I presume we could add custom parser rules relatively easy once we iron out more of how exactly parser extensions are supposed to interact with eachother (and the PEG parser matures)

An autocomplete hook and catalog enumeration for fuzzycomplete. tokenize_sql and describe_table help, but we can't list what's in the catalog, and we can't run duckdb_tables() either because of the SQL problem above.

Like I mentioned above, we'll add a catalog access interface eventually.

Everything is marked unstable right now, and duckdb_extension_v2.h won't compile without DUCKDB_V2_API_ALLOW_UNSTABLE, so a v2 extension is still tied to one exact DuckDB version. Could the function registration parts (scalar, aggregate, table, cast, types, vectors, docs) be stabilized first?

Not right now, we're still making changes (e.g. remove duckdb_v2_str will impact all of these), but surely for 2.1

That alone would let a lot of community extensions stop rebuilding every release.

Yes this is the mid-term goal, but there are still benefits of targeting the C-API even now: massively improved build/CI times and much smaller binaries (that can be aggressively LTO'd).

Is the Rust crate for v2 out yet?

Not on crates, but hopefully in the coming days/weeks. You can find it here https://github.com/duckdb/duckdb-rs/tree/main/crates/duckdb-neo

When a call succeeds, *err is left alone, so if you reuse an error slot it can still have an old error in it. Is the expectation to always check the return code and never the slot? Might be worth saying in the docs.

Yes. Im pretty sure we say that the error code is authoritative, but i'll have to check. duckdb_v2_error_info_destroy is also safe to call with a NULL so you can just always call it if you wanna be paranoid.

As far as the duckdb::cxx goes, we will continue to improve it over time (and as we start moving some of our extensions to it as well), but

It's a header plus a ~7k line .cpp you have to compile in. Header-only, or a prebuilt lib in the extension template, would make builds simpler.

Keeping the .cpp separate allow us to completely hide the actual duckdb_v2.h header. I suppose we could ship it as an amalgamated single-header stb-style lib, but the plan now is to make a cmake module that you can fetch/find_package' which would provide a target that links both the header and the implementation.

[Maxxen]

But in general, great feedback! I'll make sure to keep these use cases in mind! Having a thorough review of all the capabilities currently in use is awesome to have.

[rustyconover]

This is intentionally left out. Running SQL from inside a callback doesn't make sense as the connection is already busy executing the query which itself executed the callback. This is also why the new extension entrypoint doesnt receive a connection either.

Ducklake does this all the time.

[rustyconover]

In-out functions need some rethinking in general. As far as interruption goes, yes, I expect us to eventually expose some primitives for this as the async-io work continues to progress.

I'm interested to learn what you're thinking about in-out functions, honestly I think they're kind of misnamed and too often developers think they operate like what I call a buffered function, meaning you read all input then produce all output. But really they operate more like a unix pipe.

[Maxxen]

Ducklake does this all the time.

Right, but not really to modify the same database as the user is running the extension on. It creates a new connection on the same instance, but attaches to a separate metadata catalog, and the connection doesnt run in autocommit, but instead embeds itself into the overarching storage-extensions custom transaction, and ties its own explicit BEGIN/COMMIT calls to the start/end callbacks of the custom transaction so the two are in lockstep. Its a pretty advanced use case - which we could probably support once we get to storage-extensions - but its not the same as just executing SQL from an arbitrary callback.

But ok, I think having a context_get_instance (or an extension_get_instance) is not a bad idea, and I can imagine that there are other things we might want to scope to an instance that you may need to do from an extension/callback (like loading another extension).

I'm interested to learn what you're thinking about in-out functions, honestly I think they're kind of misnamed and too often developers think they operate like what I call a buffered function, meaning you read all input then produce all output. But really they operate more like a unix pipe.

Yeah, they function much more like a regular "operator" than a "table function", which is also the direction I'm thinking we should go with them (maybe they should share the same sink/source interface as PhysicalOperators do). They should definitely have separate bind/exec callbacks from the table_function though and much more first-class than just an extra callback on the regular table-function (that somehow changes the whole binding/execution logic if present).

[hello-world-bfree]

What works? What doesn't? What features should we add?

Works:

• pull-style client hot paths, fewer callbacks the better
• DUCKDB_V2_ERROR wrapping all functions
• clear ownership from the spec with destroy setting handle to null on failure (relying on this always being the case would be fantastic!)

Improvements/features:

• I could just be missing it, but I don't see a v2 equivalent of appender for bulk imports. Is that planned?
• a wait handle for event loops (similar to PQsocket in libpq)
• runtime abi check for ffi bindings
• optional and borrow lifetimes are a bit of a pain. making them readable by generators would be huge
• document the entrypoint contract outside of macros
• extension footer as a format to avoid having to repro append_metadata.cmake

[Maxxen]

Great points!

I could just be missing it, but I don't see a v2 equivalent of appender for bulk imports. Is that planned?

not sure, the idea for now is that you can essentially implement an appender yourself with a column_data_collection and the bind_collection - thats essentially what the old appender is under-the-hood.

a wait handle for event loops (similar to PQsocket in libpq)

I think @evertlammerts is working on this! Being able to support something like a wakeup-notification callback was one of the main goals of the query result fetching overhaul we've done internally the last couple of months.

runtime abi check for ffi bindings

Could you elaborate? Extensions do this implicitly in the entrypoint, but I suppose we could add something like this for clients to inspect as well. There is already a duckdb_v2_library_version - but it doesnt necessarily mean the ABI version.

optional and borrow lifetimes are a bit of a pain. making them readable by generators would be huge

Yeah this is def something we want to reconsider. I've been trying to make it so that some handles are always owned, or always borrowed - but there are still some exceptions here (like datachunk, qname)

document the entrypoint contract outside of macros

Maybe - we will probably change the entrypoint soon while everything is unstable, but we can consider prioritizing "stabilizing" and documenting it early on.

extension footer as a format to avoid having to repro append_metadata.cmake

Yes, we're also looking into creating and distributing dedicated/reusable tooling for extension development to e.g. scaffold/test/fetch specific API versions - and also sign/update the footer which should make this easier to deal with.