How to build a static library with extensions

[IgKh]

What does happen?
The released official source amalgamation (libduckdb-src.zip) doesn't include the source to any of the extensions. For that matter, also the pre-built shared library packages do not include any of them.

To the best of my observation, only the Python / R packages and the CLI tool are built with extensions, but those aren't as usable from a C++ program.

What should happen?
My goal is to create a static build of DuckDB C++ library which also includes extensions (in particular, Parquet and httpfs). Ideally, as a single .a file plus minimal header files. A shared object would be acceptable as well. However there doesn't seem to be a straightforward way to do so.

To Reproduce
I tried building from the full source tree, by configuring CMake with -DBUILD_PARQUET_EXTENSION=1 -DBUILD_HTTPFS_EXTENSION=1. This does build those extensions, but the result of trying to create a re-distributable with a make install is that:

• The static libraries created for the extensions are not installed, neither are the headers that are needed to be able to call duckdb::DuckDB::LoadExtension<>() for them.

• There is a separate static library for each extension, which has a somewhat problematic name like libparquet_extension.a (it would have been better if there was a "duckdb" somewhere in the name)

• Archive files for the DuckDB mangled versions of vendored third party dependencies are also installed, with names that may collide with existing files (e.g. libfmt.a, which is not compatible)

I also noticed that there is a script for generating a Parquet source amalgamation. Did that in conjunction with generating an extended core amalgamation. Together with the extension/include/parquet-extension.hpp header file, it does make it possible to easily build a combined static library. However it looks like that wasn't the purpose of it, and it does not cover other extensions.

Is this use case intended to be supported, and how to best achieve it? Thank you!

Environment (please complete the following information):

• OS: [e.g. iOS] Linux
• DuckDB Version [e.g. 22] 0.2.8 + master

Before submitting

• Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?
• Have you tried this on the latest master branch? In case you cannot compile, you may find some binaries here: https://github.com/duckdb/duckdb/releases/tag/master-builds

[IgKh]

Managed to get most of the way there, by:

1. Patching CMakeLists.txt files of extensions by adding install commands for the extension target and its' main header (the one declaring the duckdb::Extension subclass), as well as setting the OUTPUT_NAME CMake property of the extension target to something more specific.
2. Configuring with -DAMALGAMATION_BUILD=1 as well, so that third party libraries won't get built as separate library targets.

Not sure if it is the way to go though.

[IgKh]

Going with the path outlined in the previous comment ended up a bit problematic as well, as the FOSSology based scanner used by my employer takes issue with the full source tree. That appears to be due to some non-free software components included in the repository (namely: duckdb/third_party/tpce-tool and duckdb/extension/tcpch/dbgen). As far as I can see, those are only used for benchmarking, and are not required for usage of DuckDB itself.

I think it would be highly useful if one of the release artifacts was a source tarball that has amalgamations for the core DuckDB library and the main extensions. If you think it is something good, I can try and update the amalgamating script to that end.

[Mytherin]

Going with the path outlined in the previous comment ended up a bit problematic as well, as the FOSSology based scanner used by my employer takes issue with the full source tree. That appears to be due to some non-free software components included in the repository (namely: duckdb/third_party/tpce-tool and duckdb/extension/tcpch/dbgen). As far as I can see, those are only used for benchmarking, and are not required for usage of DuckDB itself.

Those components are indeed not required at all for using DuckDB.

I think it would be highly useful if one of the release artifacts was a source tarball that has amalgamations for the core DuckDB library and the main extensions. If you think it is something good, I can try and update the amalgamating script to that end.

That sounds good to me. There is already some code to create a separate parquet amalgamation (scripts/parquet_amalgamation.py), although nothing exists for the HTTPFS extension as of yet. Happy to review a PR!

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stale for 30 days with no activity.