Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support installation of extensions from central repository #2569

Merged
merged 88 commits into from
Nov 29, 2021

Conversation

hannes
Copy link
Member

@hannes hannes commented Nov 10, 2021

No description provided.

@hannes hannes changed the title Support installation from extension from central repository Support installation of extensions from central repository Nov 10, 2021
@hannes hannes mentioned this pull request Nov 26, 2021
@Mytherin
Copy link
Collaborator

This PR adds support for (remotely) installable extensions. The INSTALL command can be used to install remote extensions, and the LOAD command can be used to load those extensions.

For example:

INSTALL tpch;
LOAD tpch;

Extensions will be downloaded by default from http://extensions.duckdb.org. The system will download the extension for the correct version/platform/architecture by navigating to the correct location, e.g. by downloading from http://extensions.duckdb.org/<github_rev>/linux/amd64/tpch.duckdb_extension. Extensions will be installed in the ~/.duckdb directory.

Extensions can also be installed by providing a local path (INSTALL '/path/to/file') or by providing a custom URL (http://www.myextension.com).

In order to facilitate these installable extensions, we now ship httplib by default with DuckDB. Note that we ship httplib without OpenSSL, which means https:// addresses cannot be accessed.

How it works

This PR contains a lot of changes. The bulk of these changes is getting these installable/loadable extensions to work on Windows.

The main problem is that external extensions face a chicken-and-egg problem. The extensions need to call into methods provided by DuckDB (e.g. registering functions in the catalog). However, the extension is loaded by DuckDB. DuckDB itself might be located anywhere, and could be statically compiled into a source program, which means there might not be a duckdb.so available.

On POSIX systems, this is solved quite elegantly by leaving the symbols as undefined. When the extension is loaded the undefined symbols are resolved and everything works as-is.

On Windows, this is not supported. However, we can simulate this behavior using delay loading. We compile the extension using the /DELAYLOAD flag, and then provide a delayload hook that looks for symbols in the programs' current address space (GetModuleHandle(NULL)) when trying to load symbols from duckdb.dll.

This solves the problem, but delayloading has a few restrictions. First, as with all Windows DLL stuff, all non-inlined functions need to be marked with __declspec(export)/__declspec(import). That means that DUCKDB_API needs to be added to any function that is used in an external extension.

Second, data cannot be delay loaded, only classes/functions. This means that if we have e.g. a global array, we cannot export this using DUCKDB_API and use it in an extension. This was a problem primarily because we had a few of these in common usage (e.g. LogicalType::INTEGER was a data entry).

This PR resolves this problem by moving several of the commonly used data elements to to constexpr elements. By making them constexpr the data entries are inlined into where they are used, and no linking is required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants