Upgrade --load-extension to accept entrypoints like Datasette

[simonw]

Imitate:

• Add new entrypoint option to --load-extension datasette#1789

# would load default entrypoint like before
datasette data.db --load-extension ext

# loads the extensions with the "sqlite3_foo_init" entrpoint
datasette data.db --load-extension ext:sqlite3_foo_init

# loads the extensions with the "sqlite3_bar_init" entrpoint
datasette data.db --load-extension ext:sqlite3_bar_init

[simonw]

I can borrow Alex's example extension for the tests too: https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/tests/ext.c

[simonw]

That didn't run the tests as expected on Ubuntu: https://github.com/simonw/sqlite-utils/runs/8048229213?check_suite_focus=true

tests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] SKIPPED [ 19%]
tests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] SKIPPED [ 19%]
tests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] SKIPPED [ 19%]

That should have worked because of this:

sqlite-utils/.github/workflows/test.yml

Lines 38 to 41 in 3f694e5

	- name: Build extension for --load-extension test
	if: matrix.os == 'ubuntu-latest'
	run: |-
	(cd tests && gcc ext.c -fPIC -shared -o ext.so)

[simonw]

Extension looks to be correctly compiled:

Run (cd tests && gcc ext.c -fPIC -shared -o ext.so && ls -lah)
total 412K
drwxr-xr-x 3 runner docker 4.0K Aug 27 05:40 .
drwxr-xr-x 8 runner docker 4.0K Aug 27 05:40 ..
-rw-r--r-- 1 runner docker    0 Aug 27 05:40 __init__.py
-rw-r--r-- 1 runner docker  760 Aug 27 05:40 conftest.py
-rw-r--r-- 1 runner docker 1.6K Aug 27 05:40 ext.c
-rwxr-xr-x 1 runner docker  16K Aug 27 05:40 ext.so
drwxr-xr-x 2 runner docker 4.0K Aug 27 05:40 sniff

So why doesn't this check pass?

sqlite-utils/tests/test_cli.py

Lines 25 to 30 in 3f694e5

	def _has_compiled_ext():
	for ext in ["dylib", "so", "dll"]:
	path = Path(__file__).parent / f"ext.{ext}"
	if path.is_file():
	return True
	return False

[simonw]

To run the tests locally on my Mac I needed to run Ubuntu in Docker (for the compilation step).

This worked:

docker run -it -v "$(pwd):/app" ubuntu:latest /bin/bash

Then:

apt-get update && apt-get install python3-pip gcc libsqlite3-dev -y
cd /app/tests
gcc ext.c -fPIC -shared -o ext.so
cd /app
pip3 install -e '.[test]'
pytest -k test_load_extension

[simonw]

Tests passed in CI against Ubuntu:

tests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] PASSED [ 19%]
tests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] PASSED [ 19%]
tests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] PASSED [ 19%]

[simonw]

cc @asg017 since this feature was inspired by his work on Datasette.