Pre-compiled extension does not load on Windows 11 (`The specified module could not be found`)

[reasv]

Same issue as #13

I'm on Windows 11 64 bit, Python 3.12, SQLite 3.45 and sqlite-vec = "^0.0.1a36"

I wrote this function to make the result more explicit:

def load_sqlite_vec(conn: sqlite3.Connection) -> sqlite3.Connection:
    sqlite_version = conn.execute("select sqlite_version()").fetchone()
    print(f"SQLite version: {sqlite_version}")
    import sys

    print(f"Python version: {sys.version}")
    # Load the SQLite vector extension
    conn.enable_load_extension(True)
    # sqlite_vec.load(conn)
    sqlite_vec_path = sqlite_vec.loadable_path()
    print(f"Loading SQLite vector extension from {sqlite_vec_path}")
    if os.path.exists(sqlite_vec_path + ".dll"):
        try:
            conn.load_extension(sqlite_vec_path)
        except sqlite3.OperationalError as e:
            print(f"Error loading extension: {e}")
        try:
            conn.load_extension(sqlite_vec_path + ".dll")
        except sqlite3.OperationalError as e:
            print(f"Error loading extension: {e}")
    else:
        print(f"Could not find SQLite vector extension at {sqlite_vec_path}")
    conn.enable_load_extension(False)
    return conn

Outputs:

SQLite version: ('3.45.3',)
Python version: 3.12.4 (tags/v3.12.4:8e8a4ba, Jun  6 2024, 19:30:16) [MSC v.1940 64 bit (AMD64)]
Loading SQLite vector extension from .venv\Lib\site-packages\sqlite_vec\vec0
Error loading extension: The specified module could not be found.

Error loading extension: The specified module could not be found.

The file vec0.dll is present, as is evident from os.path.exists succeeding.

[asg017]

Thank you for the detailed report!

So looks like you're trying the windows version of the sqlite-vec pip package, which is compiled with mingw.

Could you try this specific vec0.dll? This one was compiled with msvc, but I have had trouble running it https://github.com/asg017/sqlite-vec/releases/download/v0.0.1-alpha.37/vec0.dll

import sqlite3
db = sqlite3.connect(":memory:")
db.enable_load_extension(True)
db.load_extension("./vec0.dll") #  make sure it's from the link above!

Also, just curious - how did you install your version of Python? On my machine the mingw one works fine, but I dont think im on a MSC python build

[reasv]

I tested this file and it seems to produce the same error.

def load_sqlite_vec(conn: sqlite3.Connection) -> sqlite3.Connection:
    sqlite_version = conn.execute("select sqlite_version()").fetchone()
    print(f"SQLite version: {sqlite_version}")
    import sys

    print(f"Python version: {sys.version}")
    # Load the SQLite vector extension
    conn.enable_load_extension(True)
    # sqlite_vec.load(conn)
    # sqlite_vec_path = sqlite_vec.loadable_path()
    sqlite_vec_path = "./vec0"
    print(f"Loading SQLite vector extension from {sqlite_vec_path}")
    if os.path.exists(sqlite_vec_path + ".dll"):
        md5, sha256 = calculate_hashes(sqlite_vec_path + ".dll")
        print(f"MD5: {md5}, SHA-256: {sha256}")
        try:
            conn.load_extension(sqlite_vec_path)
        except sqlite3.OperationalError as e:
            print(f"Error loading extension: {e}")
        try:
            conn.load_extension(sqlite_vec_path + ".dll")
        except sqlite3.OperationalError as e:
            print(f"Error loading extension: {e}")
    else:
        print(f"Could not find SQLite vector extension at {sqlite_vec_path}")
    conn.enable_load_extension(False)
    return conn

I updated the function to use the local vec0(.dll) and to print the sha256 and md5 hash so you can verify it is the same file.

SQLite version: ('3.45.3',)
Python version: 3.12.4 (tags/v3.12.4:8e8a4ba, Jun  6 2024, 19:30:16) [MSC v.1940 64 bit (AMD64)]
Loading SQLite vector extension from ./vec0
MD5: 382449844d4b9cbb1b7e0d017945db62, SHA-256: d179b834501aa8af22cc866cbd2e838093a594a4bf06260009169ee2cf0fc986
Error loading extension: The specified module could not be found.

Error loading extension: The specified module could not be found.

As you can see it still fails to find the module, whether I import ./vec0.dll or ./vec0

[reasv]

I forgot to say, I installed it simply using poetry add sqlite-vec

[reasv]

So, I tried something else: since you said the version in the PyPI package was for mingw I tried using that dll with mingw python.

C:\msys64\mingw64\bin\python.exe
Python 3.11.9 (main, Apr 12 2024, 09:55:31)  [GCC 13.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> conn = sqlite3.connect(":memory:")
>>> conn.enable_load_extension(True)
>>> conn.load_extension("./vec0")
>>> conn.enable_load_extension(False)
>>> print(conn.execute("select vec_version()").fetchone())
('v0.0.1-alpha.36',)

This worked. However I'd like to use it with windows' python, since that's what my project uses. Any ideas? Should I try to compile it myself using MSVC? Are there any compilation instructions for this?

[reasv]

Another update:
I tried mingw python with the MSVC compiled DLL you provided:

Could you try this specific vec0.dll? This one was compiled with msvc, but I have had trouble running it https://github.com/asg017/sqlite-vec/releases/download/v0.0.1-alpha.37/vec0.dll

And it also works:

C:\msys64\mingw64\bin\python.exe .\main.py
SQLite version: ('3.45.2',)
Python version: 3.11.9 (main, Apr 12 2024, 09:55:31)  [GCC 13.2.0 64 bit (AMD64)]
Loading SQLite vector extension from ./vec0
MD5: 382449844d4b9cbb1b7e0d017945db62, SHA-256: d179b834501aa8af22cc866cbd2e838093a594a4bf06260009169ee2cf0fc986
('v0.0.1-alpha.37',)

Mingw python works with both DLLs, windows MSVC python does not work with either of them.

[reasv]

Another update:
The DLL seems to work with some python MSC versions. Specifically, Python 3.11 and 3.12 by Anaconda seem to work:

❯ ~\.conda\envs\textrain\python.exe .\main.py                   
SQLite version: ('3.41.2',)
Python version: 3.12.0 | packaged by Anaconda, Inc. | (main, Oct  2 2023, 17:20:38) [MSC v.1916 64 bit (AMD64)]
Loading SQLite vector extension from ./vec0
MD5: 382449844d4b9cbb1b7e0d017945db62, SHA-256: d179b834501aa8af22cc866cbd2e838093a594a4bf06260009169ee2cf0fc986
('v0.0.1-alpha.37',)

❯ ~\.conda\envs\imagen\python.exe .\main.py  
SQLite version: ('3.41.2',)
Python version: 3.11.5 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:26:23) [MSC v.1916 64 bit (AMD64)]
Loading SQLite vector extension from ./vec0
MD5: 382449844d4b9cbb1b7e0d017945db62, SHA-256: d179b834501aa8af22cc866cbd2e838093a594a4bf06260009169ee2cf0fc986
('v0.0.1-alpha.37',)

❯ ~\AppData\Local\Microsoft\WindowsApps\python.exe .\main.py
SQLite version: ('3.45.3',)
Python version: 3.12.4 (tags/v3.12.4:8e8a4ba, Jun  6 2024, 19:30:16) [MSC v.1940 64 bit (AMD64)]
Loading SQLite vector extension from ./vec0
MD5: 382449844d4b9cbb1b7e0d017945db62, SHA-256: d179b834501aa8af22cc866cbd2e838093a594a4bf06260009169ee2cf0fc986
Error loading extension: The specified module could not be found.

Error loading extension: The specified module could not be found.

The windows store version of python doesn't work. Why?
The MSC version differs (1916 vs 1940)

[reasv]

I noticed that the working versions have SQLite 3.41.2, whereas Windows Store python has SQLite 3.45.3.

I installed Windows Store python 3.10 which has SQLite 3.40.1 and it still throws the same error:

❯ python3.10 main.py
SQLite version: ('3.40.1',)
Python version: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Loading SQLite vector extension from ./vec0
MD5: 382449844d4b9cbb1b7e0d017945db62, SHA-256: d179b834501aa8af22cc866cbd2e838093a594a4bf06260009169ee2cf0fc986
Error loading extension: The specified module could not be found.

Error loading extension: The specified module could not be found.

Here is a gist with the full main.py file to reproduce:
https://gist.github.com/reasv/3843659083bb66fda88f2e6c7c5c9224

[reasv]

Conclusion

Upon further testing with other SQLite extensions, the same error appears, and only with Windows Store Python versions, Conda and Mingw seem to work just fine with the DLLs.

I have to conclude this is an issue with Microsoft Store Python, but I can't find any specific information about it.
Except this:
https://docs.python.org/3.11/library/sqlite3.html#sqlite3.Connection.enable_load_extension
It seems extension support can be disabled at compile time, but I'm not sure if this is the case here. No exception is raised with enable_load_extension, only the error you see when trying to load extensions.
Looking at stack overflow, it seems like if it was compiled without extension support, it would not allow you to call enable_load_extension():
https://stackoverflow.com/questions/57977481/how-to-use-enable-load-extension

[asg017]

Thank you again for the detailed reports, really appreciate it!

That is pretty weird - if the Microsoft store Python builds did disable extensions, an error would be raised when we call enable_load_extension(True).

I'll do some more digging on my side on this. For now I'll add some documentation that recommends avoiding window store Python builds, which isn't ideal, but nothing else seems to work.

As far as building yourself: You can git clone this repository and run make loadable, which create a dist/vec0.dll file. That being said, that Makefile may not be supported on Windows, so instead you could follow these instructions with the sqlite-vec.c/sqlite-vec.h files found here: https://github.com/asg017/sqlite-vec/releases/download/v0.0.1-alpha.37/sqlite-vec-0.0.1-alpha.37-amalgamation.zip

[reasv]

I've switched to python.org python and everything seems to work for me now .
I'm closing this issue since the problem doesn't seem specific to SQLite-vec.

[reasv]

Closing

[asg017]

Thank you again @reasv for your thorough notes!

[reasv]

Thank you, no problem. I hope you'll find out why this is happening eventually.