Unable to open database file

[cdreetz]

Install and first run is successful but looks like problem with the logging function. Error is as below:

File "C:\Users\Christian\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Christian\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\llm\cli.py", line 257, in prompt
db = sqlite_utils.Database(log_path)
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\sqlite_utils\db.py", line 335, in init
self.conn = sqlite3.connect(str(filename_or_conn))
sqlite3.OperationalError: unable to open database file

Also when I llm logs off I get:

Traceback (most recent call last):
File "C:\Users\Christian\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Christian\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\Christian\desktop\simon\env\Scripts\llm.exe_main.py", line 7, in
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "C:\Users\Christian\desktop\simon\env\lib\site-packages\llm\cli.py", line 372, in logs_turn_off
path.touch()
File "C:\Users\Christian\AppData\Local\Programs\Python\Python310\lib\pathlib.py", line 1166, in touch
self._accessor.touch(self, mode, exist_ok)
File "C:\Users\Christian\AppData\Local\Programs\Python\Python310\lib\pathlib.py", line 331, in touch
fd = os.open(path, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\Christian\AppData\Roaming\io.datasette.llm\logs-off'

[jefftriplett]

I'm on macOS and seeing something similar.

I'm seeing "sqlite3.OperationalError: table sqlite_master may not be modified" after LLM run and echoes everything.

[simonw]

This issue is so frustrating. It's because macOS ships a version of SQLite that is locked down to the point that it's enable to execute advanced alter table statements.

There are a few workarounds:

1. Use the Python that comes with Homebrew
2. Run pip install sqlean.py in the same environment as LLM

That second option is probably what you need. The reason it works is that sqlean.py ships an entirely fresh SQLite installation (as a binary wheel) that doesn't have this limitation, and sqlite-utils then picks that up and uses it, see:

• sqlean support sqlite-utils#559
• Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified sqlite-utils#235

I should probably update LLM (or sqlite-utils) to catch that exception and output a recommendation to pip install sqlean.py itself directly.

[jefftriplett]

pip install sqlean.py fixed it for me. I use pyenv btw in case that's a helpful data point.

Should sqlean.py be more emphasized in the docs or installed by default if this is a known issue?

[eblume]

Hi, I am having this issue with a fresh install via either brew install llm or pipx install llm.

pip install sqlean.py doesn't fix me by default because I do not use either the homebrew python or the system default python, instead I use asdf-python. Presumably if I find the proper virtual environment and activate it, I could then install this module, but I wonder if there is an easier way? I currently use my ~/.tool-versions and ~/.Brewfile configurations to deploy my development environment across a number of machines this way, and having this manual step is a bit of a headache.

A quick fix to prevent the crash is to manually create the .../io.datasette.llm directory but this lacks portability. A slightly better kludge is to add this call:

diff --git a/llm/cli.py b/llm/cli.py
index ac3b3dc..ce47b74 100644
--- a/llm/cli.py
+++ b/llm/cli.py
@@ -254,6 +254,7 @@ def prompt(
     # Log to the database
     if (logs_on() or log) and not no_log:
         log_path = logs_db_path()
+        log_path.mkdir(parents=True, exist_ok=True)
         db = sqlite_utils.Database(log_path)
         migrate(db)
         response.log_to_db(db)

But as I understand this ticket, this isn't a real fix, as the root issue here is python pointing at an older version of sqlite3 than is otherwise available (in my case, via brew install sqlite3, which is keg-only). I don't really understand what sqlean.py is, but naively, can we just add it to the setup.py requirements? Wouldn't that then propagate the module to whatever environments are created by homebrew or pipx?

[montylounge]

pip install sqlean.py fixed it for me. I use pyenv btw in case that's a helpful data point.

Should sqlean.py be more emphasized in the docs or installed by default if this is a known issue?

Fixed it for me as well. Thanks!

[eblume]

I did some reading up on this and the following will (I think), when added to a startup profile like ~/.zshrc, allow sqlite-utils to find the proper homebrew-managed sqlite. The version I use is slightly more verbose and adds some errors hinting the user to install sqlite if it isn't already.

type brew &>/dev/null && brew ls --versions sqlite > /dev/null && export PATH="$(brew --prefix sqlite)/bin:$PATH"

I tested this with a fresh llm installation and confirmed that it is a good fix for me. I think for users who install llm via pipx install llm, and who are on MacOS using homebrew to manage sqlite but not python, this may be the cleanest fix. Personally, I think adding sqlean.py as a dependency or an "extras" is a perfectly fine use case for sqlite's embedded philosophy, but there's probably a lot to consider here that I'm not aware of.

[simonw]

This should be fixed in the latest LLM. Please re-open this issue if you run into the same problem again.