[Bug]: sqlite3.OperationalError: database or disk is full (Not really)

[barrybriggs]

What happened?

Have approx 450,000 sentences I am putting into ChromaDB; the database is on my F: drive which has about 7TB free, so plenty. I put 20,000 sentences at a time into the db and made it about halfway through when I received the SQLite error above. I have tons of memory and the database drive is practically empty...then I (almost by accident) noticed that the C: drive was down to about 34k of space available!

are there temp files I should be aware of and if so how do I point them at the f: drive?

Versions

Anaconda Python 3.10.10, Chroma latest (yesterday), Windows 11, Intel i9, 64GB RAM

Relevant log output

No response

[tazarov]

@barrybriggs, we've had a similar recent problem. Can you check how much storage you have under /tmp? SQLite uses it to store large queries in temp.

[barrybriggs]

Unfortunately I just cleared out my \temp (Windows) folder (situation was a bit critical as space on C: drive went to zero). Does SQLite use the TMP or TEMP environment variable? -- if so, I could point it somewhere else and see if that works better. (Maybe I'll just try it anyway :-)).

EDIT: looks like SQLite uses an internal pragma to set the tmp directory -- maybe you have to force it to use the environment variable value.

[tazarov]

We're tracking this issue here too #1693 (comment).

Soon we'll either put in a guidance or directly fix it.

Here's the sqlite3 docs on the temp usage and pragmas - https://www.sqlite.org/tempfiles.html