Connecting to an in-memory SQLite database for testing -- table not found

[timwilson]

First Check

• I added a very descriptive title here.
• I used the GitHub search to find a similar question and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# conftest.py
#
import pytest
import sqlite3
from sqlmodel import create_engine
from starlette.testclient import TestClient
from app import database
from app.auth import settings
from app.main import app

@pytest.fixture
def ro_client(in_memory_db):
    client_instance = TestClient(app)
    client_instance.headers["X-API-KEY"] = settings.ro_api_key
    return client_instance

@pytest.fixture
def rw_client(in_memory_db):
    client_instance = TestClient(app)
    client_instance.headers["X-API-KEY"] = settings.rw_api_key
    return client_instance

@pytest.fixture(autouse=True, scope="session")
def in_memory_db():
    # Create a new in-memory database connection
    memory_conn = sqlite3.connect(":memory:")

    # Create a connection to the on-disk production database
    production_conn = sqlite3.connect(f"{database.DB_PATH}{database.DB_DATABASE}")

    # Use the .backup method to copy the entire on-disk database into the in-memory database
    production_conn.backup(memory_conn)

    # For debugging: Print the tables in the in-memory database
    # memory_cursor = memory_conn.cursor()
    # memory_cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
    # tables = memory_cursor.fetchall()
    # print("\nTables in in-memory database:", tables)

    # Update the app's engine to use the in-memory SQLite database for the duration of the tests
    test_engine = create_engine(
        "sqlite:///:memory:", connect_args={"check_same_thread": False}, echo=True
    )
    test_engine.raw_connection().connection = memory_conn
    database.engine = test_engine

    yield  # Yield control to the tests

    # Cleanup
    memory_conn.close()
    production_conn.close()

#
# database.py
#
from sqlmodel import Session, create_engine
from app.config import get_settings

settings = get_settings()

DB_PATH = settings.db_path
DB_DATABASE = settings.db_name
DATABASE_URL = f"sqlite:///{DB_PATH}{DB_DATABASE}"

connect_args = {"check_same_thread": False}

# Allow for DATABASE_URL to be overridden (e.g., for testing)
def get_engine(database_url: str = DATABASE_URL, echo: bool = False):
    return create_engine(database_url, echo=echo, connect_args=connect_args)

# Default engine
engine = get_engine(DATABASE_URL, echo=(not settings.production))

def get_session():
    with Session(engine) as session:
        yield session

Description

I'm using SQLite for production, and for testing purpose I'd like to copy my production database into an in-memory SQLite database. I have confirmed that the in_memory_db fixture does successfully make the copy and is queryable from a Python command line. The problem is that when I run any test that requires database access, the test immediately fails and reports (along with a bunch of python code that looks like it's coming from SQLAlchemy):

FAILED test_models.py::test_create_location - sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: apikey

If both databases, the production one and the in-memory one, are the same then I'm not sure why the apikey table is not found. (I have confirmed that it exists in both databases.) Is it a problem with creating the engine?

Operating System

macOS

Operating System Details

Running inside Docker via Docker Desktop

FastAPI Version

0.104.0

Pydantic Version

1.10.13

Python Version

3.12.0

Additional Context

No response

[trentbuck]

I think I have the same problem.
Below is a slightly simpler test case.

This works fine:

import sqlmodel
import pprint

class Item(sqlmodel.SQLModel, table=True):
    id: int | None = sqlmodel.Field(primary_key=True, default=None)
    name: str

engine = sqlmodel.create_engine(
    url='sqlite:///',
    # echo=True,
    connect_args={'check_same_thread': False})

sqlmodel.SQLModel.metadata.create_all(engine)

with sqlmodel.Session(engine) as db:
    pprint.pprint(db.exec(sqlmodel.select(Item)).all())
    db.add(Item(name='test1'))
    db.add(Item(name='test2'))
    db.commit()
    pprint.pprint(db.exec(sqlmodel.select(Item)).all())

This has a problem:

import sqlmodel
import sqlmodel.ext.asyncio.session
import pprint
import fastapi
import typing
import uvicorn

class Item(sqlmodel.SQLModel, table=True):
    id: int | None = sqlmodel.Field(primary_key=True, default=None)
    name: str

engine = sqlmodel.create_engine(
    url='sqlite:///',
    # echo=True,
    connect_args={'check_same_thread': False})

sqlmodel.SQLModel.metadata.create_all(engine)

with sqlmodel.Session(engine) as db:
    pprint.pprint(db.exec(sqlmodel.select(Item)).all())
    db.add(Item(name='test1'))
    db.add(Item(name='test2'))
    db.commit()
    pprint.pprint(db.exec(sqlmodel.select(Item)).all())

app = fastapi.FastAPI()

def get_db() -> typing.Generator:
    with sqlmodel.Session(engine) as db:
        yield db

@app.get('/')
def f(db: sqlmodel.Session = fastapi.Depends(get_db)) -> list[Item]:
    return db.exec(sqlmodel.select(Item)).all()


if __name__ == '__main__':
    uvicorn.run(app)

The error (boring parts elided):

INFO:     127.0.0.1:32768 - "GET / HTTP/1.1" 500 Internal Server Error
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: item

I'm testing on Debian 12, so the libraries are relatively old:

bash5$ dpkg-query -W python3 python3-fastapi python3-sqlmodel python3-uvicorn python3-pydantic python3-sqlalchemy
python3 3.11.2-1+b1
python3-fastapi 0.92.0-1
python3-pydantic        1.10.4-1
python3-sqlalchemy      1.4.46+ds1-1
python3-sqlmodel        0.0.8-4
python3-uvicorn 0.17.6-1

Fundamentally, I think the problem is that the in-memory is thread-local or session-local, i.e. the fastapi routes see a different in-memory database to the non-fastapi code.

I'll go re-read the sqlmodel and fastapi docs, because I feel like I saw the fix for this documented in there somewhere...

[nachitox]

Any luck?

[mgreene2]

For anyone still having this issue, this should fix it:
engine = create_engine( "sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool )

More details can be found in the SqlAlchemy docs: https://docs.sqlalchemy.org/en/20/dialects/sqlite.html#using-a-memory-database-in-multiple-threads

And in the SQLModel docs: https://sqlmodel.tiangolo.com/tutorial/fastapi/tests/#memory-database

[YuriiMotov]

You have two different in-memory databases.
Every time you connect to in-memory databese, it creates new separate database. So,

    memory_conn = sqlite3.connect(":memory:")

and

    test_engine = create_engine(
        "sqlite:///:memory:", connect_args={"check_same_thread": False}, echo=True
    )

are two different databases. And you copy data only to first of them, but trying to use the second.

From the SQLite documentation:

In-memory Databases And Shared Cache

In-memory databases are allowed to use shared cache if they are opened using a URI filename. If the unadorned ":memory:" name is used to specify the in-memory database, then that database always has a private cache and is only visible to the database connection that originally opened it. However, the same in-memory database can be opened by two or more database connections as follows:

rc = sqlite3_open("file::memory:?cache=shared", &db);

Or,

ATTACH DATABASE 'file::memory:?cache=shared' AS aux1;

This allows separate database connections to share the same in-memory database. Of course, all database connections sharing the in-memory database need to be in the same process. The database is automatically deleted and memory is reclaimed when the last connection to the database closes.

If two or more distinct but shareable in-memory databases are needed in a single process, then the mode=memory query parameter can be used with a URI filename to create a named in-memory database:

rc = sqlite3_open("file:memdb1?mode=memory&cache=shared", &db);