Add proper error messages and a doc page dedicated for testing

[PawelRoman]

It's a very nice library but extremely bad at handling initialization errors and almost no documentation on how to initialize things for unit tests (the only thing is a simple fastapi example using pytest).

Consider someone installing this lib and writing their first unit test. It may look like this:

from tortoise.contrib import test


class EmptyTest(test.TestCase):

    async def test_nothing(self):
        pass

This test will not pass. It will raise KeyError: 'apps'. The message is very confusing. It should return a human-friendly message saying something like "Tortoise ORM was not initialized. You need to call Tortoise.init first", pointing the user to the right direction.

But OK, we happen to be an experienced django dev, right? We understand it's something like trying to call manage.py without DJANGO_SETTINGS_MODULE pointing to a correct settings, right? OK, so we call init(). Let's say we do this:

from tortoise import Tortoise
from tortoise.contrib import test

class EmptyTest(test.TestCase):

    async def asyncSetUp(self):
        await Tortoise.init(
            db_url="psycopg://my_user:my_pass@127.0.0.1:5432/my_db",
            modules={"models": ["db.models"]},
        )

    async def test_nothing(self):
        pass

What is the expected behaviour here? Welll it will still not pass. The error will be AttributeError: 'EmptyTest' object has no attribute '_transaction'. Wait, what? What transaction? What is it about? But luckily, we vaguely remembered tutorial saying something about "importance of cleaning up". We're GUESSING (coz we don't know that from the error message) that maybe we're missing close_connections() on teardown:

from tortoise import Tortoise
from tortoise.contrib import test

class EmptyTest(test.TestCase):

    async def asyncSetUp(self):
        await Tortoise.init(
            db_url="psycopg://my_user:my_pass@127.0.0.1:5432/my_db",
            modules={"models": ["db.models"]},
        )

    async def asyncTearDown(self) -> None:
        await Tortoise.close_connections()

    async def test_nothing(self):
        pass

And only after all that guessing game, we finally made an empty test which is actually passing.

OK. The test is passing so let's now try to call fastapi endpoint from it. We have a websocket endpoint which makes a simple DB get_or_create query and automatically accepts webscocket connection. We tested it from Postman. It works 100%. The DB connection string is correct, everything is correct. Everything works so nicely with Tortoise & Fastapi when we're making websocket connection from Postman. So we want to do that programmatically. So we write the following test:

    from fastapi.testclient import TestClient

    async def test_nothing(self):
        client = TestClient(app)
        with client.websocket_connect(
            "/ws",
            headers={
                "X-Username": "testuser",
                "X-Auth-Token": "foobar",
            },
        ) as websocket:
            pass

We run the test. We're getting RuntimeError: Event loop is closed. Hmm... wait, maybe it's our fault. Our Fastapi app has a lifespan function where we run Tortoise.init() on startup and close_connections() on shutdown (BTW, this is not documented anywhere, the documentation is completely wrong on how to actually initialize the TortoiseORM with FastAPI - see my other ticket . OK, so maybe the Fastapi test client is already calling the lifespan function and doing the initialization and close_connection for us and this is causing the error? Let's remove startUp and tearDown functions we wrote earlier, it's now kind of similar to what's in the FastAPI example page, just the test client, making websocket connection.

So we remove the setUP and tearDown classes from the test. We run the test.

We're getting KeyError: 'apps' error again. Aghrrrr!!!!!!!!!!

I have an absurd situation where I have a perfectly working functionality of my websocket FastAPI endpoint, with TortoiseORM making queries successfully, but I have zero clue on how to write a goddamn 1 line unit test.

[PawelRoman]

OK, I found initializer and finalizer pattern in the documentation. I think there should be "Testing" tab in the main page. It should have two levels. First explain basics for each main python test framework (pytest, unittest, etc.) with full working examples, how to write an empty test. Then second level on how to integrate with each individual framework (FastAPI, etc.). It should have very basic examples, also including FastAPI websockets.

Anyway, I got back to a place where I try to create an empty test case, this time armed with initialzier and finalizer methods. I have this:

from tortoise.contrib import test
from tortoise.contrib.test import initializer, finalizer


class MyTest(test.TestCase):

    @classmethod
    def setUpClass(cls):
        initializer(
            modules=["db.models"],
            db_url="psycopg://my_user:my_pass@127.0.0.1:5432/test_mydb",
        )

    @classmethod
    def tearDownClass(cls):
        finalizer()

    async def test_nothing(self):
        pass

On the DB I ran

CREATE USER 'my_user' with password 'my_pass' 
ALTER USER 'my_user' superuser  ---> this is so it can drop/create the DB

I also created an empty DB with name test_mydb

When I run the above, I'm getting

psycopg.errors.ObjectInUse: cannot drop the currently open database from db_delete method in base_postgres.client()

[PawelRoman]

I made it!

Apparently, one can't use postgres for the test database. When using default which apparently is sqlite://memory I finally got my empty test to work.

The final version which is working:

from tortoise.contrib import test
from tortoise.contrib.test import initializer, finalizer


class MyTest(test.TestCase):

    @classmethod
    def setUpClass(cls):
        initializer(
            modules=["db.models"]
        )

    @classmethod
    def tearDownClass(cls):
        finalizer()

    async def test_nothing(self):
        pass

[abondar]

Hi!

Sad to see you struggling like that

Good that you made it work with sqlite, although I think it should work with postgres too.
You issue may be because tortoise initializer/finalizer designed in way, that create and destroy database, so it can fail in the moment of it's creation/deletion.
In your case seems like something else is connected to db in moment of delete, not sure what is it, but maybe you can troubleshoot that on your side, or at least narrow down why it's tortoise fault here

In our internal testcase we use this pytest fixture:

@pytest.fixture(scope="session", autouse=True)
def initialize_tests(request):
    # Reduce the default timeout for psycopg because the tests become very slow otherwise
    try:
        from tortoise.backends.psycopg import PsycopgClient

        PsycopgClient.default_timeout = float(os.environ.get("TORTOISE_POSTGRES_TIMEOUT", "15"))
    except ImportError:
        pass

    db_url = os.environ.get("TORTOISE_TEST_DB", "sqlite://:memory:")
    initializer(["tests.testmodels"], db_url=db_url)
    request.addfinalizer(finalizer)

And it works with all databases that in our testcase

If you have ideas how to improve documentation - I will be glad to accept PR with improvements on this part

[PawelRoman]

My frustration continues, but I'm closing this. I'll create e new thread with full example.