Using Dependency Injection for Testing Purposes

[michaelschmit]

Description

I have some dependencies I want to inject, like some logging and data access classes, for testing purposes (being able to mock them out for unit testing). Looking at the documentation it seemed like using the Depends functionality on the router methods would do what I needed it to do.

@router.get("/status", response_model=Status)
async def status(config: dict = Depends(config)):

Question 1:
Is this the right tool for the job?

If so, it seems like FastAPI/pydantic is trying to figure out those parameters for the OpenAPI spec. Since I am injecting those internally, ideally they don't show up in the OpenAPI spec. Also these are classes and pydantic seems to be erring with:

file "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydantic/json.py", line 59, in pydantic_encoder
raise TypeError(f"Object of type '{obj.class.name}' is not JSON serializable")

Question 2:
Is there a way to tell Fast API/pydantic to ignore documenting that parameter?

Additional context
I've tried to use classes rather than a dictionary to inject the dependencies, that didn't seem to make a difference. I also tried to make that class inherit from pydantic BaseModel and that just pushed the JSON serialization error reference down into my data access class.

I should also note, my plan was to use the .dependency_overrides to swap in my mock objects during testing.

Thanks.

[chbndrhnns]

For question 2, I believe that dependencies are not taken into account for the docs if you put them in the route instead of in the function parameters.

[dmontagu]

Dependencies are always added to the openapi schema, whether added via a function parameter, the route decorator, or in the include_router call.

If you don't need to make use of the result of a dependency call (e.g., if it always returns None), it is better to put it in the route decorator or an include_router call. (That way there is no confusion about its intention, and in the case of include_router, it may help reduce repetition.) It sounds like this is the case for your application @michaelschmit. So yes, I think it is the right tool for the job!

I'm not sure if there is a good way to tell FastAPI to not document a dependency if you use things like = Path(...), = Body(...), or annotating as pydantic models. However, you can declare dependencies that make direct use of the Request, and those will not end up documented.

If you can share the signature of your config function it might be possible to help further explain the error you are hitting.

[michaelschmit]

This is essentially what I'm trying to do:

async def config(data_access_1=DataAccessClass1,
                 data_access_2=DataAccessClass2):
        return {"DataAccessClass1": data_access_1,
                "DataAccessClass2": data_access_2}

@router.get("/status", response_model=Status)
    async def status(config: dict = Depends(config)):
        factory = Factory(config["DataAccessClass1"], config["DataAccessClass2"])
        return await factory.get_status()

This allows me to mock out the data access components for unit tests:

async def config(data_access_1=MOCK_DATA_ACCESS_1,
                 data_access_2=MOCK_DATA_ACCESS_2):
    return {"DataAccessClass1": data_access_1,
            "DataAccessClass2": data_access_2}

server.APP.dependency_overrides[router.config] = config

I'm sure you can appreciate the merits of that (Raise exceptions, return specific results, etc.).

With that said, the issue I have is that DataAccessClass1 and 2 is not JSON serializable which is what pydantic complains about. Therefore, the OpenAPI spec generation fails when hitting the URL.

Here is the error:

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydantic/json.py", line 59, in pydantic_encoder
    raise TypeError(f"Object of type '{obj.__class__.__name__}' is not JSON serializable")
TypeError: Object of type 'DataAccessClass1' is not JSON serializable

Is there anyway to get around that?

[dmontagu]

I would remove the MOCK_DATA_ACCESS things from the config signature, and instead make use of the dependency overrides provider functionality https://fastapi.tiangolo.com/tutorial/testing-dependencies/

[michaelschmit]

I would remove the MOCK_DATA_ACCESS things from the config signature, and instead make use of the dependency overrides provider functionality https://fastapi.tiangolo.com/tutorial/testing-dependencies/

Sure. I guess the point is moot however if in the end I can't generate the OpenAPI spec due to the pydantic error I included above.

[michaelschmit]

I did some more playing around and here is what I was able to do in order to get around the pydantic error:

async def config(data_access_1=None,
                 data_access_2=None):
        return {"DataAccessClass1": data_access_1,
                "DataAccessClass2": data_access_2}

@router.get("/status", response_model=Status)
    async def status(config: dict = Depends(config)):
        factory = Factory(config["DataAccessClass1"], config["DataAccessClass2"])
        return await factory.get_status()

Then in my factory, I can say that when my data_access_1 and _2 is None then supply the default instances.

class Factory:

    _data_access_1 = None
    _data_access_2 = None

    def __init__(self, data_access_1, data_access_2):
        if data_access_1:
            self._data_access_1 = data_access_1
        else:
            self._data_access_1 = DataAccessClass1()

        if data_access_2:
            self._data_access_2 = data_access_2
        else:
            self._data_access_2 = DataAccessClass1()

With that said, this dependency injection scheme still doesn't feel right because pydantic still documents them. From the URL you linked ([https://fastapi.tiangolo.com/tutorial/testing-dependencies/]) it outlines two use cases. One is an external service and the other is for testing a database. I would argue in both of these cases, you wouldn't want to document on a public api that you are using service x or database y. I suppose you could use an extremely crypt parameter to mask these, but it is still confusing to anyone looking at the open api spec.

It just doesn't feel quite right ...

[dmontagu]

Yeah, in general you shouldn’t use arguments to your dependency if you don’t want them to be documented (unless they are other dependencies without arguments, or just use the Request).

If you want to be able to override the values, use the dependency overrides provider.

[dmontagu]

If you remove the non-documentable parameters from the dependency signature you will not have problems generating the openapi spec. It’s running into errors because it doesn’t know how to generate docs for those types. You can’t put things in the dependency signature it doesn’t know how to handle.

[michaelschmit]

I think I "hacked" my way into a solution that will work. Essentially I am always setting the dependency override which will house my config:

def config(data_access_1=None,
                 data_access_2=None):
        return {"DataAccessClass1": data_access_1,
                "DataAccessClass2": data_access_2}

APP = FastAPI()
APP.dependency_overrides.update({"config": config})
APP.include_router(Router.router)

Then in my router methods that need the config, I'm doing:

        config = request.app.dependency_overrides["config"]()
        factory = Factory(config["DataAccessClass1"], config["DataAccessClass2"])
        return await factory.get_status()

This allows me to inject my dependencies (config) while also not exposing their existence on OpenAPI. Perhaps not the cleanest approach, but it works.

Beyond the above option, the only other way I could think of doing this is extending the FastAPI class to include a config variable. At which point I could retrieve those parameters via the request having a reference to the app (FastAPI class). Not sure which option is really better than the other.

Thanks.

[dmontagu]

That's not how the dependency overrides is intended to be used, but if it works, it works!

The docs should show how to use it in a more idiomatic way. Your code could be refactored to work the way it is supposed to, but if it works now I suppose there's not too much harm leaving it as is.

[michaelschmit]

I feel my code can't be changed to work the way dependency injection is supposed to work in FastAPI due to the requirement I don't want those dependencies exposed on my OpenAPI document because they are irrelevant to my API consumers. It is confusing and we should never expose any more information about our application than we need to.

I would actually argue that the "Use case: testing database" in the documentation outlines the specific case where FastAPIs implementation fails. Any dependency injection functionality that requires you to expose your dependencies to your consumer just to test them doesn't fit the description of dependency injection outlined by the larger software development community.

Perhaps I am just missing something ... always a possibility.

With that said, I really do appreciate your time and your quick responses. Overall FastAPI is a very nice library to leverage. Kudos to all the individuals that contribute.

Thanks.

[dmontagu]

@michaelschmit Your code definitely can be changed to work this way:

from typing import Any, Type

from fastapi import APIRouter, Depends, FastAPI
from starlette.testclient import TestClient


# Declare the access types and dependencies
class DataAccessClass1:
    pass


class DataAccessClass2:
    pass


class DataAccessConfig:
    def __init__(self, access_1: Type[Any], access_2: Type[Any]) -> None:
        self.access_1 = access_1
        self.access_2 = access_2


class AccessFactory:
    def __init__(self, access_types: DataAccessConfig) -> None:
        self._data_access_1 = access_types.access_1
        self._data_access_2 = access_types.access_2

    async def get_status(self) -> str:
        return f"{self._data_access_1.__name__}, {self._data_access_2.__name__}"


def get_config() -> DataAccessConfig:
    return DataAccessConfig(DataAccessClass1, DataAccessClass2)


async def get_access_factory(
    access_types: DataAccessConfig = Depends(get_config)
) -> AccessFactory:
    return AccessFactory(access_types)


# Create the app and routes
router = APIRouter()


@router.get("/status")
async def status(factory: AccessFactory = Depends(get_access_factory)) -> str:
    return await factory.get_status()


app = FastAPI()
app.include_router(router)

# Check the response is as expected
client = TestClient(app)
print("With default dependencies:")
print(f"Status: {client.get('/status').json()!r}")
"""
With default dependencies:
Status: 'DataAccessClass1, DataAccessClass2'
"""


# Declare test-time dependencies
class TestDataAccessClass1:
    pass


class TestDataAccessClass2:
    pass


def get_testing_config() -> DataAccessConfig:
    return DataAccessConfig(TestDataAccessClass1, TestDataAccessClass2)


# Update the dependency overrides
app.dependency_overrides[get_config] = get_testing_config

# Check the response uses the overridden dependencies
print("------")
print("With testing dependency overrides:")
print(f"Status: {client.get('/status').json()!r}")
"""
------
With testing dependency overrides:
Status: 'TestDataAccessClass1, TestDataAccessClass2'
"""

# Ensure the openapi schema looks good
print("------")
import json
print(json.dumps(app.openapi(), indent=2))
"""
{
  "openapi": "3.0.2",
  "info": {
    "title": "Fast API",
    "version": "0.1.0"
  },
  "paths": {
    "/status": {
      "get": {
        "responses": {
          "200": {
            "description": "Successful Response",
            "content": {
              "application/json": {
                "schema": {}
              }
            }
          }
        },
        "summary": "Status",
        "operationId": "status_status_get"
      }
    }
  }
}
"""

The above script is self contained and should give the same output shown in-line if you copy it into a python file and run it yourself.

Hope you find that useful!

[michaelschmit]

Perhaps I am just missing something ... always a possibility.

BINGO !!!! I apologize for you having to spoon feed me the answer.

If you remove the non-documentable parameters from the dependency signature you will not have problems generating the openapi spec.

This line was lost on me ... I was reading that as your dependency (Depends) parameter needs to be removed from the router signature because it returns something not documentable. Clearly not what you said.

Many, many thanks ...

[tiangolo]

Thanks for the help here everyone! 👏 🙇

Thanks for reporting back and closing the issue @michaelschmit 👍