Default django-filter types and overriding types

[indigane]

There's a long-standing issue in django-filter, where the typing of parameters ends up wrong in the resulting OpenAPI schema, due to them being hardcoded as 'string' in django-filter: https://github.com/carltongibson/django-filter/blob/6045fff/django_filters/rest_framework/backends.py#L164

This is the closest related GitHub issue that I am aware of carltongibson/django-filter#1104

Is it at this point possible to resolve this using drf-spectacular, and I just have not figured out how?

If not, then would it be plausible to add to drf-spectacular's contrib.django_filters something similar to this https://github.com/tfranzel/drf-spectacular/blob/master/drf_spectacular/openapi.py#L396 that would check if the parameters are instances of specific django-filter filters, such as filters.NumberFilter -> would map to some numeric type?

If that is too far-fetched as well, then is it possible to override only the parameter type using drf-spectacular? It would seem that if I override the parameter with OpenApiParameter then it replaces the whole schema for the parameter.

Or if none of these are anywhere near the solution, what would you suggest for solving the wrong typing, or what approach have you or others used? The library includes some support for django-filter so it does not seem to be a completely new use case.

[tfranzel]

hi @indigane! i think what you are asking for is already existing:

drf-spectacular/drf_spectacular/contrib/django_filters.py

Line 38 in 6cd3cc8

class DjangoFilterBackend(SpectacularDjangoFilterBackendMixin, OriginalDjangoFilterBackend):

By using the patched drf-spectacular version instead of the django-filter version of DjangoFilterBackend, the types should be as expected.

though i'm thinking about deprecating that class and actually adding filter extensions. In that case everything should work out of that box. i already have a local WIP branch for that.

[tfranzel]

i chose to include the extension. please don't use the suggested DjangoFilterBackend class, as this should now work out of the box.

can you confirm this now works as expected?

[stnatic]

@tfranzel We've been using django-filter to filter on a query expression (annotation) rather than a model field. While django-filter allows it, drf-spectacular schema view is now crashing starting from e16e730.

  File "/home/krzysztof/.env/fix/lib/python3.8/site-packages/drf_spectacular/openapi.py", line 353, in _get_filter_parameters
    parameters += filter_extension.get_schema_operation_parameters(self)
  File "/home/krzysztof/.env/fix/lib/python3.8/site-packages/drf_spectacular/contrib/django_filters.py", line 42, in get_schema_operation_parameters
    schema=auto_schema._map_model_field(model_field, direction=None),
  File "/home/krzysztof/.env/fix/lib/python3.8/site-packages/drf_spectacular/openapi.py", line 373, in _map_model_field
    assert isinstance(model_field, models.Field)

Exception Type: AssertionError at /api/docs/schema

[wonderbeyond]

@tfranzel I encountered a similar problem to stnatic.

django-filter also provides filters such as django_filters.rest_framework.OrderingFilter which can't map into model fields.

[wonderbeyond]

@tfranzel I've just found another related issue, that is the "description" in the result of _map_model_field function has no effect, because the OAS 3.0 schema object is not allowed to have a "description" property.

[tfranzel]

@stnatic @wonderbeyond apparently i have not considered all use cases of django-filter, but opted-in all users. that was not the best decision on my part.

i'll try to improve this. @stnatic can you give an example of that?

in the end we can fall back on "string" if map_model_field fails, because that is exactly what django-filter is doing.

[wonderbeyond]

@tfranzel

in the end we can fall back on "string" if map_model_field fails, because that is exactly what django-filter is doing.

However, we can still do some effort if map_model_field fails. In fact I'm trying to fix in my own application scope.

FYR:

class FixDjangoFilterExtension(OpenApiFilterExtension):
    """
    See https://github.com/tfranzel/drf-spectacular/issues/234#issuecomment-752215103
    This Extension class can be removed if the problem is resolved.

    See also https://github.com/wonderbeyond/django-filter/blob/d0f82ac091d1cee96eb71f76e959b30136c51e8e/django_filters/rest_framework/backends.py#L20
    """
    priority = 1
    target_class = 'django_filters.rest_framework.DjangoFilterBackend'
    match_subclasses = True

    _FILTERS_OPANAPI_TYPE_MAP = {
        filters.CharFilter: 'string',
        filters.BooleanFilter: 'boolean',
        filters.NumberFilter: 'number',
    }

    def get_schema_operation_parameters(self, auto_schema, *args, **kwargs):
        def get_filter_description(filter):
            field = filter.field
            return field.help_text if field.help_text is not None else filter.label

        def get_filter_openapi_type(filter):
            for filter_type, openapi_type in self._FILTERS_OPANAPI_TYPE_MAP.items():
                if isinstance(filter, filter_type):
                    return openapi_type
            return 'string'

        model = get_view_model(auto_schema.view)
        if not model:
            return []

        filterset_class = self.target.get_filterset_class(auto_schema.view, model.objects.none())
        if not filterset_class:
            return []

        parameters = []
        for field_name, field in filterset_class.base_filters.items():
            path = field.field_name.split('__')
            model_field = follow_field_source(model, path)

            description = get_filter_description(field)

            if isinstance(model_field, models.Field):
                schema = auto_schema._map_model_field(model_field, direction=None)
                description = description or schema.pop("description", None)
            else:
                schema = {'type': get_filter_openapi_type(field)}

            parameters.append(build_parameter_type(
                name=field_name,
                required=field.extra['required'],
                location=OpenApiParameter.QUERY,
                description=description,
                schema=schema,
                enum=[c for c, _ in field.extra.get('choices', [])],
            ))

        return parameters

[stnatic]

@tfranzel Imagine the following data structure:

Team:

• id - the primary key field
• name - a short text field

TeamMembership:

• id - the primary key field
• team_id - foreign key (related_name="members")
• user_id - foreign key
• role - one of "primary owner", "owner", "member", etc.

In a viewset you could introduce an "extra field" by creating an annotation. I'm using an overly complicated example, but something as simple as Upper(F("name")) should break it as well:

def get_queryset():
    return Team.objects.annotate(
        primary_owner=FilteredRelation("members", condition=Q(members__role=TeamRole.PRIMARY_OWNER)
     ).select_related("primary_owner)

Then the django-filter part would look like this:

class TeamFilter(filterset.FilterSet):
    primary_owner = filters.ModelChoiceFilter(
        queryset=User.objects.all(), label="Primary owner", method="filter_by_primary_owner"
    )

    def filter_by_primary_owner(self, queryset, name, value):
        return queryset.filter(primary_owner__user=value)

    class Meta:
        model = Team
        fields = [
            "primary_owner",
        ]

Another example inspired by @wonderbeyond would be to .annotate(size=Count("members") and try to use OrderingFilter to sort by the SQL column "size".

[indigane]

Before I submitted the issue, I had tried to use the DjangoFilterBackend but still all django-filter filters were of type 'string', so I was wondering if maybe that's not what it does, or that there's something else I had not understood.

It may take a while before I can get back to test it again, so if you say it works then that is fine.

In other cases where there is need for customizing typing, what would you recommend? Is the only way to override the full schema with OpenApiParameter and write it by hand?

[tfranzel]

@indigane, when i changed the behaviour to better automatic introspection, i did not account for some edge cases of django-filter. @wonderbeyond,@stnatic and #252 pointed that out.

i will do a fix later today to remedy that. i will try to fix the assertion errors first and foremost, while also trying to pick up more edge cases

[indigane]

Updating from 0.11.1 to 0.12.0 did fix the typing of django-filter filters, thanks!

What I tried in 0.11.1 was to add

    "DEFAULT_FILTER_BACKENDS": [
        "drf_spectacular.contrib.django_filters.DjangoFilterBackend",
    ],

to the settings, and I tested with debug prints that it did run the filter backend, but the typing didn't change. Not that it matters since it has been deprecated.

[tfranzel]

thank you all for the feedback. i addressed the core issue and tried to do some improvements.

filter functions should now be handled gracefully.

@stnatic def filter_by_primary_owner(self, queryset, name, value): cannot be introspected, however you can annotate ...,value: int): to give an an override type. otherwise the filter type class will be used now (e.g. NumberFilter)

@wonderbeyond thanks for sharing your approach. i incorporated some of your changes into the solution. much appreciated.

@indigane: sounds weird with the drf_spectacular.contrib.django_filters.DjangoFilterBackend. it worked everywhere for me somehow, but yeah it does not matter anymore as long as the new system works properly.

please let me know if there are any issues left so that we can close this issue.

[wonderbeyond]

@tfranzel Great work!
Have you noticed the description issue above? #234 (comment)

[martinlehoux]

@tfranzel Thanks a lot for your work! I have the same issue with a custom ChoiceFilter. Can you explain the difficulty to implement the filters following this line

# TODO underspecified filters. fallback to STR
# filters.ChoiceFilter: None,
# filters.TypedChoiceFilter: None,
# filters.MultipleChoiceFilter: None,
# filters.DurationFilter: None,
# filters.NumericRangeFilter: None,
# filters.RangeFilter: None,
# filters.BaseCSVFilter: None,
# filters.LookupChoiceFilter: None,

I'd be happy to help improve this!

[tfranzel]

@tfranzel I've just found another related issue, that is the "description" in the result of _map_model_field function has no effect, because the OAS 3.0 schema object is not allowed to have a "description" property.

@wonderbeyond sry i saw it but overlooked it later because multiple people raised different issues here.

i see the problem but i'm pretty sure your statement is incorrect.

description - CommonMark syntax MAY be used for rich text representation.

is explicitly mentioned in the schema object. also all model properties are schema objects and thus all existing description usages would be invalid (which they are not).
apparently the parameter object description takes precedence over the description coming from the nested schema. i think that is the core of the problem. this additional commit should fix the issue.

[tfranzel]

I have the same issue with a custom ChoiceFilter.

@martinlehoux, which one specifically? there were multiple issues discussed at the same time.

Can you explain the difficulty to implement the filters following this line

yes. for example, the choice field in itself has no obvious type (at least to my knowledge). this is not that bad because if there is a model field behind it as we use that. a few of those filters can certainly be improved (e.g. MultipleChoiceFilter with collectionFormat: multi) structurally, however type-wise there is probably no easy improvement.

there are 3 approaches in the code now:

• type from model field
• type from method annotation
• type from filter class

happy to hear any suggestions.

[wonderbeyond]

@tfranzel Sorry I'm late here!

I took a quick review on your recent commits, mainly on the drf_spectacular.contrib.django_filters.resolve_filter_field function.

drf-spectacular/drf_spectacular/contrib/django_filters.py

Lines 56 to 84 in 6febe8f

	def resolve_filter_field(self, auto_schema, model, filterset_class, filter_field):
	if filter_field.method:
	filter_method = getattr(filterset_class, filter_field.method)
	filter_method_hints = typing.get_type_hints(filter_method)
	if 'value' in filter_method_hints and is_basic_type(filter_method_hints['value']):
	schema = build_basic_type(filter_method_hints['value'])
	else:
	schema = self.map_filter_field(filter_field)
	else:
	path = filter_field.field_name.split('__')
	model_field = follow_field_source(model, path)
	if isinstance(model_field, models.Field):
	schema = auto_schema._map_model_field(model_field, direction=None)
	else:
	schema = self.map_filter_field(filter_field)
	if 'choices' in filter_field.extra:
	enum = [c for c, _ in filter_field.extra['choices']]
	else:
	enum = schema.pop('enum', None)
	if filter_field.label is not None:
	description = filter_field.label
	else:
	description = schema.pop('description', None)
	return schema, description, enum

1. django-filter's Filter object has an implicit help_text parameter, see https://github.com/carltongibson/django-filter/blob/eb50a95e35a8b4fadfa95d685697ab295c11e0d6/django_filters/filters.py#L135

   This parameter is not fully documented, here are the two search results: https://django-filter.readthedocs.io/en/stable/search.html?q=help_text&check_keywords=yes&area=default

   However, I've used to take Filter.help_text as the prior OpenAPI parameter description because:

   1. Filter.label should be short.
   2. Model.help_text maybe less relevant since it's describing the model field but I want a description more about the filtering method, for example range, greater than etc.

   So I suggest to consider using below priority order to determine the filter parameter's description:

   1. Filter.help_text.
   2. Filter.label.
   3. Model.help_text.
   4. None

   I also suggest to introduce a setting key to enable and disable the ability to take Model.help_text as REST API description, since the comments on the model field may not be intended (at least for me) to exposed to front-end users.

2. I suggest that the OpenAPI parematers resolved from filter definition could take precedence over those resolved from model fields, instead of the current implementation:

   drf-spectacular/drf_spectacular/contrib/django_filters.py

   Lines 69 to 72 in 6febe8f

   	if isinstance(model_field, models.Field):
   	schema = auto_schema._map_model_field(model_field, direction=None)
   	else:
   	schema = self.map_filter_field(filter_field)

   Because the filter definitions are more specific, and a filter input could be much different from a model field. For example:

   # In models.py
   class Comment(models.Model):
       creator = UcUserField()
       created_time = models.DateTimeField(auto_now_add=True, editable=False)
       thread = ThreadField()
       parent = models.ForeignKey(
           "self",
           null=True, blank=True,
           on_delete=models.SET_NULL,
           related_name="replies",
           help_text="Reply to which comment",
       )
       content = models.TextField()
       tags = ArrayField(models.CharField(max_length=100), default=list, blank=True)
   
       class Meta:
           ordering = ['-created_time']
   
   
   # In filters.py
   class CommentFilter(filters.FilterSet):
       leading = filters.BooleanFilter(
           field_name='parent',
           label="Leading",
           lookup_expr="isnull",
           help_text='A "leading" comment is not a reply to another comment.',
       )
       parent = filters.ModelChoiceFilter(
           field_name='parent',
           to_field_name='id',
           queryset=Comment.objects.filter(parent__isnull=True),
           help_text="Reply to which comment?",
       )
       on = filters.CharFilter(field_name='thread', label="On", help_text="Comments **on** which object?")
       around = CommentAroundFilter(field_name='thread', label="Around", help_text="Comments **around** which object?")
   
       class Meta:
           model = Comment
           fields = []

   In the Comment's model definition, there's a parent field (of type Integer) that represents to which the comment replies to. And in the Filter definition, I define a leading parameter which use the Comment.parent as model query filed, but the Filter type is boolean. So in this scenario the OpenAPI schema resolved from the model field is just wrong.

[tfranzel]

@wonderbeyond

regarding 1: i added the help_text from extra. i think that should do it. it takes precedence over label as there is only one spot for documentation "description". there is no summary afaik.

regarding 2: i see the problem. the reason i chose this path is because filters are not always more descriptive. For example ModelChoiceFilter has no type in itself. you would gain nothing here, if the priority was reordered. generally i'm open to improving this, but it needs to be more intelligent than just changing the order. i see the problem in all non-basic filters.

i want to push out a new release so we ideally should wrap this up soon.

[tfranzel]

i'll close this issue as the majority of items (including the first issue) have been resolved.

for more improvements to django-filter we can create fresh issues. thanks all!