searching failing in multiple scenarios

[jerzyk]

I've found a multiple scenarios where existing column search is failing badly:

• search on fields that are foreign keys or numbers (eg. integers) - this will fail on "proper" databases (not sqlite)
• search on fields that are aliases to some other columns, e.g.:

detail_name = serializers.CharField(source='detail.name', read_only=True)

existing code assumes incorrectly that field name is the database field to query

most of those searches will fail twice - on "general" search - where top search field is used and then second time on the specific field

I do not have a clear idea how to resolve this, but those my first ideas:

• there should be an implementation that is similar to the existing ones - where searchable fields will be specified on python side - not pushed from request - this is potentially a security/performance leak
• there should be an option that is defining which fields are taken into account in "global" search
• there should be a detection and/or specification of default search operators per field type, where a safe options should be set as defaults (e.g. __exact or __iexact) and user could override them (eg. __icontains/__istartswith etc.)
• maybe there will be an option to use existing filtering backends in combination with this one?

[neighlyd]

Hi jerzyk,

I'm the person who implemented FK search using the name field. I'm guessing that your FK and integer searches are failing because you aren't including them in your name field, but I could be wrong there.

I'm not sure I would say that I "incorrectly" assumed that "field name is the database field to query," I simply made a decision to convert that field into a database query payload.

The reason I used name to specify search fields instead of a python-side approach was to reduce the amount of python code needed to implement this package. The name field was just the best way to deliver this information from frontend to backend while remaining DRY and friendly with datatables' format. It was my understanding that the original author wanted this to be a very plug-and-play friendly package, with minimal alterations to existing backend code base. Which is why the only existing alterations to backend are necessary changes to INSTALLED_APPS, REST_FRAMEWORK, and the optional datatables_always_serialize.

I'm personally not opposed to pushing the search and filter logic onto the backend python side of things. I actually would favor that approach a lot. But that seems like it might move beyond the scope that @izimobil originally envisioned.

This would most certainly also increase the complexity of the project by magnitudes as well. As you say, you do not have a clear idea of how to resolve this. I don't either, but I am definitely open to brainstorming with you. Though, I must be honest and admit that my time is extraordinarily limited and I'm quite obviously new to programming.

[izimobil]

Hey @neighlyd , @jerzyk ,
Yes, the original idea was to provide a plug'n'play app that is very simple to configure, that said I understand that the real use cases are always more complex. As I don't have such issues in my project using DRFD, I'm not sure of the right approach either... It would be very cool if you could do a brainstorming on this and propose a solution.
Thanks !

[izimobil]

In no particular order, maybe the right approach would be:

• Fixing the issue with real databases raised by @jerzyk
• Stick to our minimal filtering/searching capabilities and raise warnings for other usecases instead of failing badly
• Add an example of advanced filtering backend to support complex usecases

What do you think ?

[jerzyk]

• Stick to our minimal filtering/searching capabilities and raise warnings for other usecases instead of failing badly

imho - sorting/filtering should always be off, user should switch it on even if there are some shortcuts (like for the fields selection -> 'all') but this should be developer decision to switch it on, with possible limit of allowed operators (e.g. to allow only 'exact' on choice/model choice)

• Add an example of advanced filtering backend to support complex usecases

that's fine but maybe lets see if there is an option to use DRF filtering ecosystem instead of inventing everything again, it is really a PITA to implement some API options twice - for data grid and then for e.g. dropdowns

[stephendwolff]

Hiya, just come across this issue, and i think i agree with @jerzyk, in that search fields should be off by default.

I'm the person who implemented FK search using the name field. I'm guessing that your FK and integer searches are failing because you aren't including them in your name field, but I could be wrong there.

@neighlyd one idea could be that fields are searchable only if they have the data-name attribute in the table header? (that would have saved me some head scratching as you suggest - once i had data-name attributes the double underscore django cross table notation worked just fine)

[WillowMist]

I believe I'm experiencing a similar issue, but not with foreign objects, but calculated fields on the object. For example, I have a calculated field that checks all of the foreignkey objects for the latest entry, and failing to find that, defaults to a field on the object itself.

So I have a method on the object called 'get_medals' -- Search and sort are both failing because it can't find that field on the model.

[izimobil]

@DarkSir23 : Filtering on a field that is rendered by a method is not feasible, to avoid an error, you should set "searchable" and "orderable" to false in you JS code: https://datatables.net/reference/option/columns.searchable

[izimobil]

Thanks to @TauPan, DRFD now fully supports django-filter.
I'm closing this issue, basic filtering is good enough for small projects, and more complex filtering can no be done quite simply with django-filter.