Missing documents in NOT filtered arrays

[adrianrudnik]

Describe the bug

Just searched a bit, but did not find any documentation or hint if that is a defect or expected:

Having documents with a property tags = [] are not found if I do a NOT filter (like tags != "adult") on it.

Only documents that have 1-n tags are found.

Same behaviour as the other cases where tags is missing in the document and tags is null. These seem to be special cases I can expect. The empty array is not a thing I expected.

The docs don't go into nulls, missing or empty arrays. According to this meilisearch/meilisearch#1102 it should've been possible with meilisearch/meilisearch#1421. Then there is docs#837, but thats just about missing documentation on nulls.

Also the approach of "NOT (arrayValue = "someValue")" does not seem to work any more.

To Reproduce

curl -X POST 'http://localhost:7700/indexes/movies/documents' \
  -H 'Content-Type: application/json' \
  -d '[ { "id": "123", "tags": [] }, { "id": "124", "tags": ["free"] } ]'

curl -X POST 'http://localhost:7700/indexes/movies/settings' \
  -H 'Content-Type: application/json' \
  -d '{ "filterableAttributes": ["tags" ] }'

curl -X POST 'http://localhost:7700/indexes/movies/search' \
  -H 'Content-Type: application/json' \
  -d '{  "q": "", "filter": "tags != \"adult\"" }'

1 search result returned.

Expected behavior

2 search results expected.

MeiliSearch version: Docker image getmeili/meilisearch:v0.24.0

[gmourier]

Hi @adrianrudnik, thanks for the feedback! 👋

We are thinking of introducing capabilities like NOT attribute or attribute IS EMPTY/IS NULL for more specific filtering cases.

If I understand correctly your feedback, you expect that the expression tags != "adult" should return documents that do not have the tags attribute, or have a null/empty tags attribute. Is this correct?

Thank you!

@meilisearch/docs-team That might interest you!
@Kerollmops Is the fact that tags != "adult" does not return documents with tags: [], tags: null or documents without this attribute shouldn't be considered a bug?

[adrianrudnik]

Hello, yes that is correct. The expression tags != "adult" should at least consider all documents that have a tags attribute, even if it is just an empty array. The other cases could be special and sure, a NOT attribute would be great for those.

[netdev-julien]

Hey there,

We're migrating from Meilisearch 0.19 to 0.25.2, and we're exactly in this situation.
We were able (before expliciting filterable attributes) to search and make NOT on empty attributes, exactly like @adrianrudnik said.

I think this is a kind of regression.

We made a dirty (dirty) hack, replacing NULL/empty arrays with a 'empty' string (litteraly the string 'empty'), but this is not acceptable in a long term.

I am surprised that this issue is not especially discussed. Is this a bad practice? If that's the case, then I'm curious to hear opinions on how to better index our objects.

[gmourier]

Hi @netdev-julien 👋

Thanks for your feedback.

We have work to do on operators to improve the use of Meilisearch. In particular, add an IN or CONTAINS operator and manage this notion of null marker and undefined attribute.

We are currently focused on other topics within the core team but we will surely come back to this.

Thanks!

[gmourier]

Hello dear community, 👋

We have published a docker image allowing you to test the behavioral improvement regarding the NOT / != operator.

NOT / != behavior

tags != adult now returns documents having tags: null or tags: [] but also documents without the tags property. It's equivalent to NOT tags = adult.

The NOT operator selects all documents not selected by the filter that follows it.

---

EXISTS operator

We took the opportunity to add the operator EXISTS. This operator allows you to granularly require that the selected documents have a tag attribute regardless of its value.

In this case, tags != adult and tags EXISTS to retrieve documents that do not contain the adult value in the tag attribute and only those for which the tag attribute exists.

• tags EXISTS selects all documents having the tags attribute.
• NOT tags EXISTS selects all documents missing the tags attribute.

IN operator

Instead of writing a cumbersome clause like user_id = 1 OR user_id = 2 OR user_id = 3 .... It's possible to write user_id IN [1, 2, 3] and voila!

• user_id IN [...] selects all documents whose attribute contains at least one of the values given inside the brackets. In case there are no values, then the filter selects no documents.
• NOT user_id IN[...] selects all documents whose attribute contains values that are all different from those in the brackets. (It includes documents missing the attribute - See EXISTS operator for more granularity).

---

We would like to have your feedback before deciding whether to introduce the given changes in a future version.

To do so you have to pull the docker image we prepared.

docker run -it --rm \
    -p 7700:7700 \
    -v $(pwd)/meili_data:/meili_data \
    getmeili/meilisearch:v0.29.0-filter.beta.0

Thank you so much for being part of our community ✨

[gmourier]

Hello 👋

We have just released v0.29.0rc1, which is a release candidate of v0.29.0 🔥

You can test the just-introduced EXISTS and IN filtering operators!
More information in this issue and this one.

It also brings the behavior change regarding NOT / !=;

tags != adult now returns documents having tags: null or tags: [] but also documents without the tags property. It's equivalent to NOT tags = adult.

Binaries are attached to the release, or you can use the docker image:

docker run -it --rm \
    -p 7700:7700 \
    getmeili/meilisearch:v0.29.0rc1

Let us know about any bugs or feedback! 😄 It would be really helpful.

FYI, the official v0.29.0 release will be available on 3rd October.

[gmourier]

Hey everyone 👋

Locking this discussion since the IN and EXISTS operators have been released.

IN [valueA, valueB, …, valueN] selects all documents whose chosen field contains at least one of the specified values. The expression genres IN [horror, comedy] returns all documents whose genres includes either horror, comedy, or both

EXISTS checks for the existence of a field. Fields with empty or null values still count as existing. The expression release_date NOT EXISTS returns all documents without a release_date

📚 https://docs.meilisearch.com/learn/advanced/filtering_and_faceted_search.html#using-filters

Please open a new discussion if you have any feedback to share 🙇‍♂️

I've opened a specific discussion for NULL and EMPTY operators