Meilisearch
Experimental feature:CONTAINS and Prefix / Suffix filter operators
#544
Replies: 8 comments 22 replies
-
|
Wouldn't supporting regex expressions permits to do everything?
But it would also be possible to use regex syntax if needed. In what cases could you use a regex? Is it something that seems critical? |
Beta Was this translation helpful? Give feedback.
-
|
On the technical side;
But not;
|
Beta Was this translation helpful? Give feedback.
-
|
Any updates about that ? My use case is the following : I have a database of users with their past and current professional experiences, and I need to do a filter with it. Like someone woking on a company called %companyName%. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @qdequele ! Thanks for your answer. I would like to search for "truck safety" ; if we split the field by words as you suggest, in this example it will match even though it shouldn't. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Hello @nand2, I feel like your example is not really a filter. Perhaps I'm wrong. But the @Nakusu issue was about how to filter with regex when I thought your example was about searching in text. If you have to search in a text, I would just use the basic search feature of Meilisearch. If it's not enough for your use case today, could I know why? |
Beta Was this translation helpful? Give feedback.
-
|
@qdequele Thanks; sorry I should have said, I work on the same project as @Nakusu :) [
{
name: 'John',
surname: 'Doe',
general_description: 'I know stuff about truck safety etc',
experiences: [
{
company: 'Volkswagen',
description: 'I worked at the truck conception service and previously before at the car safety service.',
},
{
company: 'xxx',
description: 'xxx',
},
]
},
{
name: 'John2',
surname: 'Doe2',
general_description: 'I know stuff about truck safety etc',
experiences: [
{
company: 'Volkswagen',
description: 'I worked at the truck safety service.',
},
{
company: 'xxx',
description: 'xxx',
},
]
},
{
name: 'John3',
surname: 'Doe3',
general_description: 'I know stuff about truck safety etc',
experiences: [
{
company: 'Tesla',
description: 'I worked at the truck safety service.',
},
{
company: 'xxx',
description: 'xxx',
},
]
}
]I would like to search for persons who worked on 'truck safety' at Volkswagen only. It should match John2 Doe2 only. |
Beta Was this translation helpful? Give feedback.
-
|
i have same isuue please show me how i can find any company tesla ?? |
Beta Was this translation helpful? Give feedback.
-
|
We had an call with their devlopper team and the contains operator will be implete in the future version, it's not available at this time |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone 👋, Here is the current prototype you can use to test the new 💬 Current state of that featureThe aspect we are most concerned about with this feature is search performance. Indeed, it is possible that We would like to hear your feedback on using this prototype, and we would also like your input on the following questions to help us to decideing on a go-no-go decision for a generally available status of that feature in the future:
How to run the prototype?You need to start from a fresh new database (remove the previously used data.ms) and use the following Docker image: How to use it?You can use the feature this way or look at the original PR for more info. Send some documents to Meilisearchcat movies.json | xh 'localhost:7700/indexes/movies/documents' 'content-type:application/json'Define the attributes you want to filter onecho '{ "filterableAttributes": ["genres"] }' | xh PATCH 'localhost:7700/indexes/movies/settings'Use
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Kerollmops, A quick feedback on your amazing work. Everything looks fine on our side. I mainly tested the Some info you may find interesting (feel free to ask for more if that can help you):
The index contains 1,000,001 documents following the same structure (but with different values of course), e.g. We tried to filter on the So for this looks very good :) |
Beta Was this translation helpful? Give feedback.
-
|
Hey @rtjampens, Thank you very much for your time and the tests you made. It is very interesting. Fortunately, your dataset combined with your filter is an excellent example of good use of
Take the time you need to answer, |
Beta Was this translation helpful? Give feedback.
-
It is an auto-generated one unfortunately, so distribution of words does not totally represent a real-world scenario. I can provide the raw JSON files if you want.
On this very same dataset, it takes around 55ms (between 50ms and 63ms in my tests)
Still on this dataset, it takes around 20ms (between 15ms and 30ms, but clearly more biased towards 15ms)
Here are some tests on the different scenarios, response times keep getting consistent :
Response times are ~80ms
Response times are ~110ms
Response times are ~65ms
This query returns less time-consistent results, varying from 70ms to 93ms (biased towards 70ms though) Feel free to ask for moar tests :D Just as a side-note : while I understand the concerns on performance, I think developpers should be aware of the performance impact for using these operators. If it is clearly stated in the documentation, it's more than fine IMO |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much!
I am very interested, could you send me an email or link it here, please?
That is astonishing! I haven't done any optimization on the matter, and you have quite good processing times.
Then I'll ask you more questions 😄
Unfortunately, the performance impact can be used as a way of DDoSing servers. If we expose those operators, they can be used by anyone to DDoS a Meilisearch server, and we don't want that to be possible. We recently sped up the search algorithm to avoid that. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @rtjampens, Have you got time to read my message and measure the time Meilisearch takes for those queries? Have a nice day 🎨 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, The implementation of CONTAINS and STARTS WITH operators is perfect but we have another question. Index : When we make this research with filters about experiences, we have a weird comportement. What is the research correct filter to use in this case ? |
Beta Was this translation helpful? Give feedback.
-
|
Hey @Nakusu 👋 Thanks for your feedback about the CONTAINS and STARTS WITH operators. Indeed, it's impossible to have that behavior currently because nested objects are matched as a whole. There is no sense of sub-documents. Elasticsearch solves this with a particular "nested" object type which we don't have as Algolia and Typesense. A workaround for that may exist; Reformating the data around the experience could work 🤔 e.g., Every document is an experience, and you duplicate the contact information in every document. In that case, it will only return records (experiences) that match the filters. WDYT? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, We have a category field that has flexible and hierarcial structure. If we want to list a parent category's items, we think that CONTAINS keyword would help us. We suppose that every item will have a field like category tree for example: categoryTree: "-> Main Root Category -> Parent1 Category -> Parent2 Category -> ... -> ParentN Category -> Sub Category". If we want to list Parent2 category's items, we will do this by filtering "categoryTree CONTAINS '-> Parent2 Category ->'" command. Do you have any suggestions for solving this requirement without CONTAINS keyword? Kindly regards, |
Beta Was this translation helpful? Give feedback.
-
|
Hey @GelistirmeKF 👋 What do you mean by If you can add some frontend screens to see what you want to achieve, it could help a lot! Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @gmourier, We have a category field that has hierarcial structure. For example, a harddisk product is in a 'Storage' category; also in 'Computer Parts' , 'Computer and Tablets' and 'Electronics' categories which are parent category of 'Storage' category. That's why, in Meilisearch for this harddisk product, we kept this info in "categoryTree" field having 'Electronics -> Computer and Tablets -> Computer Parts -> Storage' data. So if we want to search "Computer Parts" products in Meilisearch, we can use "categoryTree CONTAINS '-> Computer Parts ->'" for products index. The category hierarchy for harddisk product is like this: Is there any way to search "Computer Parts" category's products without CONTAINS keyword in Meilisearch now? Thanks in advance :) |
Beta Was this translation helpful? Give feedback.
-
|
Hey @GelistirmeKF 👋 Thanks for the details. If I understood correctly, you don't have to use the I've edited a code sandbox so you can see it in action (note: I'm using instant-meilisearch, but it's possible to do it in a pure custom way if you want - instant-meilisearch is just a UI search kit made of built-in customizable components to build a search interface quickly) Code sandbox link: https://j5mc54.csb.app/ Here is how the documents are structured [
{
"id": 1,
"name": "Seagate BarraCuda 1 To (ST1000DM014)",
"hierarchicalCategories": {
"lvl0": "Electronics",
"lvl1": "Electronics > Computer and Tablets",
"lvl2": "Electronics > Computer and Tablets > Computer Parts",
"lvl3": "Electronics > Computer and Tablets > Computer Parts > Storage"
}
},
{
"id": 2,
"name": "Wacom Cintiq Pro 24",
"hierarchicalCategories": {
"lvl0": "Electronics",
"lvl1": "Electronics > Computer and Tablets",
"lvl2": "Electronics > Computer and Tablets > Tablets"
}
},
{
"id": 3,
"name": "LG 75QNED816RE",
"hierarchicalCategories": {
"lvl0": "Image & Sound",
"lvl1": "Image & Sound > Television",
"lvl2": "Image & Sound > Television > TV"
}
},
{
"id": 4,
"name": "Canon EOS 2000D + EF-S 18-55 mm IS II",
"hierarchicalCategories": {
"lvl0": "Image & Sound",
"lvl1": "Image & Sound > Camera",
"lvl2": "Image & Sound > Camera > Reflex"
}
}
]I defined <HierarchicalMenu
attributes={[
"hierarchicalCategories.lvl0",
"hierarchicalCategories.lvl1",
"hierarchicalCategories.lvl2",
"hierarchicalCategories.lvl3"
]}
/>Let me know if I misunderstood! Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Thank you so much! @gmourier This is a clean solution for us, but we found another way similar to your proposal. We assigned these category items to an attribute (categoryTree) in products index as an array by following: Product Item1: Product Item2: (Also added "categoryTree" attribute as an FilterableAttribute settings) If we list products index items by "filter": "(categoryTree = 'Computer Inner Parts')", then item1 and item2 are listed. If we list products index items by "filter": "(categoryTree = 'Sound Cards')", then only item2 is listed. Hope to help others for similar cases Appreciate for your detail explanation and example 🙏 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, we are also hoping for a progress update as we have need of the "starts with" filter. We have a CMS and have found Meilisearch to be amazing, the part that we've had to build a workaround for is being able to restrict search to a particular folder and subfolders. We allow users to embed their own search field on their pages and choose a base folder, so for the following data for instance, we'd like to be able to say "url starts with /parent-folder/" Thanks for all your hard work, Meilisearch really has been a godsend, and because performance is so great we've been able to workaround this issue by simply searching everything and filtering out unwanted results in our own code, but would be ideal to have a single search request |
Beta Was this translation helpful? Give feedback.
-
|
We are considering moving away from meilisearch because we need to "contain string operator" on the filter. Our data includes a parameter called crashCase = Some user entered string We cannot use the query because crashCase is not searchable. (Its too long) Is there a workaround for this? |
Beta Was this translation helpful? Give feedback.
-
|
Hello @2567910 what do you mean with "We cannot use the query because crashCase is not searchable. (Its too long)"? You mean the indexing is too long if you put |
Beta Was this translation helpful? Give feedback.
-
|
Hi, since there is already a prototype version for experience, I would like to know when this feature can be added to |
Beta Was this translation helpful? Give feedback.
-
|
Hi @honhimW ! We don't have an ETA for the feature yet as we have some concerns about how it would affect search performance depending on the dataset (see comment above ). Were you able to test the prototype? Any feedback you have on the prototype would help us assess the potential risks of adding it as an Experimental Feature. |
Beta Was this translation helpful? Give feedback.
-
Starting a fresh discussion for a
CONTAINSoperator.Initially expressed by
#539 (comment)
#407 (comment)
Comment made by @eelco-weblog.
For example, an e-commerce site wants to display
Apple Airpods, but notApple Airpods Max. Now, we can create an array of all the words in the title, save it as another property such as title_words and filter something like this:(title_words = airpod and title_words != max)When we can use
CONTAINS/NOT CONTAINS, we do not have to create another property and filter directly on the title:(title CONTAINS airpod and title NOT CONTAINS max)Beta Was this translation helpful? Give feedback.
All reactions