Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sort with string #1680

Closed
sanders41 opened this issue Sep 7, 2021 · 13 comments · Fixed by #1686
Closed

Sort with string #1680

sanders41 opened this issue Sep 7, 2021 · 13 comments · Fixed by #1686
Labels
bug Something isn't working as expected milli Related to the milli workspace
Milestone

Comments

@sanders41
Copy link
Contributor

Describe the bug
Using sort does not always return the results in the expected order. This is true for both asc and desc.

To Reproduce
Steps to reproduce the behavior:

  1. Create and index called movies
  2. Set the sortable attributes to ["title"]
  3. Index the small moves dataset
  4. Run the search curl -X POST 'http://localhost:7700/indexes/movies/search' --data '{"q": "world", "sort": ["title:asc"]}'. With this search the last title is Shazam!, but the results also contain The Kid Who Would Be King which should come after Shazam!.
  5. Run the search curl -H -X POST 'http://localhost:7700/indexes/movies/search' --data '{"q": "world", "sort": ["title:desc"]}'. Again the last result is Shazam!, but it should be Alita: Battle Angel.

Expected behavior
I expected the results to be in alphabetical order by title.

MeiliSearch version:
v0.22.0rc0

Additional context
MeiliSearch running in Docker.

@curquiza
Copy link
Member

curquiza commented Sep 7, 2021

Hello @sanders41!

We are aware of this problem, I should have open an issue on MeiliSearch side, my bad. See the related issue meilisearch/milli#333

We have to release Milli and integrate into this repo and this might fix this bug 🙂

@curquiza curquiza added the bug Something isn't working as expected label Sep 7, 2021
@curquiza curquiza added the milli Related to the milli workspace label Sep 7, 2021
@curquiza curquiza added this to the v0.22.0 milestone Sep 7, 2021
@curquiza curquiza changed the title v0.22.0rc0 sort not always correct Sort with string Sep 8, 2021
@curquiza curquiza linked a pull request Sep 8, 2021 that will close this issue
@curquiza curquiza mentioned this issue Sep 8, 2021
bors bot added a commit that referenced this issue Sep 8, 2021
1686: Bump milli r=curquiza a=irevoire

 fixes #1685, #1678, #1671, #1677 and #1680

Co-authored-by: Tamo <tamo@meilisearch.com>
@curquiza
Copy link
Member

curquiza commented Sep 8, 2021

Fixed by #1686

@curquiza curquiza closed this as completed Sep 8, 2021
@sanders41
Copy link
Contributor Author

With v0.22.0rc1 The issue is still there, but it is different now. Now the results are always returning in the same order and ignoring the sort. If I search for world with no sort, with the sort title:asc, or with the sort title:desc the results are all in the same order and not alphabetical. If I update the ranking rules and move sort back to the third position the original issue comes back, but it is closer to being sorted.

@Kerollmops
Copy link
Member

Kerollmops commented Sep 8, 2021

@sanders41 That's quite strange, as we directly tested the @shekhirin PR with the movies dataset on MeiliSearch and it was working fine.

  • Could you maybe provide the dataset and the settings you were using, please?
  • Does it work fine with a simple placeholder search? Documents should only be sorted by the sort rule.

@sanders41
Copy link
Contributor Author

sanders41 commented Sep 8, 2021

@Kerollmops I'm using the small movies dataset. I'll detail the full steps here, maybe I have something wrong.

  • Set the sortable attributes:
curl -H 'X-MEILI-API-KEY: masterKey' -X POST 'http://localhost:7700/indexes/movies/settings/sortable-attributes' --data '["title"]'
  • Check the settings:
$ curl -H 'X-MEILI-API-KEY: masterKey' -X GET 'http://localhost:7700/indexes/movies/settings'

{"displayedAttributes":["*"],"searchableAttributes":["*"],"filterableAttributes":[],"sortableAttributes":["title"],"rankingRules":["words","typo","proximity","attribute","sort","exactness"],"stopWords":[],"synonyms":{},"distinctAttribute":null}
  • Index the documents. Results:
   [2021-09-08T19:05:21Z INFO  meilisearch_http::index::updates] document addition done: DocumentAdditionResult { nb_documents: 30 }
  • Search with no sort:
curl -H 'X-MEILI-API-KEY: masterKey' -X POST 'http://localhost:7700/indexes/movies/search' --data '{"q": "world"}' | jq
{
  "hits": [
    {
      "id": "166428",
      "title": "How to Train Your Dragon: The Hidden World",
      "poster": "https://image.tmdb.org/t/p/w1280/xvx4Yhf0DVH8G4LzNISpMfFBDy2.jpg",
      "overview": "As Hiccup fulfills his dream of creating a peaceful dragon utopia, Toothless’ discovery of an untamed, elusive mate draws the Night Fury away. When danger mounts at home and Hiccup’s reign as village chief is tested, both dragon and rider must make impossible decisions to save their kind.",
      "release_date": 1546473600,
      "genre": "cartoon"
    },
    {
      "id": "543103",
      "title": "Kamen Rider Heisei Generations FOREVER",
      "poster": "https://image.tmdb.org/t/p/w1280/6sOFQDlkY6El1B2P5gklzJfVdsT.jpg",
      "overview": "In the world of Sougo Tokiwa and Sento Kiryu, their 'companions' are losing their memories one after the other as they're replaced by other people. The Super Time Jacker, Tid , appears before them. He orders his powerful underlings, Another Double and Another Den-O, to pursue a young boy called Shingo. While fighting to protect Shingo, Sougo meets Ataru, a young man who loves Riders, but Ataru says that Kamen Riders aren't real. What is the meaning of those words? While the mystery deepens, the true enemy that Sougo and Sento must defeat appears in the Kuriogatake mountain...",
      "release_date": 1545436800
    },
    {
      "id": "454294",
      "title": "The Kid Who Would Be King",
      "poster": "https://image.tmdb.org/t/p/w1280/kBuvLX6zynQP0sjyqbXV4jNaZ4E.jpg",
      "overview": "Old-school magic meets the modern world when young Alex stumbles upon the mythical sword Excalibur. He soon unites his friends and enemies, and they become knights who join forces with the legendary wizard Merlin. Together, they must save mankind from the wicked enchantress Morgana and her army of supernatural warriors.",
      "release_date": 1547596800
    },
    {
      "id": "299536",
      "title": "Avengers: Infinity War",
      "poster": "https://image.tmdb.org/t/p/w1280/7WsyChQLEftFiDOVTGkv3hFpyyt.jpg",
      "overview": "As the Avengers and their allies have continued to protect the world from threats too large for any one hero to handle, a new danger has emerged from the cosmic shadows: Thanos. A despot of intergalactic infamy, his goal is to collect all six Infinity Stones, artifacts of unimaginable power, and use them to inflict his twisted will on all of reality. Everything the Avengers have fought for has led up to this moment - the fate of Earth and existence itself has never been more uncertain.",
      "release_date": 1524618000,
      "genre": "action"
    },
    {
      "id": "399579",
      "title": "Alita: Battle Angel",
      "poster": "https://image.tmdb.org/t/p/w1280/xRWht48C2V8XNfzvPehyClOvDni.jpg",
      "overview": "When Alita awakens with no memory of who she is in a future world she does not recognize, she is taken in by Ido, a compassionate doctor who realizes that somewhere in this abandoned cyborg shell is the heart and soul of a young woman with an extraordinary past.",
      "release_date": 1548892800
    },
    {
      "id": "404368",
      "title": "Ralph Breaks the Internet",
      "poster": "https://image.tmdb.org/t/p/w1280/lvfIaThG5HA8THf76nghKinjjji.jpg",
      "overview": "Video game bad guy Ralph and fellow misfit Vanellope von Schweetz must risk it all by traveling to the World Wide Web in search of a replacement part to save Vanellope's video game, 'Sugar Rush.' In way over their heads, Ralph and Vanellope rely on the citizens of the internet -- the netizens -- to help navigate their way, including an entrepreneur named Yesss, who is the head algorithm and the heart and soul of trend-making site BuzzzTube.",
      "release_date": 1542672000
    },
    {
      "id": "456740",
      "title": "Hellboy",
      "poster": "https://image.tmdb.org/t/p/w1280/nUXCJMnAiwCpNPZuJH2n6h5hGtF.jpg",
      "overview": "Hellboy comes to England, where he must defeat Nimue, Merlin's consort and the Blood Queen. But their battle will bring about the end of the world, a fate he desperately tries to turn away.",
      "release_date": 1554944400
    },
    {
      "id": "424783",
      "title": "Bumblebee",
      "poster": "https://image.tmdb.org/t/p/w1280/fw02ONlDhrYjTSZV8XO6hhU3ds3.jpg",
      "overview": "On the run in the year 1987, Bumblebee finds refuge in a junkyard in a small Californian beach town. Charlie, on the cusp of turning 18 and trying to find her place in the world, discovers Bumblebee, battle-scarred and broken.  When Charlie revives him, she quickly learns this is no ordinary yellow VW bug.",
      "release_date": 1544832000
    },
    {
      "id": "297802",
      "title": "Aquaman",
      "poster": "https://image.tmdb.org/t/p/w1280/5Kg76ldv7VxeX9YlcQXiowHgdX6.jpg",
      "overview": "Once home to the most advanced civilization on Earth, Atlantis is now an underwater kingdom ruled by the power-hungry King Orm. With a vast army at his disposal, Orm plans to conquer the remaining oceanic people and then the surface world. Standing in his way is Arthur Curry, Orm's half-human, half-Atlantean brother and true heir to the throne.",
      "release_date": 1544140800,
      "genre": "action"
    },
    {
      "id": "485811",
      "title": "Redcon-1",
      "poster": "https://image.tmdb.org/t/p/w1280/vVPrWngVJ2cfYAncBedQty69Dlf.jpg",
      "overview": "After a zombie apocalypse spreads from a London prison, the UK is brought to its knees. The spread of the virus is temporarily contained but, without a cure, it’s only a matter of time before it breaks its boundaries and the biggest problem of all… any zombies with combat skills are now enhanced. With the South East of England quarantined from the rest of the world using fortified borders, intelligence finds that the scientist responsible for the outbreak is alive and well in London. With his recovery being the only hope of a cure, a squad of eight Special Forces soldiers is sent on a suicide mission to the city, now ruled by the undead, with a single task: get him out alive within 72 hours by any means necessary. What emerges is an unlikely pairing on a course to save humanity against ever-rising odds.",
      "release_date": 1538096400
    },
    {
      "id": "338952",
      "title": "Fantastic Beasts: The Crimes of Grindelwald",
      "poster": "https://image.tmdb.org/t/p/w1280/fMMrl8fD9gRCFJvsx0SuFwkEOop.jpg",
      "overview": "Gellert Grindelwald has escaped imprisonment and has begun gathering followers to his cause—elevating wizards above all non-magical beings. The only one capable of putting a stop to him is the wizard he once called his closest friend, Albus Dumbledore. However, Dumbledore will need to seek help from the wizard who had thwarted Grindelwald once before, his former student Newt Scamander, who agrees to help, unaware of the dangers that lie ahead. Lines are drawn as love and loyalty are tested, even among the truest friends and family, in an increasingly divided wizarding world.",
      "release_date": 1542153600,
      "genre": "fantasy"
    },
    {
      "id": "287947",
      "title": "Shazam!",
      "poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg",
      "overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.",
      "release_date": 1553299200,
      "genre": "action"
    }
  ],
  "nbHits": 12,
  "exhaustiveNbHits": false,
  "query": "world",
  "limit": 20,
  "offset": 0,
  "processingTimeMs": 2
}
  • Search with asc sort:
curl -H 'X-MEILI-API-KEY: masterKey' -X POST 'http://localhost:7700/indexes/movies/search' --data '{"q": "world", "sort": ["title:asc"]}' | jq
{
  "hits": [
    {
      "id": "166428",
      "title": "How to Train Your Dragon: The Hidden World",
      "poster": "https://image.tmdb.org/t/p/w1280/xvx4Yhf0DVH8G4LzNISpMfFBDy2.jpg",
      "overview": "As Hiccup fulfills his dream of creating a peaceful dragon utopia, Toothless’ discovery of an untamed, elusive mate draws the Night Fury away. When danger mounts at home and Hiccup’s reign as village chief is tested, both dragon and rider must make impossible decisions to save their kind.",
      "release_date": 1546473600,
      "genre": "cartoon"
    },
    {
      "id": "543103",
      "title": "Kamen Rider Heisei Generations FOREVER",
      "poster": "https://image.tmdb.org/t/p/w1280/6sOFQDlkY6El1B2P5gklzJfVdsT.jpg",
      "overview": "In the world of Sougo Tokiwa and Sento Kiryu, their 'companions' are losing their memories one after the other as they're replaced by other people. The Super Time Jacker, Tid , appears before them. He orders his powerful underlings, Another Double and Another Den-O, to pursue a young boy called Shingo. While fighting to protect Shingo, Sougo meets Ataru, a young man who loves Riders, but Ataru says that Kamen Riders aren't real. What is the meaning of those words? While the mystery deepens, the true enemy that Sougo and Sento must defeat appears in the Kuriogatake mountain...",
      "release_date": 1545436800
    },
    {
      "id": "454294",
      "title": "The Kid Who Would Be King",
      "poster": "https://image.tmdb.org/t/p/w1280/kBuvLX6zynQP0sjyqbXV4jNaZ4E.jpg",
      "overview": "Old-school magic meets the modern world when young Alex stumbles upon the mythical sword Excalibur. He soon unites his friends and enemies, and they become knights who join forces with the legendary wizard Merlin. Together, they must save mankind from the wicked enchantress Morgana and her army of supernatural warriors.",
      "release_date": 1547596800
    },
    {
      "id": "299536",
      "title": "Avengers: Infinity War",
      "poster": "https://image.tmdb.org/t/p/w1280/7WsyChQLEftFiDOVTGkv3hFpyyt.jpg",
      "overview": "As the Avengers and their allies have continued to protect the world from threats too large for any one hero to handle, a new danger has emerged from the cosmic shadows: Thanos. A despot of intergalactic infamy, his goal is to collect all six Infinity Stones, artifacts of unimaginable power, and use them to inflict his twisted will on all of reality. Everything the Avengers have fought for has led up to this moment - the fate of Earth and existence itself has never been more uncertain.",
      "release_date": 1524618000,
      "genre": "action"
    },
    {
      "id": "399579",
      "title": "Alita: Battle Angel",
      "poster": "https://image.tmdb.org/t/p/w1280/xRWht48C2V8XNfzvPehyClOvDni.jpg",
      "overview": "When Alita awakens with no memory of who she is in a future world she does not recognize, she is taken in by Ido, a compassionate doctor who realizes that somewhere in this abandoned cyborg shell is the heart and soul of a young woman with an extraordinary past.",
      "release_date": 1548892800
    },
    {
      "id": "404368",
      "title": "Ralph Breaks the Internet",
      "poster": "https://image.tmdb.org/t/p/w1280/lvfIaThG5HA8THf76nghKinjjji.jpg",
      "overview": "Video game bad guy Ralph and fellow misfit Vanellope von Schweetz must risk it all by traveling to the World Wide Web in search of a replacement part to save Vanellope's video game, 'Sugar Rush.' In way over their heads, Ralph and Vanellope rely on the citizens of the internet -- the netizens -- to help navigate their way, including an entrepreneur named Yesss, who is the head algorithm and the heart and soul of trend-making site BuzzzTube.",
      "release_date": 1542672000
    },
    {
      "id": "456740",
      "title": "Hellboy",
      "poster": "https://image.tmdb.org/t/p/w1280/nUXCJMnAiwCpNPZuJH2n6h5hGtF.jpg",
      "overview": "Hellboy comes to England, where he must defeat Nimue, Merlin's consort and the Blood Queen. But their battle will bring about the end of the world, a fate he desperately tries to turn away.",
      "release_date": 1554944400
    },
    {
      "id": "424783",
      "title": "Bumblebee",
      "poster": "https://image.tmdb.org/t/p/w1280/fw02ONlDhrYjTSZV8XO6hhU3ds3.jpg",
      "overview": "On the run in the year 1987, Bumblebee finds refuge in a junkyard in a small Californian beach town. Charlie, on the cusp of turning 18 and trying to find her place in the world, discovers Bumblebee, battle-scarred and broken.  When Charlie revives him, she quickly learns this is no ordinary yellow VW bug.",
      "release_date": 1544832000
    },
    {
      "id": "297802",
      "title": "Aquaman",
      "poster": "https://image.tmdb.org/t/p/w1280/5Kg76ldv7VxeX9YlcQXiowHgdX6.jpg",
      "overview": "Once home to the most advanced civilization on Earth, Atlantis is now an underwater kingdom ruled by the power-hungry King Orm. With a vast army at his disposal, Orm plans to conquer the remaining oceanic people and then the surface world. Standing in his way is Arthur Curry, Orm's half-human, half-Atlantean brother and true heir to the throne.",
      "release_date": 1544140800,
      "genre": "action"
    },
    {
      "id": "485811",
      "title": "Redcon-1",
      "poster": "https://image.tmdb.org/t/p/w1280/vVPrWngVJ2cfYAncBedQty69Dlf.jpg",
      "overview": "After a zombie apocalypse spreads from a London prison, the UK is brought to its knees. The spread of the virus is temporarily contained but, without a cure, it’s only a matter of time before it breaks its boundaries and the biggest problem of all… any zombies with combat skills are now enhanced. With the South East of England quarantined from the rest of the world using fortified borders, intelligence finds that the scientist responsible for the outbreak is alive and well in London. With his recovery being the only hope of a cure, a squad of eight Special Forces soldiers is sent on a suicide mission to the city, now ruled by the undead, with a single task: get him out alive within 72 hours by any means necessary. What emerges is an unlikely pairing on a course to save humanity against ever-rising odds.",
      "release_date": 1538096400
    },
    {
      "id": "338952",
      "title": "Fantastic Beasts: The Crimes of Grindelwald",
      "poster": "https://image.tmdb.org/t/p/w1280/fMMrl8fD9gRCFJvsx0SuFwkEOop.jpg",
      "overview": "Gellert Grindelwald has escaped imprisonment and has begun gathering followers to his cause—elevating wizards above all non-magical beings. The only one capable of putting a stop to him is the wizard he once called his closest friend, Albus Dumbledore. However, Dumbledore will need to seek help from the wizard who had thwarted Grindelwald once before, his former student Newt Scamander, who agrees to help, unaware of the dangers that lie ahead. Lines are drawn as love and loyalty are tested, even among the truest friends and family, in an increasingly divided wizarding world.",
      "release_date": 1542153600,
      "genre": "fantasy"
    },
    {
      "id": "287947",
      "title": "Shazam!",
      "poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg",
      "overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.",
      "release_date": 1553299200,
      "genre": "action"
    }
  ],
  "nbHits": 12,
  "exhaustiveNbHits": false,
  "query": "world",
  "limit": 20,
  "offset": 0,
  "processingTimeMs": 2
}

@Kerollmops
Copy link
Member

Kerollmops commented Sep 8, 2021

So at first glance, it seems that it is valid. In fact, we can't be sure that the sort ranking rule is invalid here as the sort of the documents is completely ruled out by the attribute criterion which sorts documents by the attribute and position a query word is found in it.

So, according to the bucket sort system, each group generated by a rule that can't be tie broke is given to the rule after. In this case, the attribute ranking rule is generating groups of one document, so they can't be sort by the sort ranking rule.

Maybe I'm wrong, that's just a first glance, if you try with a placeholder search, for example, it works fine.

@sanders41
Copy link
Contributor Author

I'm not sure if this is helpful, but if you keep everything else the same but update the ranking rules first:

curl -H 'X-MEILI-API-KEY: masterKey' -X POST 'http://localhost:7700/indexes/movies/settings/ranking-rules' --data '["words", "typo", "sort", "proximity", "attribute", "exactness"]'

Then the results of the title:asc search becomes the following (which is what I got in v0.22.0RC0)

{
  "hits": [
    {
      "id": "399579",
      "title": "Alita: Battle Angel",
      "poster": "https://image.tmdb.org/t/p/w1280/xRWht48C2V8XNfzvPehyClOvDni.jpg",
      "overview": "When Alita awakens with no memory of who she is in a future world she does not recognize, she is taken in by Ido, a compassionate doctor who realizes that somewhere in this abandoned cyborg shell is the heart and soul of a young woman with an extraordinary past.",
      "release_date": 1548892800
    },
    {
      "id": "297802",
      "title": "Aquaman",
      "poster": "https://image.tmdb.org/t/p/w1280/5Kg76ldv7VxeX9YlcQXiowHgdX6.jpg",
      "overview": "Once home to the most advanced civilization on Earth, Atlantis is now an underwater kingdom ruled by the power-hungry King Orm. With a vast army at his disposal, Orm plans to conquer the remaining oceanic people and then the surface world. Standing in his way is Arthur Curry, Orm's half-human, half-Atlantean brother and true heir to the throne.",
      "release_date": 1544140800,
      "genre": "action"
    },
    {
      "id": "299536",
      "title": "Avengers: Infinity War",
      "poster": "https://image.tmdb.org/t/p/w1280/7WsyChQLEftFiDOVTGkv3hFpyyt.jpg",
      "overview": "As the Avengers and their allies have continued to protect the world from threats too large for any one hero to handle, a new danger has emerged from the cosmic shadows: Thanos. A despot of intergalactic infamy, his goal is to collect all six Infinity Stones, artifacts of unimaginable power, and use them to inflict his twisted will on all of reality. Everything the Avengers have fought for has led up to this moment - the fate of Earth and existence itself has never been more uncertain.",
      "release_date": 1524618000,
      "genre": "action"
    },
    {
      "id": "424783",
      "title": "Bumblebee",
      "poster": "https://image.tmdb.org/t/p/w1280/fw02ONlDhrYjTSZV8XO6hhU3ds3.jpg",
      "overview": "On the run in the year 1987, Bumblebee finds refuge in a junkyard in a small Californian beach town. Charlie, on the cusp of turning 18 and trying to find her place in the world, discovers Bumblebee, battle-scarred and broken.  When Charlie revives him, she quickly learns this is no ordinary yellow VW bug.",
      "release_date": 1544832000
    },
    {
      "id": "338952",
      "title": "Fantastic Beasts: The Crimes of Grindelwald",
      "poster": "https://image.tmdb.org/t/p/w1280/fMMrl8fD9gRCFJvsx0SuFwkEOop.jpg",
      "overview": "Gellert Grindelwald has escaped imprisonment and has begun gathering followers to his cause—elevating wizards above all non-magical beings. The only one capable of putting a stop to him is the wizard he once called his closest friend, Albus Dumbledore. However, Dumbledore will need to seek help from the wizard who had thwarted Grindelwald once before, his former student Newt Scamander, who agrees to help, unaware of the dangers that lie ahead. Lines are drawn as love and loyalty are tested, even among the truest friends and family, in an increasingly divided wizarding world.",
      "release_date": 1542153600,
      "genre": "fantasy"
    },
    {
      "id": "456740",
      "title": "Hellboy",
      "poster": "https://image.tmdb.org/t/p/w1280/nUXCJMnAiwCpNPZuJH2n6h5hGtF.jpg",
      "overview": "Hellboy comes to England, where he must defeat Nimue, Merlin's consort and the Blood Queen. But their battle will bring about the end of the world, a fate he desperately tries to turn away.",
      "release_date": 1554944400
    },
    {
      "id": "166428",
      "title": "How to Train Your Dragon: The Hidden World",
      "poster": "https://image.tmdb.org/t/p/w1280/xvx4Yhf0DVH8G4LzNISpMfFBDy2.jpg",
      "overview": "As Hiccup fulfills his dream of creating a peaceful dragon utopia, Toothless’ discovery of an untamed, elusive mate draws the Night Fury away. When danger mounts at home and Hiccup’s reign as village chief is tested, both dragon and rider must make impossible decisions to save their kind.",
      "release_date": 1546473600,
      "genre": "cartoon"
    },
    {
      "id": "543103",
      "title": "Kamen Rider Heisei Generations FOREVER",
      "poster": "https://image.tmdb.org/t/p/w1280/6sOFQDlkY6El1B2P5gklzJfVdsT.jpg",
      "overview": "In the world of Sougo Tokiwa and Sento Kiryu, their 'companions' are losing their memories one after the other as they're replaced by other people. The Super Time Jacker, Tid , appears before them. He orders his powerful underlings, Another Double and Another Den-O, to pursue a young boy called Shingo. While fighting to protect Shingo, Sougo meets Ataru, a young man who loves Riders, but Ataru says that Kamen Riders aren't real. What is the meaning of those words? While the mystery deepens, the true enemy that Sougo and Sento must defeat appears in the Kuriogatake mountain...",
      "release_date": 1545436800
    },
    {
      "id": "404368",
      "title": "Ralph Breaks the Internet",
      "poster": "https://image.tmdb.org/t/p/w1280/lvfIaThG5HA8THf76nghKinjjji.jpg",
      "overview": "Video game bad guy Ralph and fellow misfit Vanellope von Schweetz must risk it all by traveling to the World Wide Web in search of a replacement part to save Vanellope's video game, 'Sugar Rush.' In way over their heads, Ralph and Vanellope rely on the citizens of the internet -- the netizens -- to help navigate their way, including an entrepreneur named Yesss, who is the head algorithm and the heart and soul of trend-making site BuzzzTube.",
      "release_date": 1542672000
    },
    {
      "id": "485811",
      "title": "Redcon-1",
      "poster": "https://image.tmdb.org/t/p/w1280/vVPrWngVJ2cfYAncBedQty69Dlf.jpg",
      "overview": "After a zombie apocalypse spreads from a London prison, the UK is brought to its knees. The spread of the virus is temporarily contained but, without a cure, it’s only a matter of time before it breaks its boundaries and the biggest problem of all… any zombies with combat skills are now enhanced. With the South East of England quarantined from the rest of the world using fortified borders, intelligence finds that the scientist responsible for the outbreak is alive and well in London. With his recovery being the only hope of a cure, a squad of eight Special Forces soldiers is sent on a suicide mission to the city, now ruled by the undead, with a single task: get him out alive within 72 hours by any means necessary. What emerges is an unlikely pairing on a course to save humanity against ever-rising odds.",
      "release_date": 1538096400
    },
    {
      "id": "454294",
      "title": "The Kid Who Would Be King",
      "poster": "https://image.tmdb.org/t/p/w1280/kBuvLX6zynQP0sjyqbXV4jNaZ4E.jpg",
      "overview": "Old-school magic meets the modern world when young Alex stumbles upon the mythical sword Excalibur. He soon unites his friends and enemies, and they become knights who join forces with the legendary wizard Merlin. Together, they must save mankind from the wicked enchantress Morgana and her army of supernatural warriors.",
      "release_date": 1547596800
    },
    {
      "id": "287947",
      "title": "Shazam!",
      "poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg",
      "overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.",
      "release_date": 1553299200,
      "genre": "action"
    }
  ],
  "nbHits": 12,
  "exhaustiveNbHits": false,
  "query": "world",
  "limit": 20,
  "offset": 0,
  "processingTimeMs": 3
}

@Kerollmops
Copy link
Member

Yup, if I remember correctly the _:desc was the thing broken and fixed in v0.22.0rc1. And yes the example you show is sort by title:asc and that's very visible as the words and typo ranking rules aren't important on a query like world, the words put all the documents in the same group and the typo working rule creates two big groups and the first group is given to the sort ranking rule which sorts the documents by title and that's quite effective!

@sanders41
Copy link
Contributor Author

Change the ranking rules makes it almost correct. Shazam! should come before The Kid Who Would Be King. As far as I have been able to tell the order seems to get off when the value contains some kind of punctuation, ie !.

@Kerollmops
Copy link
Member

Nope, not sure about that, Shazam! does not contains world but word and therefore is after all the other documents. This specific document is in the second group that the typo ranking rule generates.

@sanders41
Copy link
Contributor Author

OK, I understand now...but it makes sorting very confusing because it means the sort will rarely be in the order the end user expects :)

@Kerollmops
Copy link
Member

Kerollmops commented Sep 8, 2021

Yup, that's a correct point but that's also a complex one. We will work on extensively documenting this feature.

Imagine that you move the sort ranking rule at the top, you will get the document exactly in the title:ascorder but the relevancy will be impacted a lot (i.e. Shazam! that do not have world). On the other end, if you put it at the bottom or near the bottom, you will probably don't get the documents ordered enough. So we decided to put it in the middle, we will do more testing on the position we think should fit a maximum of use cases.

@gmourier
Copy link
Member

gmourier commented Sep 9, 2021

Hello @sanders41, thank you for testing it and for your feedback!

Some information about sorting. 🤓

Sort on string value is working in lexicographical order and not alphabetical. À will be positioned before Z in descending order because of the Unicode point representation.

We chose to place the ranking rule in the 5th position by default (after the attribute ranking rule) to favor results relevant to the user's search; It depends on the use-case and the dataset. It is more visually obvious in the following example.

Video.MP4.1064x946.mp4

For this example, we use 2 different indexes with different ranking rule positions to show differences. Most Recommended uses a sort ranking rule positioned in the third position. Relevant Most Recommended uses a sort ranking rule positioned in 5th position (the default one). It seems that the sort ranking rule positioned in the 5th position is a lot more accurate to favor what the users can expect in this use-case.

We take a stand to provide a position that favors relevancy above all by default. Fortunately, the position is configurable to fit each use case. What could be missed here is to favor a default position that does not fit the user's use-case, thus forcing him to make an effort to configure it.

If you want a sort that works exhaustively and therefore without relevancy, similar to an SQL system, you have to place it in the first position, and it will have very little interest for the end user except in some use-cases that we think are quite rare.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working as expected milli Related to the milli workspace
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants