Searching doesn't perform well

[eromoe]

Describe the bug

System (please complete the following information):

• OS: windows
• Jellyfin Version: [e.g. 10.6.4

case 1

1. There is a movie named WWW-234
2. Search by WWW 234 return nothing

case 2

1. multi words search perform bad
2. There is a movie named ***** WWW 234 ****
3. search 234 WWW return nothing

case 2

1. There are many movies within char 钟
2. search single Chinese char 钟 return nothing
3. All CJK text affected

case 3

1. In CJK (Chinese, Japanese and Korean) computing, graphic characters are traditionally classed into fullwidth and halfwidth characters ( like English lower/upper case)
2. They have actually same meanings, but not be treated as same char in searching .

[DomiStyle]

The search "engine" has annoyed me since Emby times. It has no full text search features at all.

I was thinking of integrating Lucene into Jellyfin so it finally has a proper search engine, Elasticsearch is probably overkill for this use case.

Any thoughts or pointers on this by the Jellyfin maintainers?

From a quick glance at the code it seems like Jellyfin is completely lacking any indexing tasks so far?

I found the search engine here:
https://github.com/jellyfin/jellyfin/blob/master/Emby.Server.Implementations/Library/SearchEngine.cs

Search related controllers here (although the first one doesn't seem to use the engine?):
https://github.com/jellyfin/jellyfin/blob/master/Jellyfin.Api/Controllers/ItemLookupController.cs
https://github.com/jellyfin/jellyfin/blob/master/Jellyfin.Api/Controllers/SearchController.cs

The proper procedure would probably be to add a task for indexing and create a search engine implementing the ISearchEngine interface? Maybe it's also possible to reduce the API calls when searching to further speed things up, a lot of requests for a single search request.

[crobibero]

I probably wouldn't dive too deeply into the search code, our plan is to migrate entirely to EFCore. The last database is the library database, and likely any searching that is implemented will have to be redone

[DomiStyle]

@crobibero Any rough timeline on when the switch is happening?

I will take a look if a simple implementation is possible for now, assuming the change is more than a year away.

[stale]

This issue has gone 120 days without comment. To avoid abandoned issues, it will be closed in 21 days if there are no new comments.
If you're the original submitter of this issue, please comment confirming if this issue still affects you in the latest release or nightlies, or close the issue if it has been fixed. If you're another user also affected by this bug, please comment confirming so. Either action will remove the stale label.
This bot exists to prevent issues from becoming stale and forgotten. Jellyfin is always moving forward, and bugs are often fixed as side effects of other changes. We therefore ask that bug report authors remain vigilant about their issues to ensure they are closed if fixed, or re-confirmed - perhaps with fresh logs or reproduction examples - regularly. If you have any questions you can reach us on Matrix or Social Media.

[eromoe]

How is this going on ?

[Sabilon]

Hi,

This is a hack I've written to better Jellyfin's poor search capabilities while patiently waiting it migrates to EFCore :) It should fix case 2 you've described (possibly 1 too), and better Jellyfin search overall.

Jellyfin's search is restricted to the name of the medias. The hack extends it to the following fields: Overview, Genres, Studios and ProductionYear (it is easy to include more). It also allows to omit accents.

The hack is a php cli script that should be run on a regular basis, while Jellyfin is stopped. It updates all "CleanName" fields in the database, which is a hidden data used by the search process. Any new (or modified) media is affected only after the script has run. I run it every 24 hours with a cron task that stops Jellyfin, backups the database, run the hack and restarts Jellyfin.

WARNING: use at your own risk, test on a copy of library.db.

<?php

// Open database
$db = new SQLite3('./library.db');

// Function to remove accents
function jf_transliterate($string) {
    return \Transliterator::create('NFD; [:Nonspacing Mark:] Remove; NFC')->transliterate($string);
}
$db->createFunction('jf_transliterate', 'jf_transliterate');

// Fill up all CleanName fields with data from Name, Overview, Genres, Studios and ProductionYear
$db->exec("update TypedBaseItems set CleanName = lower(Name) || ' ' || jf_transliterate(lower(Name))");
foreach (array('Overview', 'Genres', 'Studios', 'ProductionYear') as $col) $db->exec("update TypedBaseItems set CleanName = CleanName || ' ' || lower($col) || ' ' || jf_transliterate(lower($col)) where $col is not null");

?>

[edrock200]

Hi,

This is a hack I've written to better Jellyfin's poor search capabilities while patiently waiting it migrates to EFCore :) It should fix case 2 you've described (possibly 1 too), and better Jellyfin search overall.

Jellyfin's search is restricted to the name of the medias. The hack extends it to the following fields: Overview, Genres, Studios and ProductionYear (it is easy to include more). It also allows to omit accents.

The hack is a php cli script that should be run on a regular basis, while Jellyfin is stopped. It updates all "CleanName" fields in the database, which is a hidden data used by the search process. Any new (or modified) media is affected only after the script has run. I run it every 24 hours with a cron task that stops Jellyfin, backups the database, run the hack and restarts Jellyfin.

WARNING: use at your own risk, test on a copy of library.db.

<?php

// Open database
$db = new SQLite3('./library.db');

// Function to remove accents
function jf_transliterate($string) {
    return \Transliterator::create('NFD; [:Nonspacing Mark:] Remove; NFC')->transliterate($string);
}
$db->createFunction('jf_transliterate', 'jf_transliterate');

// Fill up all CleanName fields with data from Name, Overview, Genres, Studios and ProductionYear
$db->exec("update TypedBaseItems set CleanName = lower(Name) || ' ' || jf_transliterate(lower(Name))");
foreach (array('Overview', 'Genres', 'Studios', 'ProductionYear') as $col) $db->exec("update TypedBaseItems set CleanName = CleanName || ' ' || lower($col) || ' ' || jf_transliterate(lower($col)) where $col is not null");

?>

Where would this be placed exactly?

[gazhay]

had to install sqllite3 and then it errored as the db was readonly.
Attempting to correct those then gave transliterate errors.

My issue is that search doesn't work at all. Search page is blank and remains blank.

[Sabilon]

@edrock200

Where would this be placed exactly?

This php file (I've named hack-jellyfin-db.php) is placed in the same folder as Jellyfin's library.db file: <my-jellyfin-base-folder>/config/data/.

Here is the shell script I use as a cron job on my server, where Jellyfin is installed as a docker container. It could be placed anywhere but it put in config/data/ to keep things together:

#!/bin/bash

#be sure to work in folder /config/data/
cd <my-jellyfin-base-folder>/config/data/

#stops Jellyfin
<my-server-path-to>/docker container stop --time=60 <my-jellyfin-container-name>

#backups DB
cp library.db "library.save.$(date +"%Y_%m_%d_%I_%M_%p").db"

#hacks DB
<my-server-path-to>/php ./hack-jellyfin-db.php

#restarts Jellyfin
<my-server-path-to>/docker container start <my-jellyfin-container-name>

And I wipe the backups out from time to time.

[Sabilon]

@gazhay

had to install sqllite3 and then it errored as the db was readonly. Attempting to correct those then gave transliterate errors.

Not sure if I understand nor if I can help here ! Try without the transliterating, maybe.