core/database: configure fts3 tokenizer support

[Fat-Zer]

Original patch by Arfrever
This fixes #5297

[gasinvein]

Why it isn't merged?

[camgaertner]

I had the same issue as #5862 and this fixed it. Is there a reason it isn't merged?

[aairey]

@jonaski can we merge this?
People who don't implement this fix cannot even start clementine.

[hatstand]

Search performance without fts3 is tragic. Presumably, fedora is explicitly not using our built-in sqlite which is there for exactly this reason so ¯_(ツ)_/¯

[aairey]

So why is the package even in the repositories? Can you talk to the Fedora maintainer?

…

On Fri, 16 Mar 2018, 15:03 John Maguire, ***@***.***> wrote: Search performance without fts3 is tragic. Presumably, fedora is explicitly not using our built-in sqlite which is there for exactly this reason so ¯_(ツ)_/¯ — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5669 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHSjviqDHB3JX3NH75L9-hGHiyOk3SMzks5te8ZGgaJpZM4MfNrE> .

[hatstand]

We don't maintain it. If you want it fixed I suggest you talk to fedora yourself.

[aairey]

Well, you are the one believing that your app is packaged wrongly by the Fedora package maintainers, so I suggest you open a Bug Report as you have the technical details. I don't even know what FTS3 is.

…

On Fri, 16 Mar 2018, 17:43 John Maguire, ***@***.***> wrote: We don't maintain it. If you want it fixed I suggest you talk to fedora yourself. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5669 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHSjvslb4TLmKAdykVdcSdLaQIyEBI7Aks5te-ulgaJpZM4MfNrE> .

[jonaski]

@hatstand sqlite isn't built-in, it's only the QSqlDriver (3rdparty/qsqlite). If the distro has sqlite version 3.11 or newer it will run into this problem, built-in qsqlite or not. It's not the package maintainers fault, they are doing it correct. All this does is too explicitly enable fts3 when not already enabled.
https://www.sqlite.org/compile.html#enable_fts3

OpenSuse does the same thing because tumbleweed provides sqlite 3.23
https://build.opensuse.org/package/view_file/openSUSE:Factory/clementine/clementine-fix-sqlite.patch?expand=1

[phorgan1]

Thanks. I'm on fedora 28 and it still isn't applied, but after I applied it FINALLY I can use clementine again. So happy.

[aairey]

Seems fixed in Fedora 28 now?

[a17r]

Needless to say that Gentoo needs to apply this patch for a long time already.

[renn0xtek9]

I am not sure I understand it, I have build sqlite3 successfully (latest source) with --enable-fts3 option in configure script. I still get
unknown tokenizer: unicode
and this with a clementine compile from latest git.
jonaski can you precise a bit?
Best regard

[jonaski]

--enable-fts3 is not a valid option (at least not in newer versions). You need to pass: -DSQLITE_ENABLE_FTS3_TOKENIZER=1 to CFLAGS
Like this:
./configure --disable-static --enable-fts5 CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3_TOKENIZER=1"

However if your distro was shipped with a different, most likely an older version of sqlite, other libraries that depend on the sqlite library (like qt) will probably break since they were compiled against an older version of sqlite, or with different configuration options so they expose different symbols. And clementine won't work anyway in that case. So it's a good idea to use the same version of sqlite that came with the system and make sure you configure it the same way and only add DSQLITE_ENABLE_FTS3_TOKENIZER=1 to CFLAGS

Clementine remains broken unless you use a distro that ship older packages, or compile your own sqlite that actually have matching symbols.
So a better way is to patch clementine:
You can use these patches:
master: http://files.jkvinge.net/packages/clementine/clementine-fix-sqlite.patch
qt5: http://files.jkvinge.net/packages/clementine/clementine-qt5-fix-sqlite.patch

[renn0xtek9]

I tried the patch alone, I still had tokenizer error, then I compiled and install sqlite as you said, compiled clementine (with the patch), remove the database in
$HOME/.config/Clementine/clementine.db
but upon launching clementine I get this

23:39:05.775 INFO  main:326                         Clementine 1.3.1-468-g292fd0b 
23:39:05.955 INFO  Database:293                     Creating initial database schema 
23:39:05.955 DEBUG Database:455                     Applying database schema update 0 from ":/schema/schema.sql" 
23:39:05.970 DEBUG Database:455                     Applying database schema update 1 from ":/schema/schema-1.sql" 
23:39:05.972 DEBUG Database:455                     Applying database schema update 2 from ":/schema/schema-2.sql" 
23:39:05.981 DEBUG Database:455                     Applying database schema update 3 from ":/schema/schema-3.sql" 
23:39:05.989 DEBUG Database:455                     Applying database schema update 4 from ":/schema/schema-4.sql" 
23:39:05.991 DEBUG Database:455                     Applying database schema update 5 from ":/schema/schema-5.sql" 
23:39:05.994 DEBUG Database:455                     Applying database schema update 6 from ":/schema/schema-6.sql" 
23:39:05.996 DEBUG Database:455                     Applying database schema update 7 from ":/schema/schema-7.sql" 
23:39:05.998 DEBUG Database:455                     Applying database schema update 8 from ":/schema/schema-8.sql" 
23:39:06.002 DEBUG Database:455                     Applying database schema update 9 from ":/schema/schema-9.sql" 
23:39:06.005 DEBUG Database:455                     Applying database schema update 10 from ":/schema/schema-10.sql" 
23:39:06.007 DEBUG Database:455                     Applying database schema update 11 from ":/schema/schema-11.sql" 
23:39:06.008 DEBUG Database:455                     Applying database schema update 12 from ":/schema/schema-12.sql" 
23:39:06.010 DEBUG Database:455                     Applying database schema update 13 from ":/schema/schema-13.sql" 
23:39:06.019 DEBUG Database:455                     Applying database schema update 14 from ":/schema/schema-14.sql" 
23:39:06.022 DEBUG Database:455                     Applying database schema update 15 from ":/schema/schema-15.sql" 
23:39:06.024 DEBUG Database:455                     Applying database schema update 16 from ":/schema/schema-16.sql" 
23:39:06.026 DEBUG Database:455                     Applying database schema update 17 from ":/schema/schema-17.sql" 
23:39:06.028 DEBUG Database:455                     Applying database schema update 18 from ":/schema/schema-18.sql" 
23:39:06.028 INFO  Database:535                     Updating "songs" for %allsongstables 
23:39:06.029 INFO  Database:535                     Updating "magnatune_songs" for %allsongstables 
23:39:06.029 INFO  Database:535                     Updating "jamendo.songs" for %allsongstables 
23:39:06.029 ERROR Database:584                     db error:  QSqlError(1, "Unable to execute statement", "duplicate column name: skipcount") 
23:39:06.029 ERROR Database:585                     faulty query:  "ALTER TABLE jamendo.songs ADD COLUMN skipcount INTEGER NOT NULL DEFAULT 0" 
23:39:06.029 ERROR Database:586                     bound values:  QMap() 
23:39:06.029 ERROR unknown                          Unable to update music library database 

Aborted

[renn0xtek9]

shymega not sure if that is related, but today I pulled the master git, compile, installed and launched and I got this
17:01:33.419 INFO main:326 Clementine 1.3.1-472-g4ac2ded
17:01:33.588 WARN Database:285 Couldn't register FTS3 tokenizer
17:01:33.588 INFO Database:293 Creating initial database schema
17:01:33.589 DEBUG Database:455 Applying database schema update 0 from ":/schema/schema.sql"
17:01:33.591 DEBUG Database:455 Applying database schema update 1 from ":/schema/schema-1.sql"
17:01:33.593 DEBUG Database:455 Applying database schema update 2 from ":/schema/schema-2.sql"
17:01:33.595 DEBUG Database:455 Applying database schema update 3 from ":/schema/schema-3.sql"
17:01:33.597 DEBUG Database:455 Applying database schema update 4 from ":/schema/schema-4.sql"
17:01:33.599 DEBUG Database:455 Applying database schema update 5 from ":/schema/schema-5.sql"
17:01:33.601 DEBUG Database:455 Applying database schema update 6 from ":/schema/schema-6.sql"
17:01:33.604 DEBUG Database:455 Applying database schema update 7 from ":/schema/schema-7.sql"
17:01:33.606 DEBUG Database:455 Applying database schema update 8 from ":/schema/schema-8.sql"
17:01:33.609 DEBUG Database:455 Applying database schema update 9 from ":/schema/schema-9.sql"
17:01:33.611 DEBUG Database:455 Applying database schema update 10 from ":/schema/schema-10.sql"
17:01:33.613 DEBUG Database:455 Applying database schema update 11 from ":/schema/schema-11.sql"
17:01:33.615 DEBUG Database:455 Applying database schema update 12 from ":/schema/schema-12.sql"
17:01:33.617 DEBUG Database:455 Applying database schema update 13 from ":/schema/schema-13.sql"
17:01:33.617 ERROR Database:584 db error: QSqlError(1, "Unable to fetch row", "no such module: fts3")
17:01:33.617 ERROR Database:585 faulty query: "CREATE VIRTUAL TABLE songs_fts USING fts3(
ftstitle, ftsalbum, ftsartist, ftsalbumartist, ftscomposer, ftsgenre, ftscomment,
tokenize=unicode
)"
17:01:33.618 ERROR Database:586 bound values: QMap()
17:01:33.618 ERROR unknown Unable to update music library database
Aborted `

[renn0xtek9]

Can someone provide instructions instead of inaccurate comment on how to build this and fix this (after more than a year).
FYI on a Ubunut fresh install 18.04 if you do that, https://pastebin.com/t44iB3MM you will go into the same problem again .....

[renn0xtek9]

I just tried to compile the latest master git and the problem persist...

…

----- Mail original ----- De: "John Maguire" <notifications@github.com> À: "clementine-player/Clementine" <Clementine@noreply.github.com> Cc: "renn0xtek9" <renn0xtek9@laposte.net>, "Comment" <comment@noreply.github.com> Envoyé: Dimanche 30 Septembre 2018 00:31:42 Objet: Re: [clementine-player/Clementine] core/database: configure fts3 tokenizer support (#5669) Merged #5669 into master. — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread .