[4.2] Smart Search: Prevent "#__finder_tokens full" error

[Hackwar]

Summary of Changes

Smart Search has a long standing issue that the #__finder_tokens table can run full and abort the indexing process. That table is of type "MEMORY", which can run full and there is an option to reduce the number of tokens to store in that table, which unfortunately doesn't work in all cases. The MEMORY table is used since it is a lot faster than a normal InnoDB table and since this is just used for temporary storage, writing speed is the main factor here.
This change pushes the code of addTokenToDb to the tokenizeToDbShort() method and better counts the number of tokens written to DB. It also removes the option from the Smart Search configuration and instead reads from the database what it is capable of. This most likely isn't a 100% solution, since there are edge cases where this could still fail, but it should improve the situation greatly.
This removes the addTokenToDb method, but since this is a protected method in a class of a component, this would still be in accordance to our backwards compatibility promise. Last but not least, this is remade from the PR #35995.

Testing Instructions

1. Get a very long text and insert it into an article.
2. When saving, see it fail with the error message from the title.
3. Apply this PR.
4. Save again and see it pass this time.

[richard67]

@Hackwar The title says "[4.1]" but the target branch is 4.0-dev. What's right now?

[Hackwar]

Damn, I was hoping nobody would notice. I accidentally based this PR on 4.0-dev instead of 4.1-dev and since the latest changes from 4.0-dev haven't been merged upwards, I tried to get around the bigger diff for now by selecting the 4.0-dev branch. I'll select the 4.1-dev branch.

[richard67]

"I was hoping nobody would notice" does not work with me 😛

[Fedik]

It partially works :)

It works for 100 paragraphs https://loremipsum.io/generator/?n=100&t=p
But fail for 500 paragraphs https://loremipsum.io/generator/?n=500&t=p
(Warning: do not try more, or your browser will explode 😄 )

Not sure if it success ;)

[Hackwar]

Lets call it an improvement, which is why I would still consider this a success.

[Fedik]

I have tested this item ✅ successfully on fb77012

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36749.}

[richard67]

Shouldn't the parameter be removed from the params column in the extensions table in database for new installations here
https://github.com/joomla/joomla-cms/blob/4.1-dev/installation/sql/mysql/base.sql#L173
and here
https://github.com/joomla/joomla-cms/blob/4.1-dev/installation/sql/postgresql/base.sql#L179
?

@brianteeman What do you think? Should we do that here or leave it to a general cleanup for your issue #37386 ?

[richard67]

P.S. to my previous comment: @Hackwar If you want to fix the SQL with this PR, too, the best is that you just save the parameters after having made a new, clean installation with this PR applied, and then copy the value from database into the insert statement in base.sql.

[superknutsel]

I have tested this item 🔴 unsuccessfully on fb77012

Saving an article with 23000 words gives me the message that the table is full. But after aplying the patch it still gives me the message that the table is full.

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36749.}

[brianteeman]

@richard67 always do it when you see it. Another pr may not happen

[HLeithner]

This pull request has automatically rebased to 4.2-dev.

[joomla-bot]

This pull requests has been automatically converted to the PSR-12 coding standard.

[Hackwar]

Ok, I fixed a few more issues. @superknutsel , @Fedik would you be willing to test this again? I think I finally got this issue solved for good. Would be nice if we could get this into 4.2 still.

[Fedik]

I have tested this item ✅ successfully on 1b625a3

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36749.}

[Fedik]

Still works for me, tried with ~33k words

[Hackwar]

I tried it with "War and Peace" from Tolstoi and while it failed to store the whole text, (because the input was too large) about a quarter of it was saved and then properly indexed.

[superknutsel]

I have tested this item ✅ successfully on 1b625a3

Hi, this time I tested it successfully.
Thanks for this.

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36749.}

[richard67]

RTC

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36749.}

[HLeithner]

Not sure what this PR fixes because for me "War and Peace" with 597.373 words worked before and afterwards but at least we have an option less YaY.

PS: tinymce is not optimized for writing books^^

[heelc29]

@HLeithner Can you please add the milestone Joomla! 4.2.0?