dumb search vs smart search

[nzyeehaa]

J4.1.2 smart search includes ARCHIVED pages

setup
plugins > search "smart"

• disable all
• enable "content - smart search"
• enable "smart search - content"
• enable "smart search - categories"
  components > smart search
• index
• index again
• maps > unpublish category that contains archived pages
  (luckily I have this well organised and ALL archived pages are in one category)
• index again

search for archived page - still shows up = fail

components > smart search > filter > newFilter
modules > smart search

• set to use newFilter
  menu > smart search
• set to use newFilter

search for published page or archived page - nothing shows up

*** smart search is dumb and needs lessons ***

delete filters / delete index - > start again fresh
index
then forget maps
newFilter
set module for newFilter
set menu for newFilter
fail

*** smart search is dumb and needs lessons ***

• invert filter
  test = fail
• delete filter - they dont work
• • reset menu
• • reset module
• maps > unpublish category that contains archived pages
  test = fail
  re-index
  test = fail
• maps > publish all
  test = fail
  re-index
  test = fail

CONCLUSION
its just me
or
smart doesnt belong in the same sentence as search

[nzyeehaa]

cont'd...
modules > smart search

• set to use newFilter
  menu > smart search
• set to use newFilter

produces no results = dead and fail

change module to no filter
leave menu with filter
produces results ( suggesting there is a clash when you use a smart search module with filter) (also still includes archived)

test...
menu with filter
module with filter
assign module to home page only
search = fail

module with no filter
search = results

== there is a problem with the smart search module with filter applied

[nzyeehaa]

further tests with smart search

use a filter on smart search module...

• no settings on filter = shows articles and categories

• set search by type to category = shows categories with search term in category title and/or category description

• set search by type to article = shows articles

• set search by type to article and category = shows articles and categories

• set search by type to nil AND set search by category to a single (first level) category = shows articles

• set search by category with two (first level) categories = shows articles

• set search by category with two (first level) categories AND a single (second level) category (first in list)

  • FAILS - shows nothing

• set search by category to a single (second level) category (first in list) = shows articles

• set search by category to two (second level) categories = shows articles

• set search by category to parent AND all of its second level categories

  • FAILS - shows nothing

this installation is upgraded from J3.10.3 - disable old search component - retest - problems still exist

problems...
** Indexing doesnt pick up empty categories **
** indexing floods database with data that is already listed
** populate a previously empty category = manual re-index **
** smart search doesn't cope with subcategories **
** If a sub-category is empty - search can fail **
** smart search shows pages with latest articles module on - if search term is in latest articles list **
** search unable to omit archived pages **

[Hackwar]

Thank you for all the tests and these reports. I can confirm that a few of the issues are valid, others I have to test myself and one or two are mainly a misunderstanding. Maybe to give you some background, Smart Search was inherited from a company who donated it to Joomla and I would assume that they donated something that wasn't 100% finished. I also assume that they inherited/copied parts of the code from somewhere else. Then for a long time, no one cared for it or dared to touch it. For 4.0 I took this part of Joomla up and invested several hundred hours to get the worst issues solved, but there are still quite a few things to do. (as you can see when filtering the PRs for Joomla for me as an author.) But to make it short: Just blame it all on me. ;-)

I will look into all your reported issues, but it will take me some time. Unfortunately most likely months. But rest assured that I'm not forgetting this.

[nzyeehaa]

@Hackwar I appreciate all the work you have done - I wont be blaming you for anything - I know from experience how much time is invested in these projects - designing, building, testing, swearing, integrating, swearing, troubleshooting and documenting, and it is even worse taking over from where someone left off.

you will be 100% correct about some misunderstandings on my part, I was only working from the operation end and didnt dig into the code at all, just expected it to work.

Errors aside - This is a simplistic way to look at it (to save time) but if I select and export finder from my database it takes 14,012kb
the entire content (800 articles) is 1,628kb, categories (115 categories) adds about 50kb.

This website is being redeveloped and is on a fresh install of 3.10 with imported database, then upgraded to 4.1. the import only covered articles/cats/menus/users - so essentially it is a fresh install, and has been running for 2 weeks.

Why would finder need 8.5 times the space of the data it is searching through?
The data is already there, and the operational code is in php.
Finder will only get bigger as it records binary blobs of each search(even if it is a repeated search).
...for others- I would recommend turning off search statistics.

Then the errors,
if the home page has 10 latest articles listed in a module. and a unique new article was on the list - the home page was in the search results when searching for the unique term. it shouldnt have even appeared in the results

...for the following I will apologise in advance for an overly simplistic view, and I still haven't checked the code...

this may be completely incorrect but some loose logic...
first - for a visitor, a website is made of folders and pages as per the url
second - if someone is searching they want the pages with the answers they are looking for
third - if the site manager doesnt want some pages shown - they may be blocked

code
-> check searcher is not a bot (protect hits on database)
-> if finder>options = include cats...

• sql ->
  • check access level or bounce
  • check is published, (and is not in a finder exclusion list) or bounce
  • search cat/title + description
  • if hit - array cat #

-> then search articles...

• sql ->
  • check access level or bounce
  • check is published (and is not in a finder exclusion list) or bounce
  • search article/title + intro + fulltext,
  • if hit - array article #

-> add a hook for other components results

• eg: shop ( cat / product / description )

-> if finder>options = log search term

• do that - append->searchterm / add->count

-> for each # in the arrays( limit to finder>options for length) = sort by total hits on keyword = display results
done

{menus and SEF routers can complicate things somewhat}

• All finder is storing is options, and a record of the search term + count if its enabled (which should be purged after a set amount of time) [the blob in the database is superfluous (unless it is encoded MD5 and hidden for another reason- it is not shown in the admin panel) / and the search can be repeated using the same term anyway]
• any meta keywords and tags should already be in the articles so will be found. the only places to search are title/intro/fulltext (the things visitors see on the pages)
• • the waterfall is already set - category (if included) is important, then title(most important), then content - just like a search engine does - - the visitor wants to see the pages

further - as a result of joomla's decision to drop the old search from 4.0, there may be a lot of pressure to have this working correctly

It is always best to start with a fresh install of 4.1 rather than an upgrade from 3.10 - which means you only have finder
I upgraded from a fresh 3.10 install and the old search stopped working.

there are also very limited options for j4 search extensions

[Hackwar]

There is a stark difference between the old com_search and the "new" com_finder, because com_finder works with an index and not with direct queries on the original data sources. That means, that there is no difference in load by search queries when using more data sources or that it needs to do additional checking anywhere. The data is already normalised and standardised upon indexing. That is also the reason why it needs a couple times the amount of space for the index, because effectively everything is stored 4 times. Yes, that is correct and how it is supposed to be. The blob itself also is important, since that is what is the actual content of the results page. I will try to answer more indepth soon.

[nzyeehaa]

@Hackwar thanks for your description, and taking the time to reply.
I figured that would be the setup and the reasons for it. and not having every search trigger the original data source may have some benefits.

It appears the blobs work a little different - If a website is fairly dense for a single keyword, eg: this current site, may have 500 page results for one search term which would trigger 500 blobs?
currently if you hit my website and search for blue, then someone else searches for blue, then I have my original dataset, plus index, plus blob times 2.
3 searches at different times for the exact same term with 1 content page result created 3 blobs. So it seems the blobs are not re-used(i guess due to a risk of expired content) - the database is going to be massive.
The one content page result from a 'unique search term' has 52 blobs in my 2 week old website.

I get the need for a mirrored data or indexed data, and I havent done any tests on round trip efficiency, but finder is still accessing the same database and utilising the same channels/bandwidth. the only potential savings would be in not having finder trolling the dataset at the same time as a page request, however it is still trolling its own copied data and using the bandwidth.

when testing the other day I would add a unique page then search for it, and it wouldnt be there.
The index would repopulate eventually(maybe after a hit on the website), and the results would finally show.
trash the article and it was fairly quick result(without a website hit)
and as previously reported - archive it and it still shows in search

finder adds ?highlight... string to the URL so the search page needs noindex, nofollow tag set to avoid duplicate content
but the clicked URL still has ?highlight... added to the pages original URL which will still trigger duplicate content in google as well, and there is no way of avoiding it

duplicate content

• ?view=article&id=831:psycho&catid=2&highlight=WyJwc3ljaG8iXQ==
• You are here: Search > psycho

• You are here: Uncategorised > psycho
• ?view=article&id=831:psycho&catid=2

and that highliight hash, as that applies to each different search term the SEO results could be catastrophic?

[nzyeehaa]

while I have a number of issues with smart search and the old search is not working on my upgraded site...
I have made a clean install of J4.1 to test the smart search / finder operation.

smart search includes _archived_ content by default.

filters were unable to remove archived pages

on my clean install I added a page called...

• published art which was published
• unpublished art which was unpublished
• archived art which was archived
• trashed art which was trashed

and did a search for art - finder displays ...

• published art
• archived art

I don't want archived pages showing up in my search results...
to stop this "feature" from happening...
open administrator\components\com_finder\src\Indexer\Adapter.php
navigate to the bottom of the page - (line 901 on my file) and look for "case 2: " then comment it out like this...

// Published ~and archived items only~ should return a published state
case 1:
// case 2:
return 1;

go to administrator panel > components > smart search > index and press the "index" button at the top

search should omit them from search results. (Your mileage may vary - you may need to delete the index and do a re-index)

Be aware - future updates may break these changes, so until they are included in the core, you may need to redo this process every time you update Joomla.

Hope it helps someone.

(explanation...
case -2 is trashed
case 0 is unpublished
case 1 is published
case 2 is archived)