Skip to content

IULRDC-201 RDC: adding stemming to solr - #73

Merged
aploshay merged 3 commits into
mainfrom
IULRDC-201_Add_stemming_to_solr_search
Jul 28, 2026
Merged

IULRDC-201 RDC: adding stemming to solr#73
aploshay merged 3 commits into
mainfrom
IULRDC-201_Add_stemming_to_solr_search

Conversation

@rdlebeau

@rdlebeau rdlebeau commented May 7, 2026

Copy link
Copy Markdown
Contributor

Adding SnowballPorterFilterFactory to solr search indexing.
On localhost, cancelling out of lando rebuild then running landocommand served to re-index the database.
(lando destroy deleted the database.)

@rdlebeau
rdlebeau requested a review from aploshay May 7, 2026 14:19
@aploshay

Copy link
Copy Markdown
Contributor

Could you add comments in the pull request and/or pertinent commit, re: the effects of each filter removed or added?

@rdlebeau

rdlebeau commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Removing Solr token filter factory

  • EnglishMinimalStemFilterFactory reduces English plural words to singular words for search purposes. We want a more aggresive approach, so we're no longer using it.

Adding Solr token filter factories

  • EnglishPossessiveFilterFactory removes English possessives from search words. The word "cat's" becomes "cat" etc. in a search.

  • LowerCaseFilterFactory enables case-insensitive searching. It converts uppercase letters to lowercase, while leaving numbers and other characters unchanged.

  • SnowballPorterFilterFactory reduces search words to their root form using Snowball stemming algorithms. This is the big change that we want, the one that allows us to search for "Nielsen" and see "NielsenIQ" in the search results. It also reduces English plural words to singular words, so we don't lose any functionality by removing EnglishMinimalStemFilterFactory.

  • NGramFilterFactory splits up search words to support partial word matching. Adding it lets us search "IQ" and get "NielsenIQ" even though it's the second half of the word. We can also search "ielsenI" for example and see the "NielsenIQ" data set in the search results.

  • TrimFilterFactory removes leading and trailing whitespace from search words.

@aploshay
aploshay force-pushed the IULRDC-201_Add_stemming_to_solr_search branch from 3321592 to 664f917 Compare July 28, 2026 14:00
@aploshay
aploshay merged commit 39f572a into main Jul 28, 2026
@aploshay
aploshay deleted the IULRDC-201_Add_stemming_to_solr_search branch July 28, 2026 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants