-
Notifications
You must be signed in to change notification settings - Fork 131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to search for items with an embedded hyphen? #108
Comments
I suspect that this is what you're looking for: https://github.com/lucaong/minisearch#tokenization Specifically, the default Line 1239 in 8c66bd2
Both on indexing and on searching, that therefore gets split into |
That’s correct @JustAnotherArchivist , the default tokenizer splits by space or punctuation, and Unfortunately the modified regular expression is quite cumbersome if one needs to take care of non-English characters, diacritics, etc., but this should work: // This is almost the same regexp as
// the default one, but excludes the
// hyphen by having ,./ instead of ,-/
const CUSTOM_SPACE_OR_PUNCT = /[\n\r -#%-*,./:;?@[-\]_{}\u00A0\u00A1\u00A7\u00AB\u00B6\u00B7\u00BB\u00BF\u037E\u0387\u055A-\u055F\u0589\u058A\u05BE\u05C0\u05C3\u05C6\u05F3\u05F4\u0609\u060A\u060C\u060D\u061B\u061E\u061F\u066A-\u066D\u06D4\u0700-\u070D\u07F7-\u07F9\u0830-\u083E\u085E\u0964\u0965\u0970\u09FD\u0A76\u0AF0\u0C77\u0C84\u0DF4\u0E4F\u0E5A\u0E5B\u0F04-\u0F12\u0F14\u0F3A-\u0F3D\u0F85\u0FD0-\u0FD4\u0FD9\u0FDA\u104A-\u104F\u10FB\u1360-\u1368\u1400\u166E\u1680\u169B\u169C\u16EB-\u16ED\u1735\u1736\u17D4-\u17D6\u17D8-\u17DA\u1800-\u180A\u1944\u1945\u1A1E\u1A1F\u1AA0-\u1AA6\u1AA8-\u1AAD\u1B5A-\u1B60\u1BFC-\u1BFF\u1C3B-\u1C3F\u1C7E\u1C7F\u1CC0-\u1CC7\u1CD3\u2000-\u200A\u2010-\u2029\u202F-\u2043\u2045-\u2051\u2053-\u205F\u207D\u207E\u208D\u208E\u2308-\u230B\u2329\u232A\u2768-\u2775\u27C5\u27C6\u27E6-\u27EF\u2983-\u2998\u29D8-\u29DB\u29FC\u29FD\u2CF9-\u2CFC\u2CFE\u2CFF\u2D70\u2E00-\u2E2E\u2E30-\u2E4F\u3000-\u3003\u3008-\u3011\u3014-\u301F\u3030\u303D\u30A0\u30FB\uA4FE\uA4FF\uA60D-\uA60F\uA673\uA67E\uA6F2-\uA6F7\uA874-\uA877\uA8CE\uA8CF\uA8F8-\uA8FA\uA8FC\uA92E\uA92F\uA95F\uA9C1-\uA9CD\uA9DE\uA9DF\uAA5C-\uAA5F\uAADE\uAADF\uAAF0\uAAF1\uABEB\uFD3E\uFD3F\uFE10-\uFE19\uFE30-\uFE52\uFE54-\uFE61\uFE63\uFE68\uFE6A\uFE6B\uFF01-\uFF03\uFF05-\uFF0A\uFF0C-\uFF0F\uFF1A\uFF1B\uFF1F\uFF20\uFF3B-\uFF3D\uFF3F\uFF5B\uFF5D\uFF5F-\uFF65]+/u
const miniSearch = new MiniSearch({
// …
tokenize: (text) => text.split(CUSTOM_SPACE_OR_PUNCT)
}) |
Closing this issue for now, as the above solution should work. @cmcknight feel free to comment further if necessary, and I will reopen the issue in case. |
Hi All, Sorry for not replying sooner. I've been sidelined due to an illness. This makes sense and I agree that it can be closed. |
I have a use case for item titles that have embedded hyphens, i.e.:
C-8
AR-7
PGMP-15
I seem to get back a lot of completely unrelated results that do not have the search terms in them. I do find the item I'm looking for about halfway or two-thirds down the list.
This is using the generic search (no ands, etc.). I feel like I'm missing something.
The text was updated successfully, but these errors were encountered: