Better search #69
Replies: 8 comments 14 replies
-
|
I think this is one of my highest valued requests, thanks for posting it! |
Beta Was this translation helpful? Give feedback.
-
|
Agreed! Thanks @neverstew! |
Beta Was this translation helpful? Give feedback.
-
|
V2 beta algorithm is now sneakily available on the search results page. No idea how it works yet! So far, not getting great results... An example that's better:
Some test strings that are not good, for comparison with sqlite search linked above:
|
Beta Was this translation helpful? Give feedback.
-
|
We're still working on this - making sure that the solution scales. I think that the current v2 prototype is pretty dead in the water, sadly, unless we can find a way for |
Beta Was this translation helpful? Give feedback.
-
|
I believe that GIN index + tokenization + shingling + hashing could solve this issue before you move to a more robust system! This is more common in duplicate detection but is a great way to solve for FTS queries where word order really matters directly in Postgres. If interested feel free to dm me on twitter @lntschkr |
Beta Was this translation helpful? Give feedback.
-
|
Just wanted to mention here this workaround that just avoid the problem entirely by dumping all publics vals to github: https://val-town-search.pomdtr.me |
Beta Was this translation helpful? Give feedback.
-
|
Nice! Seems to perform well on the queries @neverstew mentioned: |
Beta Was this translation helpful? Give feedback.
-
|
This is a kinda wild idea and obviously doesn't work for loads of reasons buuuuuut... |
Beta Was this translation helpful? Give feedback.
-
|
I wrote a little bit about it in https://code-search-is-easy.pomdtr.me. I'm not sure it would be allowed by github guidelines, but it is quite an attractive solution indead x) |
Beta Was this translation helpful? Give feedback.
-
|
I prototyped semantic search within Val Town (userspace), based off @pomdtr's work. Three different implementations! I don't quite know how to integrate this with existing search, but ideas and forks are welcome. https://janpaul123-valtownsemanticsearch.web.val.run/ |
Beta Was this translation helpful? Give feedback.
-
|
I've made several iterations on search now. Here are some features we have now:
Try it out! It seems to work pretty well on the queries suggested in this thread. I'd love your feedback! Somethings we've thought about but not implemented yet:
|
Beta Was this translation helpful? Give feedback.
-
|
Would love filtering for type, etc |
Beta Was this translation helpful? Give feedback.
-
|
Shipped! 🎉
Also lets you filter for "my vals" and "not my vals" |
Beta Was this translation helpful? Give feedback.
-
|
We think the new search might be "good enough" for now, so we are thinking about closing this ticket. What do you think @neverstew? |
Beta Was this translation helpful? Give feedback.
-
|
Just spoke with @janpaul123 about "exact phrase search" and it sounds like we already have it. If you search for |
Beta Was this translation helpful? Give feedback.
-
|
It's funny how the new search kind of broke some of my apps (ex: https://www.val.town/v/pomdtr/bookmarklets) It would be nice to be able to opt out from the "smart search" and only get exact matches. |
Beta Was this translation helpful? Give feedback.
-
I agree, or at least in an API. As a workaround: it is possible to do this in userspace. Look at each val that is returned and see if there are exact matches with the code (we don't do anything smart, just case-insensitive matches), and once you hit a val that doesn't have an exact match, stop returning. |
Beta Was this translation helpful? Give feedback.
-
|
Good point. I might just filter all the results in case your logic change. |
Beta Was this translation helpful? Give feedback.
-
|
Close and open other smaller tickets is fine for me 👍 Exact match on the whole input works - great! All of the features must be compatible with the other features though, so I don't class exact matches as working because I can't do |
Beta Was this translation helpful? Give feedback.
-
Val Town is more interesting than e.g. AWS lambda because it aspires to also bundle in the social aspect of sharing ideas and code. But right now, discovering other vals is really difficult because the search is very limited and doesn't function like a typical search.
To understand whether this is a problem or not, you can simply scroll backwards through the #general channel in Discord and look for occasions where someone asks a question like "does anyone know where X is?" or "is there a thing that does Y?" It comes up something like once a week. This doesn't include the fairly easy-to-imagine scenarios like a new user showing up and looking for something that exists to fork, not finding anything, and then leaving.
A while ago I built https://valtown-search.fly.dev/ (code at https://github.com/neverstew/valtown-search/ ) with the stock sqlite search which, whilst not that sophisticated, returns lots of results and supports things like searching with typos or multiple words (not to mention it is fast). Here's an example of a specific search on Val Town that should yield lots of results, but yields nothing (and frustratingly so): "vtdocs telegram". Compare that to "vtdocs/telegram", which yields lots of results.
A small, incremental move to better search would make a huge difference in the social aspect of the site. Maybe, for example, a simple GIN index in the existing postgres database is all that's needed. It should, at a minimum, support everything that a text search using postgres/sqlite would do by default (lemmatisation, levenshtein distance, quotations for phrases etc.)
Beta Was this translation helpful? Give feedback.
All reactions