Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search results improvements #308

Open
holyjak opened this issue Jun 4, 2019 · 16 comments
Open

Search results improvements #308

holyjak opened this issue Jun 4, 2019 · 16 comments
Labels

Comments

@holyjak
Copy link
Contributor

@holyjak holyjak commented Jun 4, 2019

Improve results produced by the new search introduced by #85.

Report problems and bad/suboptimal results here. (Check the Known Problems below first, please!)

Known Problems

  • Include download count in the prioritization to show more popular libraries first - see #359
  • Searching for "clojure spec" does not show the library in the presented top results ("org.clojure/spec" does)
  • Should prioritize "standard" versions over weird one, e.g. 0.6.6 over 0.6.7-AlexVPopov_patch_1-SNAPSHOT (?How often does this happen, is it worth the extra complexity?)
@martinklepsch
Copy link
Member

@martinklepsch martinklepsch commented Jun 5, 2019

Not a big deal but one thing I noticed is that searching for metosin json doesn't return metosin/jsonista.

@KingMob
Copy link

@KingMob KingMob commented Jun 10, 2019

I'm not sure if search is set up for non-Clojars sources, but I couldn't find tools.reader using either tools.reader or org.clojure/tools.reader.

@holyjak
Copy link
Contributor Author

@holyjak holyjak commented Jun 12, 2019

Thanks a lot, @KingMob ! It will be fixed in a few minutes. We fetched only the first 10 instead of all 64 artifacts. (Fixed by #314.)
@martinklepsch for the record, the metosin json no results problem has been fixed by #310.

@gphilipp
Copy link

@gphilipp gphilipp commented Jul 5, 2019

I'm searching for jackdaw, but the only result that pops up is some version built off a branch https://cljdoc.org/d/fundingcircle/jackdaw/0.6.7-AlexVPopov_patch_1-SNAPSHOT, instead of the latest release: https://cljdoc.org/d/fundingcircle/jackdaw/0.6.6/doc/readme

@holyjak
Copy link
Contributor Author

@holyjak holyjak commented Jul 10, 2019

I guess the problem is that 0.6.7* > 0.6.6 and the code doesn't really look at the versions to filter out "weird ones". I will look into it, eventually.

@escherize
Copy link

@escherize escherize commented Aug 30, 2019

Firstly, cljdoc is a really cool tool that I am fond of!

I think it should be a goal that when searching for e.g. lacinia, the "main library" (aka com.walmartlabs.lacinia 0.34.0) shows up first, but what I am seeing is:

  • toolbelt-lacinia 0.0.1
  • lacinia-gen 0.1.1-SNAPSHOT
  • lacinia2datomic 0.1.0-SNAPSHOT
  • ... [snip] ...
  • com.walmartlabs.lacinia 0.34.0

Do you agree that this should be considered a bug?

@holyjak
Copy link
Contributor Author

@holyjak holyjak commented Aug 30, 2019

Hi @escherize, that would indeed be ideal. But the code can hardly guess what is the "main library". What we want to do is take into account the download count, that should push the more popular artifacts up. I have a branch where I started working on this but struggle to configure it so that it actually improves the results. It is not trivial :-(

@nha
Copy link
Contributor

@nha nha commented Sep 11, 2019

A search for "turtle" "clj-client" "com.turtlequeue" does not return https://cljdoc.org/d/com.turtlequeue/clj-client/0.0.7
(I have not yet found a way to see it)

@martinklepsch
Copy link
Member

@martinklepsch martinklepsch commented Sep 12, 2019

@nha I think the problem is that only specific group IDs (namely org.clojure) on Maven Central are whitelisted to be included in the search. Reasoning being that we don't want to actually search all of Maven Central :) Maybe we can add the turtlequeue stuff...

(defn load-maven-central-artifacts []
;; GET http://search.maven.org/solrsearch/select?q=g:org.clojure -> JSON .response.docs[] = {g: group, a: artifact-id, latestVersion, versionCount
(try
(with-open [in (io/reader "http://search.maven.org/solrsearch/select?q=g:org.clojure&rows=200")]
(format-maven-central-resp
(json/parse-stream in keyword)))
(catch Exception e
(log/info e "Failed to download artifacts from Maven Central")
nil)))

@nha
Copy link
Contributor

@nha nha commented Sep 16, 2019

@martinklepsch right that seems like a quick fix, something like this:

http://search.maven.org/solrsearch/select?q=g:%22org.clojure%22+OR+g:%22com.turtlequeue%22&rows=200

I can submit a PR it you agree with the above

@holyjak
Copy link
Contributor Author

@holyjak holyjak commented Sep 16, 2019

As soon as we get to 3-4 different groups in Maven Central, we should move them from a hardcoded string to a config file / DB table. For now it seems there will not be many and little churn so hardcoding is OK.

@martinklepsch
Copy link
Member

@martinklepsch martinklepsch commented Sep 16, 2019

@nha I'm ok with hardcoding for now 👍 Maybe use a function to URI encode instead of just manually doing it, that would at least improve readability.

@cloojure
Copy link

@cloojure cloojure commented Jan 1, 2020

Hi - I am still unable to find org.clojure/tools.reader or the ns clojure.tools.reader.edn via the search box. See slack question: https://clojurians.slack.com/archives/C8V0BQ0M6/p1577855041031500

@seancorfield
Copy link

@seancorfield seancorfield commented Jan 1, 2020

As a follow up to that tools.reader does find it but it's the last option in the list (and what look like worse matches are higher in the list).

@holyjak
Copy link
Contributor Author

@holyjak holyjak commented Jan 6, 2020

I know, I am sorry. That should be fixed by #359 but boosting search results so that you get the desired results is sadly quite complicated. I will get back to the PR in the coming weeks and try to finish it.

@holyjak
Copy link
Contributor Author

@holyjak holyjak commented Feb 11, 2021

com.fulcrologic does not find anything though a search for fulcro shows com.fulcrologic/fulcro in the list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants