New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API Help - Moving Search from front end to back end #1114

Closed
Kapeli opened this Issue Feb 26, 2014 · 41 comments

Comments

Projects
@Kapeli
Copy link

Kapeli commented Feb 26, 2014

Searching
How do I perform a search using the API and have it return the same results you get by using the search field at metacpan.org? For example, these results for LWP?

I tried http://api.metacpan.org/v0/_search?q=lwp but it doesn't return LWP.

I guess I need a by-pod search, but http://api.metacpan.org/v0/pod/_search?q=lwp doesn't seem to work.

Versioning
Is there any API to get all the available versions for a pod/module?

Sorry if any of my terminology is wrong, I don't use Perl, just trying to integrate metacpan.org with Dash. I have no idea what the difference between pod, module, release and distribution is.

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Feb 27, 2014

Oh, wow. I have Dash integrated with Alfred. This would be cool to see. :) It looks like there's a mismatch between what that API endpoint offers and what the actual front end is using:

https://metacpan.org/search/autocomplete?q=lwp&limit=20

You could use that as a hack to get started with, but the API endpoint is the stable one, so you could switch to that once we see what is going on.

I'll dig something up for you regarding the versioning question and add it to this ticket in the next day or so.

@Kapeli

This comment has been minimized.

Copy link

Kapeli commented Feb 27, 2014

I'll wait for a stable API equivalent before doing anything.

Just to give a bit more context: what I'm trying to make is the equivalent of this, but for Perl stuff:

screen shot 2014-02-27 at 17 40 16

@monken

This comment has been minimized.

Copy link
Member

monken commented Feb 28, 2014

The api endpoint for autocomplete is located here: https://api.metacpan.org/search/autocomplete?q=lwp&limit=20
You are doing a search on the whole index and that will return poor results. Not sure what you are after though... Maybe you could specify better your requirements?

@Kapeli

This comment has been minimized.

Copy link

Kapeli commented Feb 28, 2014

I don't think the autocomplete API is suitable for my needs, as it does not give the same results as performing an actual search.

I've specified my requirements in my first comment:

How do I perform a search using the API and have it return the same results you get by using the search field at metacpan.org? For example, these results for LWP?

Is there any API to get all the available versions for a pod/module?

@monken

This comment has been minimized.

@Kapeli

This comment has been minimized.

Copy link

Kapeli commented Feb 28, 2014

That's great for grabbing versions. Thanks! Is there any way to make it give more than 10 versions without using the scrolling API?

@Kapeli

This comment has been minimized.

Copy link

Kapeli commented Feb 28, 2014

Awesome, now how do I search using the API and get the same results as https://metacpan.org/search?q=Moo?

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Mar 1, 2014

@Kapeli for the https://metacpan.org/search?q=Moo, do you mean that you want the collapsed search results as well, or just the main hits as pointed out in the image below?

image

@Kapeli

This comment has been minimized.

Copy link

Kapeli commented Mar 1, 2014

Just the main hits.

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Mar 3, 2014

Well, the basic logic to find documentation can be found here, but it you would need to narrow down the search as this will just return everything. https://github.com/CPAN-API/metacpan-examples/blob/master/scripts/endpoints/module/2-fetch-all-modules.pl Having said that, I think we really need to add a convenience endpoint that handles this logic so that searching for modules becomes more straightforward. @monken what do you think? I know we've talked about this before. We could maybe call it /v0/documentation or something like that. Essentially it would apply the logic in the referenced script and then merge in and search parameters the user provides.

@tgt

This comment has been minimized.

Copy link

tgt commented Mar 19, 2014

I'm sorry for commenting on this without contributing anything useful, but I'm very interested in seeing Dash add support for CPAN modules. Do you have an ETA for implementing these changes?

Thanks :)

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Mar 20, 2014

@tgt No timeline yet. Just waiting for a volunteer to take this on. It's something that we just need in general.

@oalders oalders added the OPfW 2014 label Oct 8, 2014

@Kapeli Kapeli referenced this issue Nov 26, 2014

Closed

Add CPAN doc set #234

@marcusramberg

This comment has been minimized.

Copy link

marcusramberg commented Jan 10, 2015

I'm very interested in seeing this happen. Anything I can do to help? What does the OPfW 2014 tag mean?

@oalders oalders removed the OPfW 2014 label Jan 11, 2015

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Jan 11, 2015

@marcusramberg The OPfW label was for the "Outreach Program for Women", but since it was never assigned to an intern, I've just removed that label. I think we could follow the suggestion I made above. We should create a convenience endpoint which allows a proper module search. This would be useful outside of the Dash use case. Basically I think a lot of apps already do something similar, but they're all using some home rolled solution. This would belong under the search convenience endpoint.

Sample convenience endpoints:

http://api.metacpan.org/v0/search/autocomplete
http://api.metacpan.org/v0/search/history
http://api.metacpan.org/v0/search/reverse_dependencies/ETHER/Moose-2.1402

To do the actual dev work, your best bet would be to set up the VM: https://github.com/CPAN-API/metacpan-developer

Lots of help is available in #metacpan. :)

@skarfacegc

This comment has been minimized.

Copy link

skarfacegc commented Feb 24, 2015

So, I started on this (didn't know about the project until @oalders mentioned this thread).

What I have so far is at https://github.com/skarfacegc/CPANDash The getPods.pl currently fetches the list of Packages/Modules, writes them to the DB, generates the documentation url, then verifies all of the links (removing those that don't exist). This takes ~30 hours to run (most of the time is validating the URLs) and it actually does seem to work. Not sure how good the coverage is, or if I'm including too much junk.

I'm going to mess around with pulling down the pod docs, I'll update that repo as I get stuff.

@skarfacegc

This comment has been minimized.

Copy link

skarfacegc commented Feb 24, 2015

I have my script running now, pulling down the pod documentation and storing locally. My testing on smaller sample sizes seems to be working fine. I'll update this thread when my long run finishes (12+ hours running now, about halfway done with pulling the pods)

Also, the link to my repo above should contain the code I'm using to pull down the pod docs. Please take a look and let me know if I'm missing anything obvious. This isnt the best code in the world (no tests etc), but it appears to be working.

@lnxbil

This comment has been minimized.

Copy link

lnxbil commented Feb 24, 2015

@skarfacegc Nice work! Are you generating everything? Maybe it would be good to build module documentation on demand like the gem-stuff for ruby?

@skarfacegc

This comment has been minimized.

Copy link

skarfacegc commented Feb 25, 2015

screen shot 2015-02-24 at 7 22 44 pm
I'm downloading all of the pods in html form via the metacpan api. My first full run just finished. Moving the docs into place now to test.

And it works! 140086 docs.

@skarfacegc

This comment has been minimized.

Copy link

skarfacegc commented Feb 27, 2015

Doing a final run with some styling and syntax highlighting. Will update once I publish it.

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Aug 29, 2015

@skarfacegc what's the status on this?

@skarfacegc

This comment has been minimized.

Copy link

skarfacegc commented Aug 30, 2015

havent touched it in a while. Work got busy and I kinda lost track of it. I'll take a look to see where everything is and update. I'm pretty sure I pushed all the most recent code to my repo.

@Relequestual

This comment has been minimized.

Copy link

Relequestual commented Jan 8, 2016

Any further plans on this issue?
@skarfacegc's solution seems to work, but I guess doesn't really fit with how other third party sources work currently, right @Kapeli ?

@Kapeli

This comment has been minimized.

Copy link

Kapeli commented Jan 8, 2016

Not sure I fully understand what @skarfacegc's approach does, but I'd say it doesn't fit with what I need.

Just to reiterate: what I need is a search API that searches for modules/pods given a query and once I have that I think I have everything I need.

@Relequestual

This comment has been minimized.

Copy link

Relequestual commented Jan 8, 2016

I dunno if it's quite what you want @Kapeli , but it looks like you can do /distribution/_search?q=DBIx-Class (https://explorer.metacpan.org). However, you can search like /distribution/_search?q=DBIx::Class which is the format I expect people would use for perl modules. You could always replaces spaces with :: or - with :: though.

@Kapeli

This comment has been minimized.

Copy link

Kapeli commented Jan 8, 2016

The issue is that the results aren't very good. For example, compare the results at http://api.metacpan.org/distribution/_search?q=lwp with the ones at https://metacpan.org/search?q=lwp.

@Relequestual

This comment has been minimized.

Copy link

Relequestual commented Jan 8, 2016

Oh I totally agree! =[
Totally get that it needs to be an equally good expeirence. The annoying thing is, the CODE must be there, just needs to be put together, right? =/

@aufflick

This comment has been minimized.

Copy link

aufflick commented Feb 11, 2016

FWIW all, I have a really basic per-module script to generate docsets using the current metacpan api (inspired by @skarfacegc's script, but there's no way that would even ever finish running on Australian Internet!)

https://github.com/aufflick/p5-App-MetaCPAN-Dash

with the module installed you just:

metacpan_dash.pl Acme::Bleach

To build and install a docset just for that distribution.

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Feb 11, 2016

I haven't looked at the code, but we could look into periodically generating a docset on one of our MetaCPAN staging machines.

@Relequestual

This comment has been minimized.

Copy link

Relequestual commented Feb 11, 2016

@aufflick I could run this over the weekend! Although I'd like to have a bit more documentation before doing so =]
You could pipe in all the modules one has installed on a system

@aufflick

This comment has been minimized.

Copy link

aufflick commented Feb 12, 2016

@Relequestual I'm planning to make my module read the current cpanfile and install/update docs for those modules.

@Relequestual

This comment has been minimized.

Copy link

Relequestual commented Feb 12, 2016

@aufflick Consider making sure it's also the same verson of the module =]

@aufflick

This comment has been minimized.

Copy link

aufflick commented Feb 12, 2016

@Relequestual

This comment has been minimized.

Copy link

Relequestual commented Feb 12, 2016

I mention because often in a production envrionment, you're limited to the version of the perl module for the distribution in use. So yes, you may want multiple versions of the docs also. I may use one version for work and another for pleasure.

@ranguard

This comment has been minimized.

Copy link
Member

ranguard commented Mar 8, 2016

Closing as seems to relate to the API more than the web interface, if still need help please open an issue there.

@ranguard ranguard closed this Mar 8, 2016

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Mar 8, 2016

Actually, let's keep this one open. This is an issue I have some interest in seeing resolved. It is more about the API, but it's OK to leave it here.

@oalders oalders reopened this Mar 8, 2016

@ranguard ranguard added the in progress label Mar 8, 2016

@haarg haarg added the meta::hack label Nov 17, 2016

@ranguard ranguard changed the title API Help API Help - Moving Search from front end to back end Nov 18, 2016

@ranguard ranguard added PTS 2017 and removed meta::hack labels Apr 26, 2017

@ranguard ranguard added this to To discuss in PTS 2017 Apr 28, 2017

@oalders

This comment has been minimized.

Copy link
Member

oalders commented May 12, 2017

This search API now exists on the back end. It just needs to be documented.

@haarg haarg removed the PTS 2017 label Aug 31, 2017

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Apr 20, 2018

@Kapeli we were chatting about this issue last night. We haven't forgotten about it.

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Nov 29, 2018

@Kapeli the endpoint has now been documented: https://fastapi.metacpan.org/static/index.html I'm going to close this issue but if you have any questions at all, please open a new issue and we'll chat about it there.

@oalders oalders closed this Nov 29, 2018

@Kapeli

This comment has been minimized.

Copy link

Kapeli commented Nov 29, 2018

Thank you! 👍

@oalders

This comment has been minimized.

Copy link
Member

oalders commented Nov 29, 2018

Thank you. I use Dash pretty much every day. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment