API Help - Moving Search from front end to back end

[Kapeli]

Searching
How do I perform a search using the API and have it return the same results you get by using the search field at metacpan.org? For example, these results for LWP?

I tried http://api.metacpan.org/v0/_search?q=lwp but it doesn't return LWP.

I guess I need a by-pod search, but http://api.metacpan.org/v0/pod/_search?q=lwp doesn't seem to work.

Versioning
Is there any API to get all the available versions for a pod/module?

Sorry if any of my terminology is wrong, I don't use Perl, just trying to integrate metacpan.org with Dash. I have no idea what the difference between pod, module, release and distribution is.

[oalders]

Oh, wow. I have Dash integrated with Alfred. This would be cool to see. :) It looks like there's a mismatch between what that API endpoint offers and what the actual front end is using:

https://metacpan.org/search/autocomplete?q=lwp&limit=20

You could use that as a hack to get started with, but the API endpoint is the stable one, so you could switch to that once we see what is going on.

I'll dig something up for you regarding the versioning question and add it to this ticket in the next day or so.

[Kapeli]

I'll wait for a stable API equivalent before doing anything.

Just to give a bit more context: what I'm trying to make is the equivalent of this, but for Perl stuff:

[monken]

The api endpoint for autocomplete is located here: https://api.metacpan.org/search/autocomplete?q=lwp&limit=20
You are doing a search on the whole index and that will return poor results. Not sure what you are after though... Maybe you could specify better your requirements?

[Kapeli]

I don't think the autocomplete API is suitable for my needs, as it does not give the same results as performing an actual search.

I've specified my requirements in my first comment:

How do I perform a search using the API and have it return the same results you get by using the search field at metacpan.org? For example, these results for LWP?

Is there any API to get all the available versions for a pod/module?

[monken]

Try https://api.metacpan.org/search/history/module/Moo?fields=version,release,author

[Kapeli]

That's great for grabbing versions. Thanks! Is there any way to make it give more than 10 versions without using the scrolling API?

[oalders]

https://api.metacpan.org/search/history/module/Moo?fields=version,release,author&size=50

[Kapeli]

Awesome, now how do I search using the API and get the same results as https://metacpan.org/search?q=Moo?

[oalders]

@Kapeli for the https://metacpan.org/search?q=Moo, do you mean that you want the collapsed search results as well, or just the main hits as pointed out in the image below?

[Kapeli]

Just the main hits.

[oalders]

Well, the basic logic to find documentation can be found here, but it you would need to narrow down the search as this will just return everything. https://github.com/CPAN-API/metacpan-examples/blob/master/scripts/endpoints/module/2-fetch-all-modules.pl Having said that, I think we really need to add a convenience endpoint that handles this logic so that searching for modules becomes more straightforward. @monken what do you think? I know we've talked about this before. We could maybe call it /v0/documentation or something like that. Essentially it would apply the logic in the referenced script and then merge in and search parameters the user provides.

[tgt]

I'm sorry for commenting on this without contributing anything useful, but I'm very interested in seeing Dash add support for CPAN modules. Do you have an ETA for implementing these changes?

Thanks :)

[oalders]

@tgt No timeline yet. Just waiting for a volunteer to take this on. It's something that we just need in general.

[marcusramberg]

I'm very interested in seeing this happen. Anything I can do to help? What does the OPfW 2014 tag mean?

[oalders]

@marcusramberg The OPfW label was for the "Outreach Program for Women", but since it was never assigned to an intern, I've just removed that label. I think we could follow the suggestion I made above. We should create a convenience endpoint which allows a proper module search. This would be useful outside of the Dash use case. Basically I think a lot of apps already do something similar, but they're all using some home rolled solution. This would belong under the search convenience endpoint.

Sample convenience endpoints:

http://api.metacpan.org/v0/search/autocomplete
http://api.metacpan.org/v0/search/history
http://api.metacpan.org/v0/search/reverse_dependencies/ETHER/Moose-2.1402

To do the actual dev work, your best bet would be to set up the VM: https://github.com/CPAN-API/metacpan-developer

Lots of help is available in #metacpan. :)

[skarfacegc]

So, I started on this (didn't know about the project until @oalders mentioned this thread).

What I have so far is at https://github.com/skarfacegc/CPANDash The getPods.pl currently fetches the list of Packages/Modules, writes them to the DB, generates the documentation url, then verifies all of the links (removing those that don't exist). This takes ~30 hours to run (most of the time is validating the URLs) and it actually does seem to work. Not sure how good the coverage is, or if I'm including too much junk.

I'm going to mess around with pulling down the pod docs, I'll update that repo as I get stuff.

[skarfacegc]

I have my script running now, pulling down the pod documentation and storing locally. My testing on smaller sample sizes seems to be working fine. I'll update this thread when my long run finishes (12+ hours running now, about halfway done with pulling the pods)

Also, the link to my repo above should contain the code I'm using to pull down the pod docs. Please take a look and let me know if I'm missing anything obvious. This isnt the best code in the world (no tests etc), but it appears to be working.

[lnxbil]

@skarfacegc Nice work! Are you generating everything? Maybe it would be good to build module documentation on demand like the gem-stuff for ruby?

[skarfacegc]

I'm downloading all of the pods in html form via the metacpan api. My first full run just finished. Moving the docs into place now to test.

And it works! 140086 docs.

[skarfacegc]

Doing a final run with some styling and syntax highlighting. Will update once I publish it.

[oalders]

@skarfacegc what's the status on this?

[skarfacegc]

havent touched it in a while. Work got busy and I kinda lost track of it. I'll take a look to see where everything is and update. I'm pretty sure I pushed all the most recent code to my repo.

[Relequestual]

Any further plans on this issue?
@skarfacegc's solution seems to work, but I guess doesn't really fit with how other third party sources work currently, right @Kapeli ?

[Kapeli]

Not sure I fully understand what @skarfacegc's approach does, but I'd say it doesn't fit with what I need.

Just to reiterate: what I need is a search API that searches for modules/pods given a query and once I have that I think I have everything I need.

[Relequestual]

I dunno if it's quite what you want @Kapeli , but it looks like you can do /distribution/_search?q=DBIx-Class (https://explorer.metacpan.org). However, you can search like /distribution/_search?q=DBIx::Class which is the format I expect people would use for perl modules. You could always replaces spaces with :: or - with :: though.

[Kapeli]

The issue is that the results aren't very good. For example, compare the results at http://api.metacpan.org/distribution/_search?q=lwp with the ones at https://metacpan.org/search?q=lwp.

[Relequestual]

Oh I totally agree! =[
Totally get that it needs to be an equally good expeirence. The annoying thing is, the CODE must be there, just needs to be put together, right? =/

[aufflick]

FWIW all, I have a really basic per-module script to generate docsets using the current metacpan api (inspired by @skarfacegc's script, but there's no way that would even ever finish running on Australian Internet!)

https://github.com/aufflick/p5-App-MetaCPAN-Dash

with the module installed you just:

metacpan_dash.pl Acme::Bleach

To build and install a docset just for that distribution.

[oalders]

I haven't looked at the code, but we could look into periodically generating a docset on one of our MetaCPAN staging machines.

[Relequestual]

@aufflick I could run this over the weekend! Although I'd like to have a bit more documentation before doing so =]
You could pipe in all the modules one has installed on a system

[aufflick]

@Relequestual I'm planning to make my module read the current cpanfile and install/update docs for those modules.

[Relequestual]

@aufflick Consider making sure it's also the same verson of the module =]

[aufflick]

Hm good point - you may in fact want multiple versions for the same module installed. I'll probably have to namespace the docsets somehow.

[Relequestual]

I mention because often in a production envrionment, you're limited to the version of the perl module for the distribution in use. So yes, you may want multiple versions of the docs also. I may use one version for work and another for pleasure.

[ranguard]

Closing as seems to relate to the API more than the web interface, if still need help please open an issue there.

[oalders]

Actually, let's keep this one open. This is an issue I have some interest in seeing resolved. It is more about the API, but it's OK to leave it here.

[oalders]

This search API now exists on the back end. It just needs to be documented.

[oalders]

@Kapeli we were chatting about this issue last night. We haven't forgotten about it.

[oalders]

@Kapeli the endpoint has now been documented: https://fastapi.metacpan.org/static/index.html I'm going to close this issue but if you have any questions at all, please open a new issue and we'll chat about it there.

[Kapeli]

Thank you! 👍

[oalders]

Thank you. I use Dash pretty much every day. 👍