Hayoo spice

[headprogrammingczar]

Two notes on this code:

1. The Hayoo API returns results out of order. I've contacted the maintainer about this and haven't heard back yet.
2. As a workaround, I wrote a simple heuristic to favor exact matches and standard library results. It does not work for exact matches because DDG.get_query() (suggested by yegg) always returns 'duckduckhack-template-for-spice'. I have no clue what to make of this besides that it's a bug in 'duckpan server'.

[jagtalon]

@headprogrammingczar I think the problem with DDG.get_query() only happens locally and that it should be fine when it goes live.

Update
They just fixed it! #34 (comment)

[moollaza]

@headprogrammingczar hey thanks a lot for the spice and the head's up about the result. Regarding your heuristic, it looks pretty awesome but if you really want we have a function we're using in the movies spice DDG.isRelevant() it has 3 parameters which are

1. the term you're checking the relevancy of
2. an array of skip words (eg. for movies, the words "movie" and "film" should automatically be skipped because if they happen to match they don't add any relevancy) in this case you could add "haskell" to the skip array
3. the shortest length of a word you want to match against (ie. if it was '1' you could match against single-letter words like 'I' or 'a' -- basically the lower that limit the more relaxed the function is and so you have more results but they might not be AS relevant -- the function defaults to an empty skip array and a min word length of 2

I'd say play around with that and see if it makes a difference, again checkout the movie spice for an example of the usage.

[headprogrammingczar]

@moollaza Implemented, replacing the check for exact match. Definitely makes it more flexible.

I got a response back from the Hayoo maintainer, and I will drop the heuristic code once results come back in-order.

I was also surprised to find that on extremely generic searches like "hayoo list", Hayoo returns 5MB of JSON data. I don't know if the code to fix this belongs at Hayoo's end or DDG's, but that's more data than clients should have to download for just a search page.

[moollaza]

@headprogrammingczar hey, thanks for the updates, I'm testing it out and the isRelevant check is breaking on: DDG.isRelevant(fun['type'], ['hayoo']) it seems that most of the results don't have a "name" so its trying to use an ``undefined` hence the crash. I think you'll need to add a check to make sure those elements aren't undefined. I was querying "hayoo add" and "hayoo zipwith"

Also, for trigger I think we should definitely include "haskell" and "hackage" as triggers.

Regarding the 5MB of json...that's just crazy, is there any way we can limit the number of returned results? 50 should be plenty...

[headprogrammingczar]

That crash was actually caused by me not noticing that Hayoo uses the unusual naming convention of fun['signature'] instead of fun['type']. I got sidetracked by the 5MB thing and forgot to test it.

I'm thinking there might be (should be?) something DDG-side to do some light processing before the JSON goes to get cached and sent to the client and things like that. Hayoo can't be the only API that will ever misbehave like this.

[moollaza]

@headprogrammingczar I think I found a way to limit the size of the returned json -- otherwise I guess the hayoo spice is about done. I noticed you didn't add "Haskell" to the list of triggers, want me to do that?

EDIT:

Had a quick look and it doesn't seem like its possibly, I'll keep looking but I noticed their response header doesn't even give the content length. I think if that was there it would be possible. But it might just be easiest if they add a parameter to the api so we can limit the amount returned in the api call. I think the fact that they even allow 11K results to be returned is a little crazy.

Would you be able to contact them and see if they're willing to modify their api?

[headprogrammingczar]

Neat. I'm actually not sure if adding "haskell" to the trigger list is appropriate, since Hoogle is generally considered to be a better search engine. Once this spice is finished, I'm going to try and contact the Hoogle maintainer with a JSON API and get that added as a spice as well.

Also, "Haskell" is a name, so if someone searched for "Haskell Curry", for instance, they would get the function instead of the person:
http://hackage.haskell.org/packages/archive/base/latest/doc/html/Prelude.html#v:curry
https://duckduckgo.com/?q=haskell+curry

EDIT:
Yeah, I'll contact him.

[moollaza]

@headprogrammingczar alright sounds good then. Let me know what they have to say about limiting the size of the json.

Thanks!

[headprogrammingczar]

Got a reply from Hayoo, and I am updating the code accordingly.

---

Hi!

I have fixed the ordering and added a hard limit of 100 for the results
delivered through the API. I will extend this to a limit parameter when
I have some more time.

There is a new version of Hayoo! running here (we are currently in the
process of migrating to a new server):

http://213.39.233.13/hayoo/hayoo.html

DNS will be updated throughout the next days.

Cheers,
Timo

[moollaza]

@headprogrammingczar I pulled in your updates and they work, however they're still returning 5Mb of data...in fact I think its actually returning more?! Now it shows 12000 hits for list -- I think before it was ~11000.

I guess I'll try again Monday, perhaps it'l be fixed by then.

[headprogrammingczar]

DNS just updated on my end with the new Hayoo server. If it's updated on your end, I think this is ready to merge.

[moollaza]

@headprogrammingczar it looks like its working now. I'm not getting 5MB of JSON :)

[moollaza]

@headprogrammingczar trying out some searches but I'm not getting very relevant results "hayoo add" doesn't return the "add" function. The hayoo websites' search however returns it as the first result.

[moollaza]

@headprogrammingczar when you get a chance, would you mind sending an email to moollaza@duckduckgo.com please? Or if you don't mind, can @hunterlang I have your email address? (We have some goodies for our contributors!)

[moollaza]

Alright, fixed it up a bit and its merged. @headprogrammingczar thanks again!