change solrpy library to pysolr

[joetsoi]

solrpy is unmaintained and by some handwaving not actually realistic benchmarks, slower, even the author of solrpy has said use something else like pysolr

Leonardo Santagada  30/08/2014

I don't think solrpy supports collection api. Talking about not supporting stuff... 
I'm one of the authors of solrpy and I'm thinking why don't we just put a link to pysolr 
and declare this library deprecated? Has someone looked at pysolr and can comment 
if there is any advantage that we still have over it?

https://groups.google.com/forum/#!searchin/solrpy/pysolr/solrpy/OPuV-jxvtRU/ZyHJ5mONIPoJ

from what I recall, solrpy does more xml transformation where as pysolr just requests the json.
pysolr also uses requests instead of handling the underlying connection itself, so our code gets to be a bit cleaner as well.

We do lose the basic http auth config options for solr as pysolr does not support this, I don't mind as this was actually undocumented on solrpy if I recall anyway)

(also because i'm doing this https://github.com/joetsoi/fedora-rpms-for-ckan)

[joetsoi]

(needs to have the legacy tests / new tests stuff fixed.)

[amercader]

@joetsoi This looks good, what's left to do with the tests?

It seems like basic auth could be reimplmemented if necessary, but I agree that we shouldn't worry too much about it for now

[joetsoi]

I meant I need to fix the branch for the new_tests being moved to tests and tests being moved to legacy tests.

[amercader]

@joetsoi can you briefly explain why this is needed?

[joetsoi]

If I recall correctly

• solrpy requests xml from solr and parses and allows you to search in two different ways
  search (returns datetime types) and raw_search (returns text)
• pysolr parses the json (which does not provide a datetime type) and returns that in its search results

Since this is ckan, we like being difficult so we obviously use both raw and non raw searches in solrpy in differeent places and expect the results from solr to be a datetime and text other times.

So I had to add the decoding of dates to pysolr for the functions expecting datetime objects and some for when it wasn't

if i recall I lifted the datetime regex from pysolr (https://github.com/toastdriven/pysolr/blob/master/pysolr.py#L68) I think I should probably import it instead.

[wardi]

@amercader how do you feel about merging this for 2.5?

[rossjones]

Although this is a pretty old PR, I think updating to maintained deps is probably a good thing. Should we unassign this and see if anyone is willing to volunteer to push this through?

[amercader]

IIRC the only thing that hold me back was the dates magic going on on this PR. But if that makes sense to someone else who has time to look into it, let's go for it

[joetsoi]

If you want me to update this to master I can
On 12 Feb 2016 11:32 am, "Adrià Mercader" notifications@github.com wrote:

IIRC the only thing that hold me back was the dates magic going on on this
PR. But if that makes sense to someone else who has time to look into it,
let's go for it

—
Reply to this email directly or view it on GitHub
#2352 (comment).

[rossjones]

@joetsoi That was be amazing, thank you.

[wardi]

@TkTech since you're interested ckan/ideas#174 (comment) , would you like to provide an update for this PR so I can merge?

[joetsoi]

Changed so that we are importing the regex instead of copy pasting it.