migrate to Solr 6.x, improve search

[kltm]

There are many uses for the new features, including items like searching facets, facet filtering.

[kltm]

In addition to the many new uses, we'd hope to get rid of a lot of cruft and nuisances during the migration, mostly related to how we deal with getting fields into a searchable state. There is more information about this in kltm/amigo#131, but the short of it is that we'd like to redo enough of the loading with the new features available that:

• less "goosing" tricks are needed
• writing our own wildcarding routines becomes unnecessary
• results for p53 would be massively favoured over p53d
• issues around colons and how they act as whitespace in current setup
• etc

Likely costs will be some changes necessary to the API (especially URL construction) and more disk space necessary.

I suspect that the literal migration itself would not be too bad, with the main issues being in setting up the new configuration files (parsers, tokenizers) and tuning them to be what we wanted to be up front instead of making modifications and finding workarounds as we go (see the current "hack" with the term autocomplete profile and general search doc types (although the latter is a little more justified) due to incompatible fields)).

[kltm]

Honestly, it might be worth starting to consider jumping straight to 5.x:
https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+4+to+Solr+5
https://news.ycombinator.com/item?id=9074860

[kltm]

Poking @cmungall just so he knows where this item is and can comment as needed.

[cmungall]

+1 to the jump to 5

[kltm]

Turn up to 6.x!