Solr typeahead #1537

jywarren · 2017-07-13T21:01:14Z

May need a toggle, however, as in #1463

all tests pass -- rake test:all
code is in uniquely-named feature branch, and has been rebased on top of latest master (especially if you've been asked to make additional changes)
pull request is descriptively named with #number reference back to original issue

PublicLabBot · 2017-07-13T21:12:52Z

	3 Messages
📖	@jywarren Thank you for your pull request! I’m here to help with some tips and recommendations. Please take a look at the list provided and help us review and accept your contribution! And don’t be discouraged if you see errors – we’re here to help.
📖	This pull request doesn’t link to a issue number. Please refer to the issue it fixes (if any) in the body of your PR, in the format: `Fixes #123`.
📖	It looks like you haven’t marked all the checkboxes. Help us review and accept your suggested changes by going through the steps one by one. If it is still a ‘Work in progresss’, please include ‘[WIP]’ in the title.

Generated by 🚫 Danger

jywarren · 2017-07-13T21:22:03Z

Ok we need to follow
https://github.com/sunspot/sunspot/blob/master/README.md

jywarren · 2017-07-13T22:51:20Z

@icarito - not sure, i added a new field here, do we need to update the schema when we do that?

jywarren · 2017-07-14T02:51:27Z

May still need a toggle to switch between solr and non solr, and tests for this service.

May need a toggle, however.

Not yet using limit...

jywarren · 2017-07-14T13:46:46Z

OK -- so now, this is working (probably, once my last commit runs) but it takes a timeout to determine if Solr is working... which is too long.

Here, we only need to know if a Solr service is available and we can toggle between -- but we need to know /fast/. Ideas, @icarito ?

jywarren · 2017-07-14T13:48:36Z

Here's the toggle area:

https://github.com/publiclab/plots2/pull/1537/files#diff-1bc1c213d7a40db44441cd43fd8cd96cR35

It depends on an attempt to run a Node.search in the SolrToggle library we built in /lib/ -- but that has to time out before determining which way to go. That'll take forever and also occupy a zillion threads if we put it in production.

icarito · 2017-07-17T18:40:38Z

Is there a possibility to just instead of making a pre-query, try the actual query and fallback only if it fails?

icarito · 2017-08-03T16:26:55Z

The big thing (...) is trying to find a faster way than a timeout to determine if Solr is present (from in a rails app). I think that may be a sunspot question and could be searched on Stack overflow and/or posted on sunspot's issue tracker.

Looking for ideas.

jywarren · 2017-08-24T18:11:50Z

OK, no longer seeing the timeout, but resolving a failure and some errors. Added a check to ensure solr is not available when the test failure occurs.

jywarren · 2017-08-24T18:12:21Z

Our plan is to simply monitor publiclab.org/searches/test instead of having a fallback on production.

icarito · 2017-08-24T19:30:19Z

We could monitor pad.publiclab.org solr service directly too. Note also there are two kinds of failures: either Solr is (a) not listening or (b) not responding. Both will likely fail differently on the rails side.

…

On 24/08/17 13:12, Jeffrey Warren wrote: Our plan is to simply monitor publiclab.org/searches/test instead of having a fallback on production. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1537 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMMS9nJibXemPEsBVTdlEyRG5TOHABeks5sbb0FgaJpZM4OXif7>.

jywarren · 2017-08-24T19:51:41Z

Well, solrAvailable is evaluating to true, but not returning the blog nodes. Can we reproduce this in an interactive environment to see why?

icarito · 2017-08-24T20:10:52Z

Maybe it's a good opportunity to push to unstable branch and see the magic happen? ;-)

…

On 24/08/17 14:51, Jeffrey Warren wrote: Well, |solrAvailable| is evaluating to true, but not returning the blog nodes. Can we reproduce this in an interactive environment to see why? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1537 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMMS1bVGMchte_9LstgMzxQJ4BzVjprks5sbdRNgaJpZM4OXif7>.

jywarren · 2017-08-31T17:35:43Z

Yes, lets. We should be able to use /searches/test?q=blog and other queries to ensure Solr is actually working. for now, i'll also output a blank solr query to the log just to see.

jywarren · 2017-11-15T01:16:53Z

They should actually both be for Solr, and I'm wondering if the extra parameters makes it return nothing... we'll see -- if this passes, could you push it to unstable? Thanks!

icarito · 2017-11-15T01:20:53Z

I see what you mean! It might very well be, here's the logged query for both cases. I'll push to unstable when it passes.

solr_1  | 1549882 INFO  (qtp1112280004-15) [   x:default] o.a.s.c.S.Request [default] webapp=/solr path=/select params={q=about&defType=edismax&qf=title_text+body_text+comments_text&fl=*+score&start=0&fq=type:Node&fq=status_s:1&fq=type_s:note&sort=updated_at_s+desc&rows=5&wt=ruby} hits=2003 status=0 QTime=3 

solr_1  | 1582962 INFO  (qtp1112280004-19) [   x:default] o.a.s.c.S.Request [default] webapp=/solr path=/select params={q=test&defType=edismax&qf=title_text+body_text+comments_text&fl=*+score&start=0&fq=type:Node&rows=30&wt=ruby} hits=1960 status=0 QTime=1

jywarren · 2017-11-15T01:45:04Z

This is odd, and I'm restarting the build -- it looks to be stuck on rake db:setup -- may just be an intermittent Travis issue?

jywarren · 2017-11-15T01:58:31Z

Promising! It returned 2 results instead of 3 for a couple tests. I think that means the Solr query is working now; I adjusted the test assertion to match. Next I can re-enable some of the extra Solr parameters to see which caused the issue... hopefully there are, for example, pagination defaults we can lean on.

icarito · 2017-11-15T02:10:11Z

Pushed to unstable!

jywarren · 2017-11-15T06:59:15Z

Ug! Unstable still doesn't show results. @icarito, could you paste in the most recent solr queries from the logs and we can try to see why no results are returned?

jywarren · 2017-11-16T02:43:42Z

Huh, interesting! One where query works but not the other?

jywarren · 2017-11-16T18:15:01Z

OK, so the only line that makes the tests fail is with :type, "note" -- that's odd.

Oh, we don't index that!

https://github.com/publiclab/plots2/blob/master/app/models/node.rb#L24-L39

Changing that to see.

jywarren · 2017-11-16T18:15:53Z

Oh wait, that's weird - we do index that, at least in this branch. Never mind, I'm not sure what's going on here...

jywarren · 2017-11-16T22:29:55Z

@icarito -- finished this; pushing it to unstable too.

icarito · 2017-11-16T22:33:20Z

Already! Wow. Will reindex production then in preparation for deployment.

…

On 16/11/17 17:29, Jeffrey Warren wrote: @icarito <https://github.com/icarito> -- finished this; pushing it to unstable too. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1537 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMMSzpTuWL4Vr7kVBImWSj_8PrT03fPks5s3LdjgaJpZM4OXif7>.

jywarren · 2017-11-16T22:36:56Z

testing it in unstable now. If this passes, can you merge it and push it to stable? Thanks!

jywarren · 2017-11-16T22:37:35Z

It works in unstable:

jywarren · 2017-11-16T22:40:54Z

🍔 🔨 🍬 🔥 🎆

icarito · 2017-11-16T22:55:13Z

Reindexing production now in advance of deploying this.

icarito · 2017-11-16T22:57:12Z

Reindexing stable too.

icarito · 2017-11-16T23:37:44Z

Deployed to production!

icarito · 2017-11-16T23:37:56Z

https://publiclab.org/api/typeahead/all?srchString=precocious no results, not sure why

* Solr typeahead (#1537) * Solr typeahead May need a toggle, however. * Update typeahead_service.rb Not yet using limit... * Update typeahead_service.rb * Update node.rb * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead_service.rb * service reworking * toggle and test; very slow * note tweak * tweaks * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update and rename test/unit/typeahead_service_test.rb to test/solr/typeahead_service_test.rb * Update typeahead_service_test.rb * Create typeahead_service_test.rb * Update Gemfile * Delete typeahead_service_test.rb * Update typeahead_service_test.rb * Re-disable solr in test mode * Re-add regular test of typeahead and confirm toggle fallback * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update rusers.yml * Update typeahead_service.rb * Re-adding sunspot.yml overwriting * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Testing disabling extra Solr fields * Update typeahead_service_test.rb * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead.rb * Update restful_typeahead.js * new gemfile

* Solr typeahead May need a toggle, however. * Update typeahead_service.rb Not yet using limit... * Update typeahead_service.rb * Update node.rb * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead_service.rb * service reworking * toggle and test; very slow * note tweak * tweaks * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update and rename test/unit/typeahead_service_test.rb to test/solr/typeahead_service_test.rb * Update typeahead_service_test.rb * Create typeahead_service_test.rb * Update Gemfile * Delete typeahead_service_test.rb * Update typeahead_service_test.rb * Re-disable solr in test mode * Re-add regular test of typeahead and confirm toggle fallback * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update rusers.yml * Update typeahead_service.rb * Re-adding sunspot.yml overwriting * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Testing disabling extra Solr fields * Update typeahead_service_test.rb * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead.rb * Update restful_typeahead.js

* Solr typeahead (publiclab#1537) * Solr typeahead May need a toggle, however. * Update typeahead_service.rb Not yet using limit... * Update typeahead_service.rb * Update node.rb * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead_service.rb * service reworking * toggle and test; very slow * note tweak * tweaks * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update and rename test/unit/typeahead_service_test.rb to test/solr/typeahead_service_test.rb * Update typeahead_service_test.rb * Create typeahead_service_test.rb * Update Gemfile * Delete typeahead_service_test.rb * Update typeahead_service_test.rb * Re-disable solr in test mode * Re-add regular test of typeahead and confirm toggle fallback * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update rusers.yml * Update typeahead_service.rb * Re-adding sunspot.yml overwriting * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Update typeahead_service_test.rb * Testing disabling extra Solr fields * Update typeahead_service_test.rb * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead_service.rb * Update typeahead.rb * Update restful_typeahead.js * new gemfile

jywarren mentioned this pull request Jul 13, 2017

Planning issue for Search integration #1463

Closed

17 tasks

jywarren added 9 commits July 14, 2017 09:14

Solr typeahead

1a319d0

May need a toggle, however.

Update typeahead_service.rb

56ba4c9

Not yet using limit...

Update typeahead_service.rb

bcab5b7

Update node.rb

6c72fae

Update typeahead_service.rb

de0b670

Update typeahead_service.rb

0c5a3e2

Update typeahead_service.rb

aae8448

service reworking

ecab735

toggle and test; very slow

0aa6143

jywarren force-pushed the solr-typeahead branch from 5c3eae8 to 0aa6143 Compare July 14, 2017 13:40

note tweak

04dba2f

jywarren added 2 commits July 14, 2017 12:32

tweaks

7586ce9

Update typeahead_service.rb

c62ba11

jywarren added 2 commits August 24, 2017 14:05

Update typeahead_service.rb

87a1fdd

Update typeahead_service_test.rb

a4a272b

Update typeahead_service_test.rb

97a0575

jywarren added 2 commits November 15, 2017 09:15

Update typeahead_service.rb

4a17f13

Update typeahead_service.rb

948659d

Update typeahead_service.rb

c3dc223

jywarren added 2 commits November 16, 2017 13:18

Update typeahead.rb

a4468ad

Update restful_typeahead.js

643693e

jywarren merged commit eb2e7b6 into master Nov 16, 2017

jywarren mentioned this pull request Nov 17, 2017

Solr search performance issue with commit rate #1784

Closed

emilyashley deleted the solr-typeahead branch January 15, 2020 21:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Solr typeahead #1537

Solr typeahead #1537

jywarren commented Jul 13, 2017 •

edited

Loading

PublicLabBot commented Jul 13, 2017 •

edited

Loading

jywarren commented Jul 13, 2017

jywarren commented Jul 13, 2017

jywarren commented Jul 14, 2017

jywarren commented Jul 14, 2017

jywarren commented Jul 14, 2017

icarito commented Jul 17, 2017

icarito commented Aug 3, 2017

jywarren commented Aug 24, 2017

jywarren commented Aug 24, 2017

icarito commented Aug 24, 2017 via email

jywarren commented Aug 24, 2017

icarito commented Aug 24, 2017 via email

jywarren commented Aug 31, 2017

jywarren commented Nov 15, 2017

icarito commented Nov 15, 2017

jywarren commented Nov 15, 2017

jywarren commented Nov 15, 2017

icarito commented Nov 15, 2017

jywarren commented Nov 15, 2017

jywarren commented Nov 16, 2017

jywarren commented Nov 16, 2017

jywarren commented Nov 16, 2017

jywarren commented Nov 16, 2017

icarito commented Nov 16, 2017 via email

jywarren commented Nov 16, 2017

jywarren commented Nov 16, 2017

jywarren commented Nov 16, 2017

icarito commented Nov 16, 2017

icarito commented Nov 16, 2017

icarito commented Nov 16, 2017

icarito commented Nov 16, 2017

Solr typeahead #1537

Solr typeahead #1537

Conversation

jywarren commented Jul 13, 2017 • edited Loading

PublicLabBot commented Jul 13, 2017 • edited Loading

jywarren commented Jul 13, 2017

jywarren commented Jul 13, 2017

jywarren commented Jul 14, 2017

jywarren commented Jul 14, 2017

jywarren commented Jul 14, 2017

icarito commented Jul 17, 2017

icarito commented Aug 3, 2017

jywarren commented Aug 24, 2017

jywarren commented Aug 24, 2017

icarito commented Aug 24, 2017 via email

jywarren commented Aug 24, 2017

icarito commented Aug 24, 2017 via email

jywarren commented Aug 31, 2017

jywarren commented Nov 15, 2017

icarito commented Nov 15, 2017

jywarren commented Nov 15, 2017

jywarren commented Nov 15, 2017

icarito commented Nov 15, 2017

jywarren commented Nov 15, 2017

jywarren commented Nov 16, 2017

jywarren commented Nov 16, 2017

jywarren commented Nov 16, 2017

jywarren commented Nov 16, 2017

icarito commented Nov 16, 2017 via email

jywarren commented Nov 16, 2017

jywarren commented Nov 16, 2017

jywarren commented Nov 16, 2017

icarito commented Nov 16, 2017

icarito commented Nov 16, 2017

icarito commented Nov 16, 2017

icarito commented Nov 16, 2017

jywarren commented Jul 13, 2017 •

edited

Loading

PublicLabBot commented Jul 13, 2017 •

edited

Loading