Es search view 839214

[groovecoder]

(Would like to get @lmorchard and/or @ubernostrum to think about my forcing the override of settings from test-utils)

Supersedes #898. The same testing instructions apply, plus some new ones:

You'll need to fetch my kuma-lib repos, which means the leeroy/mdn-github jenkins build is going to fail.

1. git submodule update --init --recursive
2. vagrant provision
3. Run tests. Make sure to include the 'search' app.

To spot-check, make sure you're running ./manage.py celeryd, then:

1. update a doc in the site UI
2. go to https://developer-local.allizom.org/elastic/_plugin/head/
3. connect it to https://developer-local.allizom.org/elastic/
4. go to the 'Browser' tab and make sure there's a record in the mdn-main_index for the doc

To spot-check the search view (wiring only, nothing fancy):

1. go to https://developer-local.allizom.org/admin/waffle/flag/
2. add flag 'elasticsearch' active for your user
3. go to https://developer-local.allizom.org/en-US/search

You should see a simple search result list matching the ES index.

[lmorchard]

Think I have time to peek at this right now...

[openjck]

Is this behind a feature flag, or will the view be entirely powered by Elastic Search once this lands? Just want to be sure we have high quality results before this becomes official, considering how important search is to the site.

[groovecoder]

Yup, there's an 'elasticsearch' feature flag on the view

[groovecoder]

There's also an ES_DISABLED setting that is set to True by default, which prevents any of the backend operations. We won't enable it in production until everything is configured; it looks like MDN already has flows to RabbitMQ on dev and to ES on dev, stage, and prod.

[lmorchard]

Attempt #2 at finding time to look at this. First thing, the "my forcing the override of settings from test-utils" link in the description leads to an attempt to create a new pull request (ie. pull/new). Not sure to what this was meant to link, but might be able to figure it out.

[lmorchard]

Okay, found the link for "my forcing the override of settings from test-utils", but not quite sure what's happening there. @ubernostrum might have something smarter to say about it, since I just basically stole the code from a django snippet and I'm not quite sure what UserSettingsHolder does.

[lmorchard]

Tests seem to pass ok:

vagrant@developer-local:/vagrant$ (cd /; /vagrant/manage.py test search wiki actioncounters contentflagging demos devmo landing users)
...
Ran 491 tests in 164.760s

OK (SKIP=47)

Moving on to spot checking after getting some lunch...

[lmorchard]

Working through spotcheck, getting an exception in kumascript execution that seems related to the Requests module upgrade in kuma-lib. Investigating how this might be happening.

[lmorchard]

Yeah, looks like the Requests API has changed significantly from v0.6.1 (what MDN uses now) to v1.1.0 (what your kuma-lib uses), and that breaks in kumascript.py.

The fix is probably just one line, but I'm not sure what else might break with the Requests upgrade - especially since we try to avoid doing real HTTP requests in tests that would use Requests.

[groovecoder]

Hmm ... that sounds familiar. I want to to say I fixed it already and maybe I have a kumascript commit to push for it?

[groovecoder]

Updated kumascript tests to match response api from new version of requests library. All tests passed for me!

[lmorchard]

So, tests all pass now, but:

• tests are still hammering elasticsearch with real requests like PUT /mdn-main_index/wiki_document/447 HTTP/1.1, and
• those tests are using "mdn-main_index"

Are these something that the test settings should be overriding?

[groovecoder]

ARGH! Forcing into settings.__dict__ is not so easy to disable. So any tests that run after the override are back to hitting the real index. :( Been banging on it for an hour or so now ... any ideas?

[groovecoder]

OMG finally fixed with help from @willkg ...

Ran 520 tests in 101.900s

OK (SKIP=66)

And only the task test hits the ES index.

[lmorchard]

none of the tests should hit a real es service though. there's no es on Jenkins and in general we need to mock out network services

[willkg]

Just as a thing, Jenkins (ci.mozilla.org) does have an ES. We use this for kitsune, fjord, amo, etc. We sort of think of this like we think of the db.

[lmorchard]

ah cool. then ignore my grumblings and I'll plan on giving this another look when I'm not on airport wifi

[darkwing]

Sport-check worked, tests passed for me.

[lmorchard]

Ack! This really needed to be squashed!

[lmorchard]

Also, did the kuma-lib change get merged too? Need to do that fast before things break