* the ES connection already has code for forcing bulk, so we don't need to repeat that. this changes the code to push the setting to ES. * this also tweaks the estimation code so that it shows minutes and seconds and shows the total delta later. Now I can stop running "time ./manage.py esreindex". * fix esreindex so that you can specify doctypes. This will appropriately create/delete indexes so that what you don't want to delete won't get deleted. * adds basic handling for bad data. This does a log.exception, but we really should log more than that and/or make it more obvious to developers that there's bad data out there. In the meantime, this allows us to continue indexing. * reduced memory usage of indexing by iterating over ids---now it runs on my laptop. * ghanges _get_index() to get_es_index(). We use it so often it might as well be part of the "public API". * fixed create/delete indexes so that to switch doctypes to their own index is now just a change in settings---no code changes needed. * fix DEBUG = True case by reseting queries * this also adds a bunch of helpful comments, moves reindex_model to SearchMixin.index_all, and has some other cosmetic code cleanup. End result of this is that indexing doesn't die if it hits bad data, indexing takes much less memory to run, you can specify specific doctypes to index at the command line, and the code is better.
* New form for marketplace question flow. * Added zendesk lib to vendor. * Integrated to zendesk. * Linked to from marketplace landing page. * Tests.
…d, not just on the thread that imported the module.
… handling. This completely restores my motivation to work on search stuff. Ahhhhh. Make add_index_task and add_unindex_task into instance methods that do just a single document. We never used them for more than one, and it simplifies things.
This reduces the triplification of everything in es_search modules. This reduces the triplification of everything in es_search modules. This reduces the triplification of everything in es_search modules. Also, I introduced ES_INDEXING_TIMEOUT, fixed documentation, and cleaned up some other things while I was poking around. Also also, I nixed all the ES mapping constants. Calling put_mapping with typos will kick up an error, so there's no need to additionally have constants around to prevent typos.
Reindex questions and documents when tags are added and removed.
* New landing page at /marketplace * Added "marketplace" product for KB articles * Fixed base template not linking to right landing page
Incremental updates (not delete!) tasks all get tossed in a thread-local queue. At request_finished, there's a handler that creates tasks from the queued tasks. Because the queue is a set, this removes task duplicates so we're only creating one index task per request per document. This also fixes refresh calls so they pass timesleep=0--we don't need to wait for a second to update the index. Note: I didn't change deletes because they only get triggered by one model per document (e.g. Question model for question documents in the index) and therefore aren't going to hit the same issue we have with inserts/updates where in a single http request, we could trigger > 1 for a specific object.
It doesn't cost us significant performance, it'll go away eventually, and it saves repetition in test code.
…sier) calls to model constructors.
test_added was redundant with test_deleted, so I removed it. What was the uuid stuff for?
…remove dependency on fixtures. This way, we don't need an almost identical ESTestCase subclass in every app. Refactor questions tests to compensate.
Also kill useless fallthrough subclass of TestCase.