Concurrency issue when WSGIDaemonProcess is set to multiprocess

[notanumber]

update, clear, remove, may cause concurrency issues with the xapian.WritableDatabase raising a DatabaseLockError.

This is due to multi-processes attempting to create write instances at the same time.

[notanumber]

This isn't as serious of an issue as I initially thought. This will only happen when WSGIDaemonProcess is set to use more than one process.

Until this can be fixed, it's better to leave the default of no multiprocessing. See here for more information: http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading

[notanumber]

Not sure this is really an issue that should be tackled at the backend level seeing as Haystack now supports QueueSearchIndex that would alleviate the need for this.

http://haystacksearch.org/docs/best_practices.html#use-of-a-queue-for-a-better-user-experience

Perhaps a note detailing how to implement a Xapian specific queue would be adequate.

[notanumber]

Updated README with details on how to avoid this situation.

Either set WSGIDaemonProcess processes=1 or override the default SearchIndex class to remove the post-save and post-delete signals on model changes.

[eaigner]

WSGIDaemonProcess processes=1 doesn't seem to work (at least for me). Still got the lock error on my server. On the local dev server it works just fine.

[notanumber]

Hmm. Can you confirm that there is only one wsgi process running by looking at ps or top? Besides that I'd suggest removing the contents of the index folder and reindexing to see if it happens again. Is it possible that you restarted the server while it was writing the indexes (maybe a model updated triggered an re-index)?

To be honest, you're not the first person to report this happening to them. My suggestion would be to go the cronjob indexing route and to remove the signals causing the automatic reindexing. I've opened a defect in the Haystack queue and attached a patch that would make this easier to do, but for now, you can use this gist http://gist.github.com/214254 to get an idea of how to do it, should you decide to go this route.

[notanumber]

The Haystack ticket I referred to is here: http://github.com/toastdriven/django-haystack/issues#issue/121. Feel free to comment on this one as well with your thoughts.

[eaigner]

Yep i made a ps and only one wsgi process is there. I think I'll go with the queue framework for now http://code.google.com/p/queues/ and setup a separate QueueSearchIndex.

[notanumber]

I'd very interested in hearing how that works out. I haven't had the chance to try it myself.