Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Re-checking/re-building all indices on restart? #179

Closed
andreyvk opened this Issue · 17 comments

2 participants

@andreyvk

Every time i restart couchdb-lucene and do a first query on it, it would start recalculating its indices. Why does it need to do it? Is there a way to avoid this behavior?

I am using couchdb 1.3.1 with couchdb-lucene 0.10.0

@rnewson
Owner

Hi,

couchdb-lucene is designed not to do that. In your couchdb.log, search for lines about writing to a document called "_local/lucene". Is is succeeding (status code 201)?

@andreyvk
@rnewson
Owner

ok, and you see _changes?since= calls too? What value is being passed to since= ?

@rnewson
Owner

fyi: I hang out on #couchdb on freenode IRC if that's easier.

@andreyvk

What's your nick there (just in case)?

This is the log it get right after restarting couchdb-lucene and making a simple query with a curl:

[Wed, 04 Sep 2013 09:27:33 GMT] [info] [<0.5920.2>] 127.0.0.1 - - GET /beta4_dev/ 200
[Wed, 04 Sep 2013 09:27:33 GMT] [info] [<0.5920.2>] 127.0.0.1 - - GET /beta4_dev/_local%2Flucene 200
[Wed, 04 Sep 2013 09:27:33 GMT] [info] [<0.5920.2>] 127.0.0.1 - - GET /beta4_dev/ 200
[Wed, 04 Sep 2013 09:27:33 GMT] [info] [<0.5920.2>] 127.0.0.1 - - GET /beta4_dev/_all_docs?startkey=%22_design%22&endkey=%22_design0%22&include_docs=true 200
[Wed, 04 Sep 2013 09:27:34 GMT] [info] [<0.5920.2>] 127.0.0.1 - - GET /beta4_dev/_changes?feed=continuous&heartbeat=15000&include_docs=true&since=813104 200
[Wed, 04 Sep 2013 09:27:34 GMT] [info] [<0.5923.2>] 127.0.0.1 - - GET /beta4_dev/_changes?limit=0&descending=true 200
[Wed, 04 Sep 2013 09:27:34 GMT] [info] [<0.5923.2>] 127.0.0.1 - - POST /beta4_dev/_all_docs?include_docs=true 200
[Wed, 04 Sep 2013 09:27:34 GMT] [info] [<0.5921.2>] 127.0.0.1 - - GET /beta4_dev/_fti/_design/lucene_contact/default?q=cuid:3968&include_docs=true 200

Here's also database info, if you need it:

{"db_name":"beta4_dev","doc_count":78328,"doc_del_count":351498,"update_seq":905973,"purge_seq":0,"compact_running":false,"disk_size":895225979,"data_size":588719940,"instance_start_time":"1378200484931844","disk_format_version":6,"committed_update_seq":905973}

@rnewson
Owner

'rnewson' :)

@rnewson
Owner

&since=813104 shows that c-l picked up from where it left off...

@andreyvk

I see now. But then according to database info current sequence is 905973. I've just stopped and restarted lucene several times. The 'since' value has increased, but it doesnt get up to the current sequence value. Is that normal?

Below are cut outs from log from several restarts:

[Wed, 04 Sep 2013 09:40:07 GMT] [info] [<0.5548.2>] 127.0.0.1 - - GET /beta4_dev/_changes?feed=continuous&heartbeat=15000&include_docs=true&since=818547 200

...

[Wed, 04 Sep 2013 09:41:30 GMT] [info] [<0.6765.2>] 127.0.0.1 - - GET /beta4_dev/_changes?feed=continuous&heartbeat=15000&include_docs=true&since=824126 200

...

[Wed, 04 Sep 2013 09:41:58 GMT] [info] [<0.6547.2>] 127.0.0.1 - - GET /beta4_dev/_changes?feed=continuous&heartbeat=15000&include_docs=true&since=831237 200

@andreyvk

PS - lucene has stopped reindexing each time i restarted.

@rnewson
Owner

It's normal not to "get up to the current sequence value" since that changes feed continues indefinitely, indexing changes as they are made to the database. You'll only see a new call if the connection is severed (typically because one service or the other is restarted).

Since it is going up, I suspect everything is fine, but couchdb-lucene isn't up long enough to catch up to all the changes. couchdb-lucene will commit the updates to the index periodically, so if you start c-l and promptly stop it, the index won't advance at all.

So, why are you restarting couchdb-lucene this often?

@andreyvk

I dont restart lucene. I just shut down my VM in the evening after finishing work and start it up in the morning. That's when I always notice that lucene is "trying" to catch up with the doc sequence. It's been like this since I've moved to 0.10.0 (for a few reasons I am not entirely sure, but i dont remember 0.9.0 having the same issue). After this everything is smooth and is running nicely until I start my VM in the morning again.

Btw, it's not just me, but a couple of my colleagues are also experiencing the same symptoms. We are all running couchdb 1.3.1 with couchdb-lucene 0.10.0 on Ubuntu server (12.04 and 11.10).

It's not big deal and it doesnt matter much on our production servers, because they are almost never restarted, but I just wanted to know if this is some sort of bug or not.

@rnewson
Owner

Ok, that could be a regression then, I'll look.

@rnewson
Owner

I will note that 0.10.0 isn't released, you're running the master/head revision there.

@andreyvk

Thanks. Yes, we pulled it from the head. Works great anyways. :)

@rnewson
Owner

I just checked, there's not a line of code difference between 0.9.0 and master right now. The only thing that changed is the README.md and the version number. So, 0.10.0-SNAPSHOT can't be behaving differently to 0.9.0...

@rnewson
Owner

hm, sorry, I can't reproduce this. I added docs, queried the index, saw the commit message ("View[name=_design/couchapp/all, digest=3yz0hiaucewo9svc1lklgb5ky] now at update_seq 28002"), stopped couchdb-lucene, started it again, queried it, and I immediately got a response (and a " Indexing from update_seq 28002" log message).

Please re-open if you can reproduce this. (I tested with master at 233442a but that's code-identical to 0.9.0).

@rnewson rnewson closed this
@andreyvk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.