View Access via listener stuck indefinitely in queryDesignDoc.

[NitzDKoder]

Hi Folks,

1)App design #82 and CBL 1.4 used.

2. @

[09/20 14:58:37:250] [822] Listener: [REQUEST] (Acme.Utils.ThreadPool(2)-PooledThread: Acme.Serve.Serve$ServeConnection@75201cb) GET: https://localdomain:17393/data/_design%2FgetHistory/_view/getHistory?key=%22tel%3A%2B919888880010%3Bkn-corp-groups%3D16_1308%22&reduce=false&group=false

Request done.

3)@)[09/20 14:58:37:275] [822] View: Re-indexing view: getHistory/getHistory

https://github.com/couchbase/couchbase-lite-java-core/blob/1ae4c0d1873d1287c725406ceaf8035014fb4927/src/main/java/com/couchbase/lite/store/SQLiteViewStore.java#L247

Thread stuck here..

4)@

	Line 189: [09/20 14:58:48:564] [833] Listener: [REQUEST] (Acme.Utils.ThreadPool(4)-PooledThread: Acme.Serve.Serve$ServeConnection@62e4284) GET: https://localdomain:17393/data/_design%2FgetHistory/_view/getHistory?key=%22tel%3A%2B919888880010%3Bkn-corp-groups%3D16_1283%22&reduce=false&group=false
	Line 262: [09/20 14:58:56:842] [939] Listener: [REQUEST] (Acme.Utils.ThreadPool(6)-PooledThread: Acme.Serve.Serve$ServeConnection@681ca25) GET: https://localdomain:17405/data/_design%2FgetHistory/_view/getHistory?key=%22tel%3A%2B919888881013%22&reduce=false&group=false
	Line 398: [09/20 14:59:01:855] [945] Listener: [REQUEST] (Acme.Utils.ThreadPool(9)-PooledThread: Acme.Serve.Serve$ServeConnection@c4ca677) GET: https://localdomain:21569/data/_design%2FgetHistory/_view/getHistory?key=%22tel%3A%2B919888880010%3Bkn-corp-groups%3D16_1308%22&reduce=false&group=false
	Line 439: [09/20 14:59:03:162] [946] Listener: [REQUEST] (Acme.Utils.ThreadPool(10)-PooledThread: Acme.Serve.Serve$ServeConnection@9c8384e) GET: https://localdomain:21569/data/_design%2FgetHistory/_view/getHistory?key=%22tel%3A%2B919888880032%22&reduce=false&group=false
	Line 476: [09/20 14:59:03:539] [947] Listener: [REQUEST] (Acme.Utils.ThreadPool(11)-PooledThread: Acme.Serve.Serve$ServeConnection@e086268) GET: https://localdomain:21569/data/_design%2FgetHistory/_view/getHistory?key=%22tel%3A%2B919888880010%3Bkn-corp-groups%3D16_1894%22&reduce=false&group=false
	Line 515: [09/20 14:59:04:397] [948] Listener: [REQUEST] (Acme.Utils.ThreadPool(12)-PooledThread: Acme.Serve.Serve$ServeConnection@b182bb2) GET: https://localdomain:21569/data/_design%2FgetHistory/_view/getHistory?key=%22tel%3A%2B919888880010%3Bkn-corp-groups%3D16_1280%22&reduce=false&group=false
	Line 552: [09/20 14:59:04:753] [949] Listener: [REQUEST] (Acme.Utils.ThreadPool(13)-PooledThread: Acme.Serve.Serve$ServeConnection@6731aac) GET: https://localdomain:21569/data/_design%2FgetHistory/_view/getHistory?key=%22tel%3A%2B919888880010%3Bkn-corp-groups%3D16_1249%22&reduce=false&group=false

all the next request are stuck in queryDesignDoc function.

https://github.com/couchbase/couchbase-lite-java-core/blob/1.4.0/src/main/java/com/couchbase/lite/router/Router.java#L2524

	Line 153: [09/20 14:58:45:531] [810] Router:  queryDesignDoc Before updateIndex enteringThreadId : 810  enteringThreadName :Acme.Utils.ThreadPool(0)-PooledThread: Acme.Serve.Serve$ServeConnection@acaf3bb
	Line 203: [09/20 14:58:48:581] [833] Router:  queryDesignDoc Before updateIndex enteringThreadId : 833  enteringThreadName
	Line 276: [09/20 14:58:56:856] [939] Router:  queryDesignDoc Before updateIndex enteringThreadId : 939  enteringThreadName 
	Line 412: [09/20 14:59:01:868] [945] Router:  queryDesignDoc Before updateIndex enteringThreadId : 945  enteringThreadName :Acme.Utils.ThreadPool(9)-PooledThread: Acme.Serve.Serve$ServeConnection@c4ca677
	Line 453: [09/20 14:59:03:172] [946] Router:  queryDesignDoc Before updateIndex enteringThreadId : 946  enteringThreadName :Acme.Utils.ThreadPool(10)-PooledThread: Acme.Serve.Serve$ServeConnection@9c8384e
	Line 490: [09/20 14:59:03:549] [947] Router:  queryDesignDoc Before updateIndex enteringThreadId : 947  enteringThreadName :Acme.Utils.ThreadPool(11)-PooledThread: Acme.Serve.Serve$ServeConnection@e086268
	Line 529: [09/20 14:59:04:408] [948] Router:  queryDesignDoc Before updateIndex enteringThreadId : 948  enteringThreadName :Acme.Utils.ThreadPool(12)-PooledThread: Acme.Serve.Serve$ServeConnection@b182bb2
	Line 566: [09/20 14:59:04:764] [949] Router:  queryDesignDoc Before updateIndex enteringThreadId : 949  enteringThreadName :Acme.Utils.ThreadPool(13)-PooledThread: Acme.Serve.Serve$ServeConnection@6731aac

Detail logs are in below:

stuck.txt

Thanks
Nithin.

[hideki]

@NitzDKoder,

Thank you for your inputs.

1. I'd like to confirm if one thread obtains View instance as the synchronized lock by the thread-dump.
2. If No. 1 is yes, I'd like to identify why View.updateIndex() or View.getLastSequenceIndexed() is stuck. For example, updateIndex() is pretty complex code. Which part of updateIndex() is stuck. Usually, SQLite method does not respond. To identify this, we need further help from you.

Thanks,
Hideki

[NitzDKoder]

@hideki tried taking thread-dump as suggest but failed , the app restarted. will try getting one.

when the issue happens just hit Debug --> Debugger --> Threads -->capture. Right?

https://github.com/couchbase/couchbase-lite-java-core/blob/1.4.0/src/main/java/com/couchbase/lite/router/Router.java#L2495

"view" variable is local to this function right..different threads can access in parallel right?

Can you please confirm on below?

When are these synchronised added in queryDesignDoc function? Dont see in the branch code.

https://github.com/couchbase/couchbase-lite-java-core/blob/feature/issue_855_fetchRemoteCheckpointDoc/src/main/java/com/couchbase/lite/router/Router.java#L2035

Thanks
Nithin

[hideki]

when the issue happens just hit Debug --> Debugger --> Threads -->capture. Right?

yes

"view" variable is local to this function right..different threads can access in parallel right?

No. view is from Database. so the view instance could be referenced from other methods with other threads.

When are these synchronised added in queryDesignDoc function? Dont see in the branch code.

This branch is pretty old. We added many fixes for multi-threading issues since then.

[NitzDKoder]

" view is from Database. so the view instance could be referenced from other methods threads."
Ok
So if View1 instance enters to Re-indexing it should return so that next same View1 can enter right..

We check why and where this stuck
[09/20 14:58:37:275] [822] View: Re-indexing view: getHistory/getHistory

[hideki]

HI @NitzDKoder
Do you have any updates for this ticket?

[hideki]

Hi @NitzDKoder,
It seems this is duplicate with #84 . I will close this ticket, and we keep track this issue with #84.