2.0: Replication of 1M documents never completes and high CPU/memory usage #1970

rajagp · 2017-12-01T17:28:20Z

Version: DB 19?
Client OS: Swift

From forums https://forums.couchbase.com/t/couchbaselite-2-0-swift-and-gateway-1-5-slow-pulling/14902/2

pasin · 2018-01-19T06:09:19Z

@raghusarangapani and @sridevi-15 Do we have similar functional test to this?

raghusarangapani · 2018-01-19T06:13:53Z

@hrajput89 and @sridevi-15 were working on setting up a system test with 1 million docs. They might be able to answer more.

sridevi-15 · 2018-01-19T15:07:00Z

we have system test, just that it did not run with 1 Million yet due to other priorities.

snej · 2018-01-19T18:25:46Z

I have some large databases on S3, including a 20GB subset of Wikipedia, but I'm not sure if any have 1M documents.

hrajput89 · 2018-01-23T09:33:38Z

My application crashed with below error -

E/art: Unexpected time out during dump checkpoint.
W/art: Attempted to destroy barrier with non zero count 1
A/art: art/runtime/runtime.cc:422] Runtime aborting...
A/art: art/runtime/runtime.cc:422] Aborting thread:
A/art: art/runtime/runtime.cc:422] "Thread-46019" prio=10 tid=18 Runnable
A/art: art/runtime/runtime.cc:422]   | group="" sCount=0 dsCount=0 obj=0x13eeb430 self=0xc1cdac00
A/art: art/runtime/runtime.cc:422]   | sysTid=17434 nice=-10 cgrp=default sched=0/0 handle=0xd72a7920
A/art: art/runtime/runtime.cc:422]   | state=R schedstat=( 30383575822 14307169963 56030 ) utm=745 stm=2292 core=2 HZ=100
A/art: art/runtime/runtime.cc:422]   | stack=0xd71ab000-0xd71ad000 stackSize=1014KB
A/art: art/runtime/runtime.cc:422]   | held mutexes= "abort lock" "JNI weak global reference table lock" "ObjectRegistry lock" "mutator lock"(shared held)
A/art: art/runtime/runtime.cc:422]   native: #00 pc 0057302e  /system/lib/libart.so (_ZN3art15DumpNativeStackERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEEiP12BacktraceMapPKcPNS_9ArtMethodEPv+238)
A/art: art/runtime/runtime.cc:422]   native: #01 pc 0053f40e  /system/lib/libart.so (_ZNK3art6Thread9DumpStackERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEEbP12BacktraceMap+526)
A/art: art/runtime/runtime.cc:422]   native: #02 pc 0053c40b  /system/lib/libart.so (_ZNK3art6Thread4DumpERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEEbP12BacktraceMap+75)
A/art: art/runtime/runtime.cc:422]   native: #03 pc 00528013  /system/lib/libart.so (_ZNK3art10AbortState10DumpThreadERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEEPNS_6ThreadE+67)
A/art: art/runtime/runtime.cc:422]   native: #04 pc 00527cca  /system/lib/libart.so (_ZNK3art10AbortState4DumpERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEE+330)
A/art: art/runtime/runtime.cc:422]   native: #05 pc 0051832b  /system/lib/libart.so (_ZN3art7Runtime5AbortEPKc+155)
A/art: art/runtime/runtime.cc:422]   native: #06 pc 0011a653  /system/lib/libart.so (_ZN3art10LogMessageD1Ev+1747)
A/art: art/runtime/runtime.cc:422]   native: #07 pc 002cd248  /system/lib/libart.so (_ZN3art22IndirectReferenceTable3AddEjPNS_6mirror6ObjectE+376)
A/art: art/runtime/runtime.cc:422]   native: #08 pc 003932eb  /system/lib/libart.so (_ZN3art9JavaVMExt16AddWeakGlobalRefEPNS_6ThreadEPNS_6mirror6ObjectE+91)
A/art: art/runtime/runtime.cc:422]   native: #09 pc 00403c14  /system/lib/libart.so (_ZN3art3JNI16NewWeakGlobalRefEP7_JNIEnvP8_jobject+788)
A/art: art/runtime/runtime.cc:422]   native: #10 pc 0013f9f2  /system/lib/libart.so (_ZN3art8CheckJNI6NewRefEPKcP7_JNIEnvP8_jobjectNS_15IndirectRefKindE+1106)
A/art: art/runtime/runtime.cc:422]   native: #11 pc 00139769  /system/lib/libart.so (_ZN3art8CheckJNI16NewWeakGlobalRefEP7_JNIEnvP8_jobject+57)
A/art: art/runtime/runtime.cc:422]   native: #12 pc 003b1d9e  /system/lib/libart.so (_ZN3art14ObjectRegistry11InternalAddINS_6mirror6ObjectEEEyNS_6HandleIT_EE+1358)
A/art: art/runtime/runtime.cc:422]   native: #13 pc 003b2776  /system/lib/libart.so (_ZN3art14ObjectRegistry3AddEPNS_6mirror6ObjectE+166)
A/art: art/runtime/runtime.cc:422]   native: #14 pc 001bc593  /system/lib/libart.so (_ZN3art3Dbg11GetThreadIdEPNS_6ThreadE+691)
A/art: art/runtime/runtime.cc:422]   native: #15 pc 0039f864  /system/lib/libart.so (_ZN3art4JDWP9JdwpState16PostThreadChangeEPNS_6ThreadEb+948)
A/art: art/runtime/runtime.cc:422]   native: #16 pc 001ce4ee  /system/lib/libart.so (_ZN3art3Dbg15PostThreadStartEPNS_6ThreadE+62)
A/art: art/runtime/runtime.cc:422]   native: #17 pc 0053d513  /system/lib/libart.so (_ZN3art6Thread6AttachEPKcbP8_jobjectb+2595)
A/art: art/runtime/runtime.cc:422]   native: #18 pc 00517b5f  /system/lib/libart.so (_ZN3art7Runtime19AttachCurrentThreadEPKcbP8_jobjectb+95)
A/art: art/runtime/runtime.cc:422]   native: #19 pc 003980a2  /system/lib/libart.so (_ZN3art3JII27AttachCurrentThreadInternalEP7_JavaVMPP7_JNIEnvPvb+338)
A/art: art/runtime/runtime.cc:422]   native: #20 pc 00397d86  /system/lib/libart.so (_ZN3art3JII19AttachCurrentThreadEP7_JavaVMPP7_JNIEnvPv+54)
A/art: art/runtime/runtime.cc:422]   native: #21 pc 00149113  /system/lib/libart.so (_ZN3art8CheckJII19AttachCurrentThreadEP7_JavaVMPP7_JNIEnvPv+147)
A/art: art/runtime/runtime.cc:422]   native: #22 pc 000c4f91  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (???)
A/art: art/runtime/runtime.cc:422]   native: #23 pc 000f0a25  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZN12C4Replicator23replicatorStatusChangedEPN8litecore4repl10ReplicatorERKNS1_6Worker6StatusE+261)
A/art: art/runtime/runtime.cc:422]   native: #24 pc 000f0c6a  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZThn8_N12C4Replicator23replicatorStatusChangedEPN8litecore4repl10ReplicatorERKNS1_6Worker6StatusE+42)
A/art: art/runtime/runtime.cc:422]   native: #25 pc 001252c3  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZN8litecore4repl10Replicator12reportStatusEv+259)
A/art: art/runtime/runtime.cc:422]   native: #26 pc 000f0f80  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZNSt6__ndk110__function6__funcINS_6__bindIRMN8litecore4repl10ReplicatorEFvvEJPS5_EEENS_9allocatorISA_EEFvvEEclEv+48)
A/art: art/runtime/runtime.cc:422]   native: #27 pc 001883e8  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZN8litecore5actor15ThreadedMailbox18performNextMessageEv+136)
A/art: art/runtime/runtime.cc:422]   native: #28 pc 001882ec  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZN8litecore5actor9Scheduler4taskEj+204)
A/art: art/runtime/runtime.cc:422]   native: #29 pc 001891a6  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (???)
A/art: art/runtime/runtime.cc:422]   native: #30 pc 000752b2  /system/lib/libc.so (_ZL15__pthread_startPv+210)
A/art: art/runtime/runtime.cc:422]   native: #31 pc 000202ae  /system/lib/libc.so (__start_thread+30)
A/art: art/runtime/runtime.cc:422]   native: #32 pc 0001e086  /system/lib/libc.so (__bionic_clone+70)
A/art: art/runtime/runtime.cc:422]   (no managed stack frames)
A/art: art/runtime/runtime.cc:422] Dumping all threads without appropriate locks held: thread list lock
A/art: art/runtime/runtime.cc:422] All threads:
A/art: art/runtime/runtime.cc:422] DALVIK THREADS (19):
A/art: art/runtime/runtime.cc:422] "Thread-46019" prio=10 tid=18 Runnable
A/art: art/runtime/runtime.cc:422]   | group="" sCount=0 dsCount=0 obj=0x13eeb430 self=0xc1cdac00
A/art: art/runtime/runtime.cc:422]   | sysTid=17434 nice=-10 cgrp=default sched=0/0 handle=0xd72a7920
A/art: art/runtime/runtime.cc:422]   | state=R schedstat=( 30397130165 14308076891 56039 ) utm=746 stm=2292 core=2 HZ=100
A/art: art/runtime/runtime.cc:422]   | stack=0xd71ab000-0xd71ad000 stackSize=1014KB
A/art: art/runtime/runtime.cc:422]   | held mutexes= "abort lock" "JNI weak global reference table lock" "ObjectRegistry lock" "mutator lock"(shared held)
A/art: art/runtime/runtime.cc:422]   native: #00 pc 0057302e  /system/lib/libart.so (_ZN3art15DumpNativeStackERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEEiP12BacktraceMapPKcPNS_9ArtMethodEPv+238)
A/art: art/runtime/runtime.cc:422]   native: #01 pc 0053f40e  /system/lib/libart.so (_ZNK3art6Thread9DumpStackERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEEbP12BacktraceMap+526)
A/art: art/runtime/runtime.cc:422]   native: #02 pc 0053c40b  /system/lib/libart.so (_ZNK3art6Thread4DumpERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEEbP12BacktraceMap+75)
A/art: art/runtime/runtime.cc:422]   native: #03 pc 0055c00b  /system/lib/libart.so (_ZN3art14DumpCheckpoint3RunEPNS_6ThreadE+1115)
A/art: art/runtime/runtime.cc:422]   native: #04 pc 005522ce  /system/lib/libart.so (_ZN3art10ThreadList13RunCheckpointEPNS_7ClosureE+590)
A/art: art/runtime/runtime.cc:422]   native: #05 pc 00551e42  /system/lib/libart.so (_ZN3art10ThreadList4DumpERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEEb+962)
A/art: art/runtime/runtime.cc:422]   native: #06 pc 00528268  /system/lib/libart.so (_ZNK3art10AbortState14DumpAllThreadsERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEEPNS_6ThreadE+424)
A/art: art/runtime/runtime.cc:422]   native: #07 pc 00527fb6  /system/lib/libart.so (_ZNK3art10AbortState4DumpERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEE+1078)
A/art: art/runtime/runtime.cc:422]   native: #08 pc 0051832b  /system/lib/libart.so (_ZN3art7Runtime5AbortEPKc+155)
A/art: art/runtime/runtime.cc:422]   native: #09 pc 0011a653  /system/lib/libart.so (_ZN3art10LogMessageD1Ev+1747)
A/art: art/runtime/runtime.cc:422]   native: #10 pc 002cd248  /system/lib/libart.so (_ZN3art22IndirectReferenceTable3AddEjPNS_6mirror6ObjectE+376)
A/art: art/runtime/runtime.cc:422]   native: #11 pc 003932eb  /system/lib/libart.so (_ZN3art9JavaVMExt16AddWeakGlobalRefEPNS_6ThreadEPNS_6mirror6ObjectE+91)
A/art: art/runtime/runtime.cc:422]   native: #12 pc 00403c14  /system/lib/libart.so (_ZN3art3JNI16NewWeakGlobalRefEP7_JNIEnvP8_jobject+788)
A/art: art/runtime/runtime.cc:422]   native: #13 pc 0013f9f2  /system/lib/libart.so (_ZN3art8CheckJNI6NewRefEPKcP7_JNIEnvP8_jobjectNS_15IndirectRefKindE+1106)
A/art: art/runtime/runtime.cc:422]   native: #14 pc 00139769  /system/lib/libart.so (_ZN3art8CheckJNI16NewWeakGlobalRefEP7_JNIEnvP8_jobject+57)
A/art: art/runtime/runtime.cc:422]   native: #15 pc 003b1d9e  /system/lib/libart.so (_ZN3art14ObjectRegistry11InternalAddINS_6mirror6ObjectEEEyNS_6HandleIT_EE+1358)
A/art: art/runtime/runtime.cc:422]   native: #16 pc 003b2776  /system/lib/libart.so (_ZN3art14ObjectRegistry3AddEPNS_6mirror6ObjectE+166)
A/art: art/runtime/runtime.cc:422]   native: #17 pc 001bc593  /system/lib/libart.so (_ZN3art3Dbg11GetThreadIdEPNS_6ThreadE+691)
A/art: art/runtime/runtime.cc:422]   native: #18 pc 0039f864  /system/lib/libart.so (_ZN3art4JDWP9JdwpState16PostThreadChangeEPNS_6ThreadEb+948)
A/art: art/runtime/runtime.cc:422]   native: #19 pc 001ce4ee  /system/lib/libart.so (_ZN3art3Dbg15PostThreadStartEPNS_6ThreadE+62)
A/art: art/runtime/runtime.cc:422]   native: #20 pc 0053d513  /system/lib/libart.so (_ZN3art6Thread6AttachEPKcbP8_jobjectb+2595)
A/art: art/runtime/runtime.cc:422]   native: #21 pc 00517b5f  /system/lib/libart.so (_ZN3art7Runtime19AttachCurrentThreadEPKcbP8_jobjectb+95)
A/art: art/runtime/runtime.cc:422]   native: #22 pc 003980a2  /system/lib/libart.so (_ZN3art3JII27AttachCurrentThreadInternalEP7_JavaVMPP7_JNIEnvPvb+338)
A/art: art/runtime/runtime.cc:422]   native: #23 pc 00397d86  /system/lib/libart.so (_ZN3art3JII19AttachCurrentThreadEP7_JavaVMPP7_JNIEnvPv+54)
A/art: art/runtime/runtime.cc:422]   native: #24 pc 00149113  /system/lib/libart.so (_ZN3art8CheckJII19AttachCurrentThreadEP7_JavaVMPP7_JNIEnvPv+147)
A/art: art/runtime/runtime.cc:422]   native: #25 pc 000c4f91  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (???)
A/art: art/runtime/runtime.cc:422]   native: #26 pc 000f0a25  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZN12C4Replicator23replicatorStatusChangedEPN8litecore4repl10ReplicatorERKNS1_6Worker6StatusE+261)
A/art: art/runtime/runtime.cc:422]   native: #27 pc 000f0c6a  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZThn8_N12C4Replicator23replicatorStatusChangedEPN8litecore4repl10ReplicatorERKNS1_6Worker6StatusE+42)
A/art: art/runtime/runtime.cc:422]   native: #28 pc 001252c3  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZN8litecore4repl10Replicator12reportStatusEv+259)
A/art: art/runtime/runtime.cc:422]   native: #29 pc 000f0f80  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZNSt6__ndk110__function6__funcINS_6__bindIRMN8litecore4repl10ReplicatorEFvvEJPS5_EEENS_9allocatorISA_EEFvvEEclEv+48)
A/art: art/runtime/runtime.cc:422]   native: #30 pc 001883e8  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZN8litecore5actor15ThreadedMailbox18performNextMessageEv+136)
A/art: art/runtime/runtime.cc:422]   native: #31 pc 001882ec  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (_ZN8litecore5actor9Scheduler4taskEj+204)
A/art: art/runtime/runtime.cc:422]   native: #32 pc 001891a6  /data/app/com.couchbase.androidclient-1/lib/x86/libLiteCoreJNI.so (???)
A/art: art/runtime/runtime.cc:422]   native: #33 pc 000752b2  /system/lib/libc.so (_ZL15__pthread_startPv+210)
A/art: art/runtime/runtime.cc:422]   native: #34 pc 000202ae  /system/lib/libc.so (__start_thread+30)
A/art: art/runtime/runtime.cc:422]   native: #35 pc 0001e086  /system/lib/libc.so (__bionic_clone+70)
A/art: art/runtime/runtime.cc:422]   (no managed stack frames)
A/art: art/runtime/runtime.cc:422] 
A/art: art/runtime/runtime.cc:422] "main" prio=10 tid=1 Suspended
A/art: art/runtime/runtime.cc:422]   | group="" sCount=2 dsCount=1 obj=0x74de0610 self=0xf090b400
A/art: art/runtime/runtime.cc:422]   | sysTid=17330 nice=-10 cgrp=default sched=0/0 handle=0xf527d534
A/art: art/runtime/runtime.cc:422]   | state=S schedstat=( 85358600918 443731456 8467 ) utm=8196 stm=339 core=3 HZ=100
A/art: art/runtime/runtime.cc:422]   | stack=0xff4e0000-0xff4e2000 stackSize=8MB
A/art: art/runtime/runtime.cc:422]   | held mutexes=
A/art: art/runtime/runtime.cc:422]   kernel: futex_wait_queue_me+0xc0/0x110
A/art: art/runtime/runtime.cc:422]   kernel: futex_wait+0x112/0x250
A/art: art/runtime/runtime.cc:422]   kernel: do_futex+0xdc/0xb30
A/art: art/runtime/runtime.cc:422]   kernel: compat_SyS_futex+0x75/0x150
A/art: art/runtime/runtime.cc:422]   kernel: do_fast_syscall_32+0xa4/0x250
A/art: art/runtime/runtime.cc:422]   kernel: sysenter_flags_fixed+0x8/0x12
A/art: art/runtime/runtime.cc:422]   native: #00 pc 00000bb7  [vdso] (__kernel_vsyscall+7)
A/art: art/runtime/runtime.cc:422]   native: #01 pc 0001e268  /system/lib/libc.so (syscall+40)
A/art: art/runtime/runtime.cc:422]   native: #02 pc 0011d27e  /system/lib/libart.so (_ZN3art17ConditionVariable16WaitHoldingLocksEPNS_6ThreadE+142)
A/art: art/runtime/runtime.cc:422]   native: #03 pc 0011d1e7  /system/lib/libart.so (_ZN3art17ConditionVariable4WaitEPNS_6ThreadE+39)
A/art: art/runtime/runtime.cc:422]   native: #04 pc 00558f9d  /system/lib/libart.so (_ZN3art10ThreadList22SuspendSelfForDebuggerEv+589)
A/art: art/runtime/runtime.cc:422]   native: #05 pc 001bceeb  /system/lib/libart.so (_ZN3art3Dbg11SuspendSelfEv+43)
A/art: art/runtime/runtime.cc:422]   native: #06 pc 0039a96c  /system/lib/libart.so (_ZN3art4JDWP9JdwpState15SuspendByPolicyENS0_17JdwpSuspendPolicyEy+252)
A/art: art/runtime/runtime.cc:422]   native: #07 pc 0039aec3  /system/lib/libart.so (_ZN3art4JDWP9JdwpState29SendRequestAndPossiblySuspendEPNS0_9ExpandBufENS0_17JdwpSuspendPolicyEy+499)
A/art: art/runtime/runtime.cc:422]   native: #08 pc 0039da42  /system/lib/libart.so (_ZN3art4JDWP9JdwpState17PostLocationEventEPKNS0_13EventLocationEPNS_6mirror6ObjectEiPKNS_6JValueE+1810)
A/art: art/runtime/runtime.cc:422]   native: #09 pc 001c1e76  /system/lib/libart.so (_ZN3art3Dbg17PostLocationEventEPNS_9ArtMethodEiPNS_6mirror6ObjectEiPKNS_6JValueE+278)
A/art: art/runtime/runtime.cc:422]   native: #10 pc 001c353f  /system/lib/libart.so (_ZN3art3Dbg14UpdateDebuggerEPNS_6ThreadEPNS_6mirror6ObjectEPNS_9ArtMethodEjiPKNS_6JValueE+2559)
A/art: art/runtime/runtime.cc:422]   native: #11 pc 001d13f1  /system/lib/libart.so (_ZN3art28DebugInstrumentationListener10DexPcMovedEPNS_6ThreadEPNS_6mirror6ObjectEPNS_9ArtMethodEj+129)
A/art: art/runtime/runtime.cc:422]   native: #12 pc 002d26f1  /system/lib/libart.so (_ZNK3art15instrumentation15Instrumentation19DexPcMovedEventImplEPNS_6ThreadEPNS_6mirror6ObjectEPNS_9ArtMethodEj+97)
A/art: art/runtime/runtime.cc:422]   native: #13 pc 0034b0f2  /system/lib/libart.so (_ZN3art11interpreter17ExecuteSwitchImplILb0ELb0EEENS_6JValueEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameES2_b+64610)
A/art: art/runtime/runtime.cc:422]   native: #14 pc 002e601c  /system/lib/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameENS_6JValueEb+700)
A/art: art/runtime/runtime.cc:422]   native: #15 pc 002ed790  /system/lib/libart.so (_ZN3art11interpreter33ArtInterpreterToInterpreterBridgeEPNS_6ThreadEPKNS_7DexFile8CodeItemEPNS_11ShadowFrameEPNS_6JValueE+192)
A/art: art/runtime/runtime.cc:422]   native: #16 pc 003154cb  /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+587)
A/art: art/runtime/runtime.cc:422]   native: #17 pc 0034ec8e  /system/lib/libart.so (_ZN3art11interpreterL8DoInvokeILNS_10InvokeTypeE2ELb0ELb0EEEbPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+734)
A/art: art/runtime/runtime.cc:422]   native: #18 pc 00340115  /system/lib/libart.so (_ZN3art11interpreter17ExecuteSwitchImplILb0ELb0EEENS_6JValueEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameES2_b+19589)
A/art: art/runtime/runtime.cc:422]   native: #19 pc 002e601c  /system/lib/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameENS_6JValueEb+700)
A/art: art/runtime/runtime.cc:422]   native: #20 pc 002ed790  /system/lib/libart.so (_ZN3art11interpreter33ArtInterpreterToInterpreterBridgeEPNS_6ThreadEPKNS_7DexFile8CodeItemEPNS_11ShadowFrameEPNS_6JValueE+192)
A/art: art/runtime/runtime.cc:422]   native: #21 pc 003154cb  /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+587)
A/art: art/runtime/runtime.cc:422]   native: #22 pc 0034ec8e  /system/lib/libart.so (_ZN3art11interpreterL8DoInvokeILNS_10InvokeTypeE2ELb0ELb0EEEbPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+734)
A/art: art/runtime/runtime.cc:422]   native: #23 pc 00340115  /system/lib/libart.so (_ZN3art11interpreter17ExecuteSwitchImplILb0ELb0EEENS_6JValueEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameES2_b+19589)
A/art: art/runtime/runtime.cc:422]   native: #24 pc 002e601c  /system/lib/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameENS_6JValueEb+700)
A/art: art/runtime/runtime.cc:422]   native: #25 pc 002ed790  /system/lib/libart.so (_ZN3art11interpreter33ArtInterpreterToInterpreterBridgeEPNS_6ThreadEPKNS_7DexFile8CodeItemEPNS_11ShadowFrameEPNS_6JValueE+192)
A/art: art/runtime/runtime.cc:422]   native: #26 pc 003154cb  /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+587)
A/art: art/runtime/runtime.cc:422]   native: #27 pc 0034ec8e  /system/lib/libart.so (_ZN3art11interpreterL8DoInvokeILNS_10InvokeTypeE2ELb0ELb0EEEbPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+734)
A/art: art/runtime/runtime.cc:422]   native: #28 pc 00340115  /system/lib/libart.so (_ZN3art11interpreter17ExecuteSwitchImplILb0ELb0EEENS_6JValueEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameES2_b+19589)
A/art: art/runtime/runtime.cc:422]   native: #29 pc 002e601c  /system/lib/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameENS_6JValueEb+700)
A/art: art/runtime/runtime.cc:422]   native: #30 pc 002ed790  /system/lib/libart.so (_ZN3art11interpreter33ArtInterpreterToInterpreterBridgeEPNS_6ThreadEPKNS_7DexFile8CodeItemEPNS_11ShadowFrameEPNS_6JValueE+192)
A/art: art/runtime/runtime.cc:422]   native: #31 pc 003154cb  /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+587)
A/art: art/runtime/runtime.cc:422]   native: #32 pc 00350f5d  /system/lib/libart.so (_ZN3art11interpreterL8DoInvokeILNS_10InvokeTypeE1ELb0ELb0EEEbPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+461)
A/art: art/runtime/runtime.cc:422]   native: #33 pc 003411f1  /system/lib/libart.so (_ZN3art11interpreter17ExecuteSwitchImplILb0ELb0EEENS_6JValueEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameES2_b+23905)
A/art: art/runtime/runtime.cc:422]   native: #34 pc 002e601c  /system/lib/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameENS_6JValueEb+700)
A/art: art/runtime/runtime.cc:422]   native: #35 pc 002ed790  /system/lib/libart.so (_ZN3art11interpreter33ArtInterpreterToInterpreterBridgeEPNS_6ThreadEPKNS_7DexFile8CodeItemEPNS_11ShadowFrameEPNS_6JValueE+192)
A/art: art/runtime/runtime.cc:422]   native: #36 pc 003154cb  /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+587)
A/art: art/runtime/runtime.cc:422]   native: #37 pc 00350f5d  /system/lib/libart.so (_ZN3art11interpreterL8DoInvokeILNS_10InvokeTypeE1ELb0ELb0EEEbPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+461)
A/art: art/runtime/runtime.cc:422]   native: #38 pc 003411f1  /system/lib/libart.so (_ZN3art11interpreter17ExecuteSwitchImplILb0ELb0EEENS_6JValueEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameES2_b+23905)
A/art: art/runtime/runtime.cc:422]   native: #39 pc 002e601c  /system/lib/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameENS_6JValueEb+700)
A/art: art/runtime/runtime.cc:422]   native: #40 pc 002ed790  /system/lib/libart.so (_ZN3art11interpreter33ArtInterpreterToInterpreterBridgeEPNS_6ThreadEPKNS_7DexFile8CodeItemEPNS_11ShadowFrameEPNS_6JValueE+192)
A/art: art/runtime/runtime.cc:422]   native: #41 pc 003154cb  /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+587)
A/art: art/runtime/runtime.cc:422]   native: #42 pc 00352778  /system/lib/libart.so (_ZN3art11interpreterL8DoInvokeILNS_10InvokeTypeE0ELb0ELb0EEEbPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+424)
A/art: art/runtime/runtime.cc:422]   native: #43 pc 0034391c  /system/lib/libart.so (_ZN3art11interpreter17ExecuteSwitchImplILb0ELb0EEENS_6JValueEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameES2_b+33932)
A/art: art/runtime/runtime.cc:422]   native: #44 pc 002e601c  /system/lib/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameENS_6JValueEb+700)
A/art: art/runtime/runtime.cc:422]   native: #45 pc 002ed790  /system/lib/libart.so (_ZN3art11interpreter33ArtInterpreterToInterpreterBridgeEPNS_6ThreadEPKNS_7DexFile8CodeItemEPNS_11ShadowFrameEPNS_6JValueE+192)
A/art: art/runtime/runtime.cc:422]   native: #46 pc 003154cb  /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+587)
A/art: art/runtime/runtime.cc:422]   native: #47 pc 0034ec8e  /system/lib/libart.so (_ZN3art11interpreterL8DoInvokeILNS_10InvokeTypeE2ELb0ELb0EEEbPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+734)
A/art: art/runtime/runtime.cc:422]   native: #48 pc 00340115  /system/lib/libart.so (_ZN3art11interpreter17ExecuteSwitchImplILb0ELb0EEENS_6JValueEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameES2_b+19589)
A/art: art/runtime/runtime.cc:422]   native: #49 pc 002e601c  /system/lib/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadEPKNS_7DexFile8CodeItemERNS_11ShadowFrameENS_6JValueEb+700)
A/art: art/runtime/runtime.cc:422]   native: #50 pc 002ed6a0  /system/lib/libart.so (_ZN3art11interpreter30EnterInterpreterFromEntryPointEPNS_6ThreadEPKNS_7DexFile8CodeItemEPNS_11ShadowFrameE+176)
A/art: art/runtime/runtime.cc:422]   native: #51 pc 0066d199  /system/lib/libart.so (artQuickToInterpreterBridge+1049)
A/art: art/runtime/runtime.cc:422]   native: #52 pc 0010f39d  /system/lib/libart.so (art_quick_to_interpreter_bridge+77)
A/art: art/runtime/runtime.cc:422]   native: #53 pc 007add0a  /system/framework/x86/boot-framework.oat (???)
A/art: art/runtime/runtime.cc:422]   at com.couchbase.androidclient.MainActivity.onStart(MainActivity.java:115)
A/art: art/runtime/runtime.cc:422]   at android.app.Instrumentation.callActivityOnStart(Instrumentation.java:1248)
A/art: art/runtime/runtime.cc:422]   at android.app.Activity.performStart(Activity.java:6696)
A/art: art/runtime/runtime.cc:422]   at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2628)
A/art: art/runtime/runtime.cc:422]   at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2726)
A/art: art/runtime/runtime.cc:422]   at android.app.ActivityThread.-wrap12(ActivityThread.java:-1)
A/art: art/runtime/runtime.cc:422]   at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1477)
A/art: art/runtime/runtime.cc:422]   at android.os.Handler.dispatchMessage(Handler.java:102)
A/art: art/runtime/runtime.cc:422]   at android.os.Looper.loop(Looper.java:154)
A/art: art/runtime/runtime.cc:422]   at android.app.ActivityThread.main(ActivityThread.java:6119)
A/art: art/runtime/runtime.cc:422]   at java.lang.reflect.Method.invoke!(Native method)
A/art: art/runtime/runtime.cc:422]   at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886)
A/art: art/runtime/runtime.cc:422]   at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776)
A/art: art/runtime/runtime.cc:422] 
A/art: art/runtime/runtime.cc:422] "Jit thread pool worker thread 0" prio=10 tid=2 Native (still starting up)
A/art: art/runtime/runtime.cc:422]   | group="" sCount=1 dsCount=0 obj=0x0 self=0xebe0f000
A/art: art/runtime/runtime.cc:422]   | sysTid=17335 nice=9 cgrp=default sched=0/0 handle=0xefd50920
A/art: art/runtime/runtime.cc:422]   | state=S schedstat=( 732741554 1676670357 1748 ) utm=19 stm=54 core=2 HZ=100
A/art: art/runtime/runtime.cc:422]   | stack=0xefc52000-0xefc54000 stackSize=1022KB
A/art: art/runtime/runtime.cc:422]   | held mutexes=
A/art: art/runtime/runtime.cc:422]   kernel: futex_wait_queue_me+0xc0/0x110
A/art: art/runtime/runtime.cc:422]   kernel: futex_wait+0x112/0x250
A/art: art/runtime/runtime.cc:422]   kernel: do_futex+0xdc/0xb30
A/art: art/runtime/runtime.cc:422]   kernel: compat_SyS_futex+0x75/0x150
A/art: art/runtime/runtime.cc:422]   kernel: do_fast_syscall_32+0xa4/0x250
A/art: art/runtime/runtime.cc:422]   kernel: sysenter_flags_fixed+0x8/0x12
A/art: art/runtime/runtime.cc:422]   native: #00 pc 00000bb7  [vdso] (__kernel_vsyscall+7)
A/art: art/runtime/runtime.cc:422]   native: #01 pc 0001e268  /system/lib/libc.so (syscall+40)
A/art: art/runtime/runtime.cc:422]   native: #02 pc 0011d27e  /system/lib/libart.so (_ZN3art17ConditionVariable16WaitHoldingLocksEPNS_6ThreadE+142)
A/art: art/runtime/runtime.cc:422]   native: #03 pc 0011d1e7  /system/lib/libart.so (_ZN3art17ConditionVariable4WaitEPNS_6ThreadE+39)
A/art: art/runtime/runtime.cc:422]   native: #04 pc 0055e312  /system/lib/libart.so (_ZN3art10ThreadPool7GetTaskEPNS_6ThreadE+258)
A/art: art/runtime/runtime.cc:422]   native: #05 pc 0055d615  /system/lib/libart.so (_ZN3art16ThreadPoolWorker3RunEv+117)
A/art: art/runtime/runtime.cc:422]   native: #06 pc 0055cdea  /system/lib/libart.so (_ZN3art16ThreadPoolWorker8CallbackEPv+106)
A/art: art/runtime/runtime.cc:422]   native: #07 pc 000752b2  /system/lib/libc.so (_ZL15__pthread_startPv+210)
A/art: art/runtime/runtime.cc:422]   native: #08 pc 000202ae  /system/lib/libc.so (__start_thread+30)
A/art: art/runtime/runtime.cc:422]   native: #09 pc 0001e086  /system/lib/libc.so (__bionic_clone+70)
A/art: art/runtime/runtime.cc:422]   (no managed stack frames)
A/art: art/runtime/runtime.cc:422] 
A/art: art/runtime/runtime.cc:422] "Signal Catcher" prio=10 tid=3 WaitingInMainSignalCatcherLoop
A/art: art/runtime/runtime.cc:422]   | group="" sCount=1 dsCount=0 obj=0x12c66d30 self=0xe7641e00
A/art: art/runtime/runtime.cc:422]   | sysTid=17336 nice=0 cgrp=default sched=0/0 handle=0xefc4f920
A/art: art/runtime/runtime.cc:422]   | state=S schedstat=( 1582240 354703 3 ) utm=0 stm=0 core=2 HZ=100
A/art: art/runtime/runtime.cc:422]   | stack=0xefb53000-0xefb55000 stackSize=1014KB
A/art: art/runtime/runtime.cc:422]   | held mutexes=
A/art: art/runtime/runtime.cc:422]   kernel: do_sigtimedwait+0x152/0x1e0
A/art: art/runtime/runtime.cc:422]   kernel: C_SYSC_rt_sigtimedwait+0x8d/0xe0
A/art: art/runtime/runtime.cc:422]   kernel: compat_SyS_rt_sigtimedwait+0xe/0x10
A/art: art/runtime/runtime.cc:422]   kernel: do_fast_syscall_32+0xa4/0x250
A/art: art/runtime/runtime.cc:422]   kernel: sysenter_flags_fixed+0x8/0x12
A/art: art/runtime/runtime.cc:422]   native: #00 pc 00000bb9  [vdso] (__kernel_vsyscall+9)
A/art: art/runtime/runtime.cc:422]   native: #01 pc 00078221  /system/lib/libc.so (__rt_sigtimedwait+33)
A/art: art/runtime/runtime.cc:422]   native: #02 pc 000289ef  /system/lib/libc.so (sigwait+95)
A/art: art/runtime/runtime.cc:422]   native: #03 pc 0052ebeb  /system/lib/libart.so (_ZN3art9SignalSet4WaitEv+59)
A/art: art/runtime/runtime.cc:422]   native: #04 pc 0052e5b7  /system/lib/libart.so (_ZN3art13SignalCatcher13WaitForSignalEPNS_6ThreadERNS_9SignalSetE+215)
A/art: art/runtime/runtime.cc:422]   native: #05 pc 0052c71a  /system/lib/libart.so (_ZN3art13SignalCatcher3RunEPv+314)
A/art: art/runtime/runtime.cc:422]   native: #06 pc 000752b2  /system/lib/libc.so (_ZL15__pthread_startPv+210)
A/art: art/runtime/runtime.cc:422]   native: #07 pc 000202ae  /system/lib/libc.so (__start_thread+30)
A/art: art/runtime/runtime.cc:422]   native: #08 pc 0001e086  /system/lib/libc.so (__bionic_clone+70)
A/art: art/runtime/runtime.cc:422]   (no managed stack frames)
A/art: art/runtime/runtime.cc:422] 
A/art: art/runtime/runtime.cc:422] "JDWP" prio=10 tid=4 WaitingInMainDebuggerLoop
A/art: art/runtime/runtime.cc:422]   | group="" sCount=1 dsCount=0 obj=0x12c66dc0 self=0xebe10400
A/art: art/runtime/runtime.cc:422]   | sysTid=17337 nice=0 cgrp=default sched=0/0 handle=0xefb50920
A/art: art/runtime/runtime.cc:422]   | state=S schedstat=( 126525621 183243820 1040 ) utm=7 stm=4 core=0 HZ=100
A/art: art/runtime/runtime.cc:422]   | stack=0xefa54000-0xefa56000 stackSize=1014KB
A/art: art/runtime/runtime.cc:422]   | held mutexes=
A/art: art/runtime/runtime.cc:422]   kernel: poll_schedule_timeout+0x49/0x70
A/art: art/runtime/runtime.cc:422]   kernel: do_select+0x57d/0x7c0
A/art: art/runtime/runtime.cc:422]   kernel: compat_core_sys_select+0x1cf/0x290
A/art: art/runtime/runtime.cc:422]   kernel: compat_SyS_pselect6+0x223/0x240
A/art: art/runtime/runtime.cc:422]   kernel: do_fast_syscall_32+0xa4/0x250
A/art: art/runtime/runtime.cc:422]   kernel: sysenter_flags_fixed+0x8/0x12
A/art: art/runtime/runtime.cc:422]   native: #00 pc 00000bb9  [vdso] (__kernel_vsyscall+9)
A/art: art/runtime/runtime.cc:422]   native: #01 pc 0007802b  /system/lib/libc.so (__pselect6+43)
A/art: art/runtime/runtime.cc:422]   native: #02 pc 000268fd  /system/lib/libc.so (select+173)
A/art: art/runtime/runtime.cc:422]   native: #03 pc 00678d7c  /system/lib/libart.so (_ZN3art4JDWP12JdwpAdbState15ProcessIncomingEv+476)
A/art: art/runtime/runtime.cc:422]   native: #04 pc 003aba81  /system/lib/libart.so (_ZN3art4JDWP9JdwpState3RunEv+593)
A/art: art/runtime/runtime.cc:422]   native: #05 pc 003aaf00  /system/lib/libart.so (_ZN3art4JDWPL15StartJdwpThreadEPv+48)
A/art: art/runtime/runtime.cc:422]   native: #06 pc 000752b2  /system/lib/libc.so (_ZL15__pthread_startPv+210)
A/art: art/runtime/runtime.cc:422]   native: #07 pc 000202ae  /system/lib/libc.so (__start_thread+30)
A/art: art/runtime/runtime.cc:422]   native: #08 pc 0001e086  /system/lib/libc.so (__bionic_clone+70)
A/art: art/runtime/runtime.cc:422]   (no managed stack frames)
A/art: art/runtime/runtime.cc:422] 
A/art: art/runtime/runtime.cc:422] "ReferenceQueueDaemon" prio=10 tid=5 Waiting
A/art: art/runtime/runtime.cc:422]   | group="" sCount=1 dsCount=0 obj=0x12c66e50 self=0xe7643200
A/art: art/runtime/runtime.cc:422]   native: #04 pc 00148b50  /system/lib/libart.so (_ZN3art8CheckJNI23SetPrimitiveArrayRegionEPKcNS_9Primitive4TypeEP7_JNIEnvP7_jarrayiiPKv+592)
A/art: art/runtime/runtime.cc:422]   native: #05 pc 00136ae0  /system/lib/libart.so (_ZN3art8CheckJNI18SetByteArrayRegionEP7_JNIEnvP11_jbyteArrayiiPKa+80)
A/art: art/runtime/runtime.cc:422]   native: #06 pc 00028777  /system/lib/libopenjdk.so (SocketInputStream_socketRead0+295)
A/art: art/runtime/runtime.cc:422]   native: #07 pc 000eef04  /system/framework/x86/boot.oat (Java_java_net_SocketInputStream_socketRead0__Ljava_io_FileDescriptor_2_3BIII+192)
A/art: art/runtime/runtime.cc:422]   at java.net.SocketInputStream.socketRead0(Native method)
A/art: art/runtime/runtime.cc:422]   at java.net.SocketInputStream.read(SocketInputStream.java:151)
A/art: art/runtime/runtime.cc:422]   at java.net.SocketInputStream.read(SocketInputStream.java:120)
A/art: art/runtime/runtime.cc:422]   at okio.Okio$2.read(Okio.java:139)
A/art: art/runtime/runtime.cc:422]   at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
A/art: art/runtime/runtime.cc:422]   at okio.RealBufferedSource.request(RealBufferedSource.java:67)
A/art: art/runtime/runtime.cc:422]   at okio.RealBufferedSource.require(RealBufferedSource.java:60)
A/art: art/runtime/runtime.cc:422]   at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
A/art: art/runtime/runtime.cc:422]   at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
A/art: art/runtime/runtime.cc:422]   at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
A/art: art/runtime/runtime.cc:422]   at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
A/art: art/runtime/runtime.cc:422]   kernel: compat_SyS_futex+0x75/0x150
A/art: art/runtime/runtime.cc:422]   kernel: do_fast_syscall_32+0xa4/0x250
A/art: art/runtime/runtime.cc:422]   kernel: sysenter_flags_fixed+0x8/0x12
A/art: art/runtime/runtime.cc:422]   native: #00 pc 00000bb9  [vdso] (__kernel_vsyscall+9)
A/art: art/runtime/runtime.cc:422]   native: #01 pc 0001e268  /system/lib/libc.so (syscall+40)
A/art: art/runtime/runtime.cc:422]   native: #02 pc 0011d968  /system/lib/libart.so (_ZN3art17ConditionVariable9TimedWaitEPNS_6ThreadExi+200)
A/art: art/runtime/runtime.cc:422]   native: #03 pc 0043728c  /system/lib/libart.so (_ZN3art7Monitor4WaitEPNS_6ThreadExibNS_11ThreadStateE+572)
A/art: art/runtime/runtime.cc:422]   native: #04 pc 00439953  /system/lib/libart.so (_ZN3art7Monitor4WaitEPNS_6ThreadEPNS_6mirror6ObjectExibNS_11ThreadStateE+339)
A/art: art/runtime/runtime.cc:422]   native: #05 pc 00451d4d  /system/lib/libart.so (_ZN3artL13Object_waitJIEP7_JNIEnvP8_jobjectxi+93)
A/art: art/runtime/runtime.cc:422]   native: #06 pc 0000083c  /system/framework/x86/boot.oat (Java_java_lang_Object_wait__JI+136)
A/art: art/runtime/runtime.cc:422]   at java.lang.Object.wait!(Native method)
A/art: art/runtime/runtime.cc:422]   - waiting on <0x08a50e6e> (a okhttp3.ConnectionPool)
A/art: art/runtime/runtime.cc:422]   at okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
A/art: art/runtime/runtime.cc:422]   - locked <0x08a50e6e> (a okhttp3.ConnectionPool)
A/art: art/runtime/runtime.cc:422]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
A/art: art/runtime/runtime.cc:422]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
A/art: art/runtime/runtime.cc:422]   at java.lang.Thread.run(Thread.java:761)
A/art: art/runtime/runtime.cc:422] 
A/art: art/runtime/runtime.cc:422] "OkHttp WebSocket http://192.168.0.112:4985/..." prio=10 tid=17 Waiting
A/art: art/runtime/runtime.cc:422]   | group="" sCount=1 dsCount=0 obj=0x12c66670 self=0xebe13600
A/art: art/runtime/runtime.cc:422]   | sysTid=18492 nice=-8 cgrp=default sched=0/0 handle=0xbe3ff920
A/art: art/runtime/runtime.cc:422]   | state=S schedstat=( 21054823464 12161981045 79592 ) utm=890 stm=1214 core=2 HZ=100
A/art: art/runtime/runtime.cc:422]   | stack=0xbe2fd000-0xbe2ff000 stackSize=1038KB
A/art: art/runtime/runtime.cc:422]   | held mutexes=
A/art: art/runtime/runtime.cc:422]   kernel: futex_wait_queue_me+0xc0/0x110
A/art: art/runtime/runtime.cc:422]   kernel: futex_wait+0x112/0x250
A/art: art/runtime/runtime.cc:422]   kernel: do_futex+0xdc/0xb30
A/art: art/runtime/runtime.cc:422]   kernel: compat_SyS_futex+0x75/0x150
A/art: art/runtime/runtime.cc:422]   kernel: do_fast_syscall_32+0xa4/0x250
A/art: art/runtime/runtime.cc:422]   kernel: sysenter_flags_fixed+0x8/0x12
A/art: art/runtime/runtime.cc:422]   native: #00 pc 00000bb7  [vdso] (__kernel_vsyscall+7)
A/art: art/runtime/runtime.cc:422]   native: #01 pc 0001e268  /system/lib/libc.so (syscall+40)
A/art: art/runtime/runtime.cc:422]   native: #02 pc 0011d27e  /system/lib/libart.so (_ZN3art17ConditionVariable16WaitHoldingLocksEPNS_6ThreadE+142)
A/art: art/runtime/runtime.cc:422]   native: #03 pc 0011d1e7  /system/lib/libart.so (_ZN3art17ConditionVariable4WaitEPNS_6ThreadE+39)
A/art: art/runtime/runtime.cc:422]   native: #04 pc 00437269  /system/lib/libart.so (_ZN3art7Monitor4WaitEPNS_6ThreadExibNS_11ThreadStateE+537)
A/art: art/runtime/runtime.cc:422]   native: #05 pc 00439953  /system/lib/libart.so (_ZN3art7Monitor4WaitEPNS_6ThreadEPNS_6mirror6ObjectExibNS_11ThreadStateE+339)
A/art: art/runtime/runtime.cc:422]   native: #06 pc 00451d4d  /system/lib/libart.so (_ZN3artL13Object_waitJIEP7_JNIEnvP8_jobjectxi+93)
A/art: art/runtime/runtime.cc:422]   native: #07 pc 0000083c  /system/framework/x86/boot.oat (Java_java_lang_Object_wait__JI+136)
A/art: art/runtime/runtime.cc:422]   at java.lang.Object.wait!(Native method)
A/art: art/runtime/runtime.cc:422]   - waiting on <0x0fc0d90f> (a java.lang.Object)
A/art: art/runtime/runtime.cc:422]   at java.lang.Thread.parkFor$(Thread.java:2127)
A/art: art/runtime/runtime.cc:422]   - locked <0x0fc0d90f> (a java.lang.Object)
A/art: art/runtime/runtime.cc:422]   at sun.misc.Unsafe.park(Unsafe.java:325)
A/art: art/runtime/runtime.cc:422]   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:161)
A/art: art/runtime/runtime.cc:422]   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2035)
A/art: art/runtime/runtime.cc:422]   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1091)
A/art: art/runtime/runtime.cc:422]   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1084)
A/art: art/runtime/runtime.cc:422]   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1058)
A/art: art/runtime/runtime.cc:422]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
A/art: art/runtime/runtime.cc:422]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
A/art: art/runtime/runtime.cc:422]   at java.lang.Thread.run(Thread.java:761)
A/art: art/runtime/runtime.cc:422] 
A/art: art/runtime/runtime.cc:422] 
A/art: art/runtime/runtime.cc:427] JNI ERROR (app bug): weak global reference table overflow (max=51200)
A/art: art/runtime/runtime.cc:427] weak global reference table dump:
A/art: art/runtime/runtime.cc:427]   Last 10 entries (of 51200):
A/art: art/runtime/runtime.cc:427]     51199: 0x13eeb310 java.lang.Thread
A/art: art/runtime/runtime.cc:427]     51198: 0x13eeb160 java.lang.Thread
A/art: art/runtime/runtime.cc:427]     51197: 0x13d40e50 java.lang.Thread
A/art: art/runtime/runtime.cc:427]     51196: 0x13d40d30 java.lang.Thread
A/art: art/runtime/runtime.cc:427]     51195: 0x13d40a60 java.lang.Thread
A/art: art/runtime/runtime.cc:427]     51194: 0x13d408b0 java.lang.Thread
A/art: art/runtime/runtime.cc:427]     51193: 0x13d40700 java.lang.Thread
A/art: art/runtime/runtime.cc:427]     51192: 0x13d40550 java.lang.Thread
A/art: art/runtime/runtime.cc:427]     51191: 0x13d403a0 java.lang.Thread
A/art: art/runtime/runtime.cc:427]     51190: 0x13d400d0 java.lang.Thread
A/art: art/runtime/runtime.cc:427]   Summary:
A/art: art/runtime/runtime.cc:427]         1 of android.support.v4.util.SparseArrayCompat
A/art: art/runtime/runtime.cc:427]         1 of android.arch.lifecycle.LifecycleRegistry
A/art: art/runtime/runtime.cc:427]         1 of com.couchbase.lite.DatabaseConfiguration
A/art: art/runtime/runtime.cc:427]         1 of android.support.v4.util.SimpleArrayMap
A/art: art/runtime/runtime.cc:427]         1 of com.couchbase.lite.Replicator
A/art: art/runtime/runtime.cc:427]         1 of android.support.v4.app.FragmentActivity$1
A/art: art/runtime/runtime.cc:427]         1 of com.couchbase.androidclient.MainActivity
A/art: art/runtime/runtime.cc:427]         1 of android.support.v4.app.FragmentController
A/art: art/runtime/runtime.cc:427]         1 of com.couchbase.lite.Database
A/art: art/runtime/runtime.cc:427]         1 of android.support.v7.app.AppCompatDelegateImplN
A/art: art/runtime/runtime.cc:427]         1 of okio.AsyncTimeout$Watchdog
A/art: art/runtime/runtime.cc:427]       680 of java.lang.Thread (680 unique instances)
A/art: art/runtime/runtime.cc:427]         4 of java.lang.String (4 unique instances)
A/art: art/runtime/runtime.cc:427]         2 of java.lang.ThreadGroup (2 unique instances)
A/art: art/runtime/runtime.cc:427]         3 of byte[] (2016 elements) (3 unique instances)
A/art: art/runtime/runtime.cc:427]         3 of byte[] (6864 elements) (3 unique instances)
A/art: art/runtime/runtime.cc:427]         1 of byte[] (9408 elements)
A/art: art/runtime/runtime.cc:427]         5 of byte[] (12544 elements) (5 unique instances)
A/art: art/runtime/runtime.cc:427]         4 of byte[] (13720 elements) (4 unique instances)
A/art: art/runtime/runtime.cc:427]         1 of byte[] (23520 elements)
A/art: art/runtime/runtime.cc:427]         9 of byte[] (28224 elements) (9 unique instances)
A/art: art/runtime/runtime.cc:427]        27 of byte[] (36100 elements) (27 unique instances)
A/art: art/runtime/runtime.cc:427]         2 of byte[] (47432 elements) (2 unique instances)
A/art: art/runtime/runtime.cc:427]         2 of byte[] (48400 elements) (2 unique instances)
A/art: art/runtime/runtime.cc:427]        14 of byte[] (50176 elements) (14 unique instances)
A/art: art/runtime/runtime.cc:427]         3 of byte[] (63504 elements) (3 unique instances)
A/art: art/runtime/runtime.cc:427]         2 of byte[] (64512 elements) (2 unique instances)
A/art: art/runtime/runtime.cc:427]         3 of byte[] (108192 elements) (3 unique instances)
A/art: art/runtime/runtime.cc:427]         3 of byte[] (112896 elements) (3 unique instances)
A/art: art/runtime/runtime.cc:427]         4 of byte[] (147000 elements) (4 unique instances)
A/art: art/runtime/runtime.cc:427]         5 of byte[] (367236 elements) (5 unique instances)
A/art: art/runtime/runtime.cc:427]         1 of java.net.URI
A/art: art/runtime/runtime.cc:427]      4984 of java.lang.Class (4984 unique instances)
A/art: art/runtime/runtime.cc:427]         3 of dalvik.system.PathClassLoader (1 unique instances)
A/art: art/runtime/runtime.cc:427]        23 of java.lang.DexCache (23 unique instances)
A/art: art/runtime/runtime.cc:427]         1 of com.android.internal.policy.PhoneLayoutInflater
A/art: art/runtime/runtime.cc:427]         1 of com.android.internal.policy.PhoneWindow
A/art: art/runtime/runtime.cc:427]         1 of android.content.res.Resources
A/art: art/runtime/runtime.cc:427]         1 of android.content.pm.ActivityInfo
A/art: art/runtime/runtime.cc:427]         1 of android.app.ContextImpl
A/art: art/runtime/runtime.cc:427]         1 of android.app.ActivityThread
A/art: art/runtime/runtime.cc:427]         1 of android.app.SharedElementCallback$1
A/art: art/runtime/runtime.cc:427]         1 of android.app.FragmentController
A/art: art/runtime/runtime.cc:427]        13 of android.view.RenderNode (13 unique instances)
A/art: art/runtime/runtime.cc:427]         1 of android.os.Handler
A/art: art/runtime/runtime.cc:427]         1 of android.app.ActivityManager$TaskDescription
A/art: art/runtime/runtime.cc:427]         1 of android.os.BinderProxy
A/art: art/runtime/runtime.cc:427]         1 of android.content.res.Resources$Theme
A/art: art/runtime/runtime.cc:427]         1 of android.os.StrictMode$InstanceTracker
A/art: art/runtime/runtime.cc:427]         1 of com.android.internal.os.ZygoteInit$MethodAndArgsCaller
A/art: art/runtime/runtime.cc:427]         1 of android.content.ComponentName
A/art: art/runtime/runtime.cc:427]         1 of android.app.Application
A/art: art/runtime/runtime.cc:427]         1 of android.content.Intent
A/art: art/runtime/runtime.cc:427]         1 of android.app.ActivityThread$H
A/art: art/runtime/runtime.cc:427]         1 of android.content.res.Configuration
A/art: art/runtime/runtime.cc:427]         1 of android.app.Instrumentation
A/art: art/runtime/runtime.cc:427]         1 of android.view.WindowManagerImpl
A/art: art/runtime/runtime.cc:427]         1 of android.app.ActivityTransitionState
A/art: art/runtime/runtime.cc:427] 
A/art: art/runtime/runtime.cc:427] 
A/libc: Fatal signal 6 (SIGABRT), code -6 in tid 17434 (Thread-46019)

        [ 01-22 17:12:47.524   116:  116 W/         ]
        debuggerd: handling request: pid=17330 uid=10190 gid=10190 tid=17434
Disconnected from the target VM, address: 'localhost:9224', transport: 'socket'

snej · 2018-01-23T18:03:43Z

@hrajput89,

Your test was running on Android. This is the iOS issue tracker. Please post this in couchbase-lite-java-core instead.
This is related to the 1M docs test, right?
Please provide details of what you were testing -- what software versions, what's in the database, what type of replication, etc. (I really don't think I should have to keep reminding Couchbase engineers of the minimum standards of a useable bug report.)
Please don't paste in a huge log; it makes it difficult to scroll through to find the comments. Instead, paste it into a gist and put a link to that in the bug report.

pasin · 2018-02-07T09:45:27Z

I was able to either push and pull 1M docs (each doc has 20 5 string and 1 number).

PUSH :
- Time 1292.5 s
- Memory is up to 250M and gradually dropped to 39M at the end
- CPU 100% - 150%
PULL
- Time 977.787 s
- Memory is up to 275M (Memory didn't drop back)
- CPU 100% - 150%

The memory and CPU usage seemed to be high. I will need to retest PULL replicator to see why the memory didn't go back to normal after the replication is done.

Note:

SGW mode is memory walrus.
The replicator type is push_and_pull.

snej · 2018-02-07T17:16:25Z

@pasin, could you save a copy of the 1M-doc database somewhere? The performance is slower than I'd expect; were you using a debug or release build?

pasin · 2018-02-08T06:47:40Z

I have retested with the current 2.0-dev binary. I'm using SGW with persistent walrus.

PUSH (PUSH AND PULL):

Time: 1157.71 seconds
CPU: 0% - 231% (Swing all the time)
Memory : ~Max 240MB (Down to 36MB after the replication is finished).

PULL (PUSH AND PULL):

Time: Too Slow, 43195 docs were pulled after 30 mins.
CPU: Stay at 100%
Memory : Stay at ~80MB

PUSH (PUSH ONLY)

Time: 844.55 seconds
CPU: 0% - 181% (Pretty swing all the time)
Memory : Max 253.6MB (Down to 53.1MB after the replication is finished).

PULL (PULL ONLY)

Time: Too Slow, 69967 docs were pulled after 45 mins.
CPU: Stay at ~102%
Memory : Stay at ~91MB

Walrus db file : https://www.dropbox.com/s/sjkgbbxd6wxqc8d/db.zip?dl=0

Pull replicator is very slow when using the persistent walrus. I'm testing with Couchbase Server to see if the result is different.

pasin · 2018-02-08T06:56:06Z

The result is much better especially the pull replication when testing with Couchbase Server (v5.01). There seems to be a memory issue with pull replicator.

PUSH (PUSH ONLY)

Time: 743.559 seconds
CPU: 100% - 160%
Memory : Stay ~45M

PULL (PULL ONLY)

Time: 670.447 Seconds
CPU: 20% - 100%
Memory : 284.2 MB (Didn’t drop after the replicator is finished)

snej · 2018-02-08T17:45:51Z

Strange that pull would be slower than push with Walrus. The performance problem with persistent Walrus stores is on writes, since they rewrite the entire database every time.

snej · 2018-02-14T01:38:19Z

@pasin, what client device were your tests run on?

I just ran a test using the cblite tool on my MBP, and SG running on the same MBP (non-persistent Walrus storage). I didn't watch memory usage. Obviously this isn't reflective of running on a real iOS device; I'm just noting this for future reference.

Push: 119 sec (8374 docs/sec)
Pull: 222 sec (4492 docs/sec) -- I noticed ~15 sec of delay before SG sends any docs

And with the client on my iMac, but SG still on the MBP, over WiFi:

Push: 110 secs (9089 docs/sec) -- memory went up to 243MB then declined
Pull: 246 secs (4057 docs/sec) -- memory stayed flat at 29MB

pasin · 2018-02-14T01:56:53Z

I used an iPhone 6 device.

snej · 2018-02-14T20:01:40Z

I fixed couchbase/couchbase-lite-core#404, which will help push memory usage a lot.

For couchbase/couchbase-lite-ios#1970

snej · 2018-02-16T17:58:37Z

Still TBD: Testing performance of the replication on a real iOS device. @pasin said he'd send me the app project he used to run the replication, since we don't have any existing tool to do this.

pasin · 2018-02-16T19:53:49Z

@snej here is the app : https://github.com/pasin/TestMDocs.

Note :

I have included two SGW config files (walrus.config and server.config) at root folder for running SGW in walrus or CB Server mode.
From the app, you could create a million docs and start a continuous push_and_pull replicator to sync with SGW.
See the console log for the usage time. The time is measured from connecting to idle.
I modified the app to include couchbase-lite-ios as a subproject instead of linking with the binary.

snej · 2018-02-20T00:40:57Z

Results on an iPhone 6s+ using Pasin's test app with latest CBL, and SG with a non-persistent Walrus bucket:

Push: 335 sec (2985 docs/sec) -- memory steady at ~39MB
Pull: 494 sec (2024 docs/sec) -- memory goes up steadily to 275MB and doesn't decrease

Since the pull memory leak does not occur with LiteCore alone, I'm wondering if there's a leak in CBLReplication.

snej · 2018-02-20T00:53:39Z

Ah, the memory usage looks to come mostly from SequenceTracker; there are about 400k 112-byte malloc blocks created by the following backtrace:

  13 CouchbaseLite c4db_endTransaction /Couchbase/2.0/vendor/couchbase-lite-core/Replicator/DBWorker.cc:222
  12 CouchbaseLite c4Internal::tryCatch(C4Error*, litecore::function_ref<void ()>) /Couchbase/2.0/vendor/couchbase-lite-core/C/c4ExceptionUtils.cc:35
  11 CouchbaseLite litecore::function_ref<void ()>::operator()() const /Couchbase/2.0/vendor/couchbase-lite-core/C/c4ExceptionUtils.cc:72
  10 CouchbaseLite c4Internal::Database::_cleanupTransaction(bool) /Couchbase/2.0/vendor/couchbase-lite-core/LiteCore/Database/Database.cc:85
   9 CouchbaseLite litecore::function_ref<void (litecore::DataFile*)>::operator()(litecore::DataFile*) const /Couchbase/2.0/vendor/couchbase-lite-core/LiteCore/Storage/DataFile+Shared.hh:72
   8 CouchbaseLite void litecore::function_ref<void (litecore::DataFile*)>::callback_fn<c4Internal::Database::_cleanupTransaction(bool)::$_1>(long, litecore::DataFile*) /Couchbase/2.0/vendor/couchbase-lite-core/Xcode/../LiteCore/Support/function_ref.hh:175
   7 CouchbaseLite litecore::SequenceTracker::_documentChanged(fleece::alloc_slice const&, fleece::alloc_slice const&, unsigned long long, unsigned long long) /Couchbase/2.0/vendor/couchbase-lite-core/LiteCore/Database/SequenceTracker.cc:136
   6 CouchbaseLite void std::__1::list<litecore::SequenceTracker::Entry, std::__1::allocator<litecore::SequenceTracker::Entry> >::emplace_back<fleece::alloc_slice const&, fleece::alloc_slice const&, unsigned long long&, unsigned int&>(fleece::alloc_slice const&&&, fleece::alloc_slice const&&&, unsigned long long&&&, unsigned int&&&) /Couchbase/2.0/vendor/couchbase-lite-core/LiteCore/Database/SequenceTracker.cc:1626
   5 CouchbaseLite std::__1::allocator_traits<std::__1::allocator<std::__1::__list_node<litecore::SequenceTracker::Entry, void*> > >::allocate(std::__1::allocator<std::__1::__list_node<litecore::SequenceTracker::Entry, void*> >&, unsigned long) /Couchbase/2.0/vendor/couchbase-lite-core/LiteCore/Database/SequenceTracker.cc:1502
   4 CouchbaseLite std::__1::allocator<std::__1::__list_node<litecore::SequenceTracker::Entry, void*> >::allocate(unsigned long, void const*) /Couchbase/2.0/vendor/couchbase-lite-core/LiteCore/Database/SequenceTracker.cc:1747
   3 CouchbaseLite std::__1::__allocate(unsigned long) /Couchbase/2.0/vendor/couchbase-lite-core/LiteCore/Database/SequenceTracker.cc:226
   2 libc++abi.dylib operator new(unsigned long)
   1 libsystem_malloc.dylib malloc
   0 libsystem_malloc.dylib malloc_zone_malloc

snej · 2018-02-20T03:09:29Z

Found a sort-of memory leak in LiteCore's SequenceTracker class -- sequence entries were being added to the main thread's SequenceTracker (copied from the replicator thread's) but the main thread didn't have any observers so it never ended up cleaning up obsolete entries (calling removeObsoleteEntries().) Thus the main thread kept accumulating more and more entries, up to a million.

Now that I've fixed the bug, pull memory usage stays in the range of 30-40MB. Pulls are slightly faster too, about 430sec (~2300 docs/sec).

pasin changed the title ~~Replication of 1M documents never completes and high CPU/memory usage~~ 2.0: Replication of 1M documents never completes and high CPU/memory usage Dec 11, 2017

djpongh added this to the 2.0.0 milestone Dec 15, 2017

djpongh added P1: high backlog labels Dec 15, 2017

rajagp added P1: high and removed P1: high labels Dec 19, 2017

djpongh added P2: medium P1: high and removed P1: high P2: medium labels Jan 18, 2018

djpongh added ready and removed backlog labels Jan 26, 2018

djpongh assigned pasin Feb 7, 2018

sridevi-15 added the feature-blocker label Feb 7, 2018

djpongh assigned snej and unassigned pasin Feb 8, 2018

snej added a commit to couchbase/couchbase-lite-core that referenced this issue Feb 15, 2018

Improved pull performance by throttling rate of handling change lists

be107d1

For couchbase/couchbase-lite-ios#1970

snej closed this as completed in 68b8096 Feb 20, 2018

snej removed the ready label Feb 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2.0: Replication of 1M documents never completes and high CPU/memory usage #1970

2.0: Replication of 1M documents never completes and high CPU/memory usage #1970

rajagp commented Dec 1, 2017

pasin commented Jan 19, 2018

raghusarangapani commented Jan 19, 2018

sridevi-15 commented Jan 19, 2018

snej commented Jan 19, 2018

hrajput89 commented Jan 23, 2018 •

edited by snej

Loading

snej commented Jan 23, 2018

pasin commented Feb 7, 2018 •

edited

Loading

snej commented Feb 7, 2018

pasin commented Feb 8, 2018 •

edited

Loading

pasin commented Feb 8, 2018 •

edited

Loading

snej commented Feb 8, 2018

snej commented Feb 14, 2018 •

edited

Loading

pasin commented Feb 14, 2018

snej commented Feb 14, 2018

snej commented Feb 16, 2018

pasin commented Feb 16, 2018 •

edited

Loading

snej commented Feb 20, 2018

snej commented Feb 20, 2018

snej commented Feb 20, 2018 •

edited

Loading

2.0: Replication of 1M documents never completes and high CPU/memory usage #1970

2.0: Replication of 1M documents never completes and high CPU/memory usage #1970

Comments

rajagp commented Dec 1, 2017

pasin commented Jan 19, 2018

raghusarangapani commented Jan 19, 2018

sridevi-15 commented Jan 19, 2018

snej commented Jan 19, 2018

hrajput89 commented Jan 23, 2018 • edited by snej Loading

snej commented Jan 23, 2018

pasin commented Feb 7, 2018 • edited Loading

snej commented Feb 7, 2018

pasin commented Feb 8, 2018 • edited Loading

pasin commented Feb 8, 2018 • edited Loading

snej commented Feb 8, 2018

snej commented Feb 14, 2018 • edited Loading

pasin commented Feb 14, 2018

snej commented Feb 14, 2018

snej commented Feb 16, 2018

pasin commented Feb 16, 2018 • edited Loading

snej commented Feb 20, 2018

snej commented Feb 20, 2018

snej commented Feb 20, 2018 • edited Loading

hrajput89 commented Jan 23, 2018 •

edited by snej

Loading

pasin commented Feb 7, 2018 •

edited

Loading

pasin commented Feb 8, 2018 •

edited

Loading

pasin commented Feb 8, 2018 •

edited

Loading

snej commented Feb 14, 2018 •

edited

Loading

pasin commented Feb 16, 2018 •

edited

Loading

snej commented Feb 20, 2018 •

edited

Loading