Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

arangodb stopped responding (Arangodb 2.7.0 on OS X) #1571

Closed
markterm opened this issue Nov 14, 2015 · 3 comments
Closed

arangodb stopped responding (Arangodb 2.7.0 on OS X) #1571

markterm opened this issue Nov 14, 2015 · 3 comments
Labels
1 Question

Comments

@markterm
Copy link

We're looking at using arangodb for our application, and have created a proof-of-concept. While testing some large AQL queries from the web interface, arandod pegged up to full CPU and is no longer responsive - http://localhost:8529 doesn't respond and if I connect by commandline it gets as far as: "Connected to ArangoDB 'tcp://127.0.0.1:8529' version: 2.7.0 [standalone], database: '_system', username: 'root'" and doesn't display anything further.

I wasn't running any change to the database or collection structure at the same time. This is obviously a bit worrying were the same thing to happen in production!

Here is a list of threads running at the time:

 thread #1: tid = 0x1ff129, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  thread #2: tid = 0x1ff12e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #3: tid = 0x1ff134, 0x00007fff945b72b2 libsystem_kernel.dylib`__semwait_signal + 10
  thread #4: tid = 0x1ff13c, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #5: tid = 0x1ff13d, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #6: tid = 0x1ff13e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #7: tid = 0x1ff13f, 0x00007fff945b72b2 libsystem_kernel.dylib`__semwait_signal + 10
  thread #8: tid = 0x1ff145, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #9: tid = 0x1ff146, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #10: tid = 0x1ff148, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #11: tid = 0x1ff149, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #12: tid = 0x1ff14a, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #13: tid = 0x1ff14b, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #14: tid = 0x1ff14c, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #15: tid = 0x1ff14d, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #16: tid = 0x1ff14e, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #17: tid = 0x1ff21d, 0x00007fff945b7222 libsystem_kernel.dylib`__select + 10
  thread #18: tid = 0x1ff21e, 0x00007fff945b7222 libsystem_kernel.dylib`__select + 10
  thread #19: tid = 0x1ff21f, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #20: tid = 0x1ff223, 0x00007fff8cd2f0f6 libsystem_malloc.dylib`malloc + 68
  thread #21: tid = 0x1ff239, 0x00007fff8cd344d8 libsystem_malloc.dylib`get_tiny_free_size + 73
  thread #22: tid = 0x29f166, 0x0000000102ca3876 arangod`TRI_CopyToJson(TRI_memory_zone_s*, TRI_json_t*, TRI_json_t const*) + 14
  thread #23: tid = 0x29f1fd, 0x00007fff8cd345e5 libsystem_malloc.dylib`tiny_free_list_remove_ptr + 258
  thread #24: tid = 0x3f0d0d, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #25: tid = 0x3f0d0e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #26: tid = 0x3f0e5f, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #27: tid = 0x3f0e60, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #28: tid = 0x95438b, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #29: tid = 0x95438c, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #30: tid = 0x106aaa1, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #31: tid = 0x106aaa2, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
(lldb)
Process 55051 stopped
* thread #1: tid = 0x1ff129, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  thread #2: tid = 0x1ff12e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #3: tid = 0x1ff134, 0x00007fff945b72b2 libsystem_kernel.dylib`__semwait_signal + 10
  thread #4: tid = 0x1ff13c, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #5: tid = 0x1ff13d, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #6: tid = 0x1ff13e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #7: tid = 0x1ff13f, 0x00007fff945b72b2 libsystem_kernel.dylib`__semwait_signal + 10
  thread #8: tid = 0x1ff145, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #9: tid = 0x1ff146, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #10: tid = 0x1ff148, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #11: tid = 0x1ff149, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #12: tid = 0x1ff14a, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #13: tid = 0x1ff14b, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #14: tid = 0x1ff14c, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #15: tid = 0x1ff14d, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #16: tid = 0x1ff14e, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #17: tid = 0x1ff21d, 0x00007fff945b7222 libsystem_kernel.dylib`__select + 10
  thread #18: tid = 0x1ff21e, 0x00007fff945b7222 libsystem_kernel.dylib`__select + 10
  thread #19: tid = 0x1ff21f, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #20: tid = 0x1ff223, 0x00007fff8cd2f0f6 libsystem_malloc.dylib`malloc + 68
  thread #21: tid = 0x1ff239, 0x00007fff8cd344d8 libsystem_malloc.dylib`get_tiny_free_size + 73
  thread #22: tid = 0x29f166, 0x0000000102ca3876 arangod`TRI_CopyToJson(TRI_memory_zone_s*, TRI_json_t*, TRI_json_t const*) + 14
  thread #23: tid = 0x29f1fd, 0x00007fff8cd345e5 libsystem_malloc.dylib`tiny_free_list_remove_ptr + 258
  thread #24: tid = 0x3f0d0d, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #25: tid = 0x3f0d0e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #26: tid = 0x3f0e5f, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #27: tid = 0x3f0e60, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #28: tid = 0x95438b, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #29: tid = 0x95438c, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #30: tid = 0x106aaa1, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #31: tid = 0x106aaa2, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
(lldb) thread list
Process 55051 stopped
* thread #1: tid = 0x1ff129, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  thread #2: tid = 0x1ff12e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #3: tid = 0x1ff134, 0x00007fff945b72b2 libsystem_kernel.dylib`__semwait_signal + 10
  thread #4: tid = 0x1ff13c, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #5: tid = 0x1ff13d, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #6: tid = 0x1ff13e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #7: tid = 0x1ff13f, 0x00007fff945b72b2 libsystem_kernel.dylib`__semwait_signal + 10
  thread #8: tid = 0x1ff145, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #9: tid = 0x1ff146, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #10: tid = 0x1ff148, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #11: tid = 0x1ff149, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #12: tid = 0x1ff14a, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #13: tid = 0x1ff14b, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #14: tid = 0x1ff14c, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #15: tid = 0x1ff14d, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #16: tid = 0x1ff14e, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
  thread #17: tid = 0x1ff21d, 0x00007fff945b7222 libsystem_kernel.dylib`__select + 10
  thread #18: tid = 0x1ff21e, 0x00007fff945b7222 libsystem_kernel.dylib`__select + 10
  thread #19: tid = 0x1ff21f, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #20: tid = 0x1ff223, 0x00007fff8cd2f0f6 libsystem_malloc.dylib`malloc + 68
  thread #21: tid = 0x1ff239, 0x00007fff8cd344d8 libsystem_malloc.dylib`get_tiny_free_size + 73
  thread #22: tid = 0x29f166, 0x0000000102ca3876 arangod`TRI_CopyToJson(TRI_memory_zone_s*, TRI_json_t*, TRI_json_t const*) + 14
  thread #23: tid = 0x29f1fd, 0x00007fff8cd345e5 libsystem_malloc.dylib`tiny_free_list_remove_ptr + 258
  thread #24: tid = 0x3f0d0d, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #25: tid = 0x3f0d0e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #26: tid = 0x3f0e5f, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #27: tid = 0x3f0e60, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #28: tid = 0x95438b, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #29: tid = 0x95438c, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #30: tid = 0x106aaa1, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #31: tid = 0x106aaa2, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
(lldb) thread backtrace all
* thread #1: tid = 0x1ff129, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102a1bbab arangod`triagens::rest::ApplicationServer::wait() + 69
    frame #4: 0x0000000102ba64fb arangod`triagens::arango::ArangoServer::startupServer() + 3359
    frame #5: 0x0000000102b7bbb0 arangod`triagens::rest::AnyServer::start() + 186
    frame #6: 0x0000000102a15fe0 arangod`main + 86
    frame #7: 0x00007fff9888d5ad libdyld.dylib`start + 1
    frame #8: 0x00007fff9888d5ad libdyld.dylib`start + 1

  thread #2: tid = 0x1ff12e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102ca8e99 arangod`MessageQueueWorker(void*) + 424
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #3: tid = 0x1ff134, 0x00007fff945b72b2 libsystem_kernel.dylib`__semwait_signal + 10
    frame #0: 0x00007fff945b72b2 libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fff8eef9a75 libsystem_c.dylib`nanosleep + 199
    frame #2: 0x00007fff8eef9968 libsystem_c.dylib`usleep + 54
    frame #3: 0x0000000102bb55ce arangod`StatisticsQueueWorker(void*) + 551
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #4: tid = 0x1ff13c, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6c7e arangod`TRI_WaitCondition(TRI_condition_s*) + 18
    frame #3: 0x0000000102cd4aa4 arangod`triagens::basics::ThreadPool::dequeue(std::__1::function<void ()>&) + 56
    frame #4: 0x0000000102cd4e33 arangod`triagens::basics::WorkerThread::run() + 61
    frame #5: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #6: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #7: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #8: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #9: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #10: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #5: tid = 0x1ff13d, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6c7e arangod`TRI_WaitCondition(TRI_condition_s*) + 18
    frame #3: 0x0000000102cd4aa4 arangod`triagens::basics::ThreadPool::dequeue(std::__1::function<void ()>&) + 56
    frame #4: 0x0000000102cd4e33 arangod`triagens::basics::WorkerThread::run() + 61
    frame #5: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #6: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #7: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #8: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #9: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #10: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #6: tid = 0x1ff13e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c1daa1 arangod`TRI_CleanupVocBase(void*) + 1077
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #7: tid = 0x1ff13f, 0x00007fff945b72b2 libsystem_kernel.dylib`__semwait_signal + 10
    frame #0: 0x00007fff945b72b2 libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fff8eef9a75 libsystem_c.dylib`nanosleep + 199
    frame #2: 0x00007fff8eef9968 libsystem_c.dylib`usleep + 54
    frame #3: 0x0000000102c42d1f arangod`DatabaseManager(void*) + 961
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #8: tid = 0x1ff145, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c58632 arangod`triagens::wal::AllocatorThread::run() + 644
    frame #4: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #5: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #6: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #7: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #8: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #9: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #9: tid = 0x1ff146, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c6fb89 arangod`triagens::wal::SynchronizerThread::run() + 383
    frame #4: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #5: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #6: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #7: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #8: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #9: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #10: tid = 0x1ff148, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c58a43 arangod`triagens::wal::CollectorThread::run() + 239
    frame #4: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #5: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #6: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #7: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #8: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #9: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #11: tid = 0x1ff149, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c6dda4 arangod`triagens::wal::RemoverThread::run() + 262
    frame #4: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #5: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #6: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #7: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #8: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #9: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #12: tid = 0x1ff14a, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c22ea2 arangod`TRI_CompactorVocBase(void*) + 5034
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #13: tid = 0x1ff14b, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
    frame #0: 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame #1: 0x00000001031ac9e5 arangod`v8::base::Semaphore::Wait() + 11
    frame #2: 0x00000001031af4bf arangod`v8::platform::TaskQueue::GetNext() + 45
    frame #3: 0x00000001031af64a arangod`v8::platform::WorkerThread::Run() + 36
    frame #4: 0x00000001031ade7f arangod`v8::base::ThreadEntry(void*) + 84
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #14: tid = 0x1ff14c, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
    frame #0: 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame #1: 0x00000001031ac9e5 arangod`v8::base::Semaphore::Wait() + 11
    frame #2: 0x00000001031af4bf arangod`v8::platform::TaskQueue::GetNext() + 45
    frame #3: 0x00000001031af64a arangod`v8::platform::WorkerThread::Run() + 36
    frame #4: 0x00000001031ade7f arangod`v8::base::ThreadEntry(void*) + 84
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #15: tid = 0x1ff14d, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
    frame #0: 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame #1: 0x00000001031ac9e5 arangod`v8::base::Semaphore::Wait() + 11
    frame #2: 0x00000001031af4bf arangod`v8::platform::TaskQueue::GetNext() + 45
    frame #3: 0x00000001031af64a arangod`v8::platform::WorkerThread::Run() + 36
    frame #4: 0x00000001031ade7f arangod`v8::base::ThreadEntry(void*) + 84
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #16: tid = 0x1ff14e, 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'V8 WorkerThread'
    frame #0: 0x00007fff945b1cd2 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame #1: 0x00000001031ac9e5 arangod`v8::base::Semaphore::Wait() + 11
    frame #2: 0x00000001031af4bf arangod`v8::platform::TaskQueue::GetNext() + 45
    frame #3: 0x00000001031af64a arangod`v8::platform::WorkerThread::Run() + 36
    frame #4: 0x00000001031ade7f arangod`v8::base::ThreadEntry(void*) + 84
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #17: tid = 0x1ff21d, 0x00007fff945b7222 libsystem_kernel.dylib`__select + 10
    frame #0: 0x00007fff945b7222 libsystem_kernel.dylib`__select + 10
    frame #1: 0x0000000102cf8cbd arangod`select_poll + 173
    frame #2: 0x0000000102cf5115 arangod`ev_run + 1049
    frame #3: 0x0000000102bb221c arangod`triagens::rest::SchedulerThread::run() + 210
    frame #4: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #5: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #6: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #7: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #8: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #9: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #18: tid = 0x1ff21e, 0x00007fff945b7222 libsystem_kernel.dylib`__select + 10
    frame #0: 0x00007fff945b7222 libsystem_kernel.dylib`__select + 10
    frame #1: 0x0000000102cf8cbd arangod`select_poll + 173
    frame #2: 0x0000000102cf5115 arangod`ev_run + 1049
    frame #3: 0x0000000102bb221c arangod`triagens::rest::SchedulerThread::run() + 210
    frame #4: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #5: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #6: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #7: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #8: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #9: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #19: tid = 0x1ff21f, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102bbe27a arangod`triagens::arango::ApplicationV8::collectGarbage() + 278
    frame #4: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #5: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #6: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #7: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #8: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #9: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #20: tid = 0x1ff223, 0x00007fff8cd2f0f6 libsystem_malloc.dylib`malloc + 68
    frame #0: 0x00007fff8cd2f0f6 libsystem_malloc.dylib`malloc + 68
    frame #1: 0x0000000102ca9cbc arangod`TRI_Allocate(TRI_memory_zone_s*, unsigned long long, bool) + 31
    frame #2: 0x0000000102cd3beb arangod`TRI_CopyToBlob(TRI_memory_zone_s*, TRI_blob_s*, TRI_blob_s const*) + 38
    frame #3: 0x0000000102ca3908 arangod`TRI_CopyToJson(TRI_memory_zone_s*, TRI_json_t*, TRI_json_t const*) + 160
    frame #4: 0x0000000102ca4456 arangod`TRI_CopyJson(TRI_memory_zone_s*, TRI_json_t const*) + 50
    frame #5: 0x0000000102a25a27 arangod`triagens::basics::Json::copy(triagens::basics::Json::autofree_e) const + 71
    frame #6: 0x0000000102a22d9c arangod`triagens::aql::AqlValue::clone() const + 96
    frame #7: 0x0000000102a45fc3 arangod`triagens::aql::EnumerateListBlock::getAqlValue(triagens::aql::AqlValue const&) + 151
    frame #8: 0x0000000102a45d08 arangod`triagens::aql::EnumerateListBlock::getSome(unsigned long, unsigned long) + 772
    frame #9: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #10: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #11: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #12: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #13: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #14: 0x0000000102a36b98 arangod`triagens::aql::FilterBlock::getBlock(unsigned long, unsigned long) + 356
    frame #15: 0x0000000102a36c40 arangod`triagens::aql::FilterBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 134
    frame #16: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #17: 0x0000000102a4695d arangod`triagens::aql::ExecutionBlock::getSome(unsigned long, unsigned long) + 15
    frame #18: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #19: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #20: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #21: 0x0000000102a37253 arangod`triagens::aql::ReturnBlock::getSome(unsigned long, unsigned long) + 25
    frame #22: 0x0000000102af599e arangod`triagens::aql::SubqueryBlock::executeSubquery() + 96
    frame #23: 0x0000000102af5863 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 83
    frame #24: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #25: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #26: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #27: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #28: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #29: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #30: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #31: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #32: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #33: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #34: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #35: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #36: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #37: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #38: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #39: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #40: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #41: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #42: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #43: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #44: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #45: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #46: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #47: 0x0000000102a37253 arangod`triagens::aql::ReturnBlock::getSome(unsigned long, unsigned long) + 25
    frame #48: 0x0000000102ad88e0 arangod`triagens::aql::Query::execute(triagens::aql::QueryRegistry*) + 844
    frame #49: 0x0000000102b80e56 arangod`triagens::arango::RestCursorHandler::processQuery(TRI_json_t const*) + 320
    frame #50: 0x0000000102b80554 arangod`triagens::arango::RestCursorHandler::createCursor() + 130
    frame #51: 0x0000000102b804b0 arangod`triagens::arango::RestCursorHandler::execute() + 60
    frame #52: 0x0000000102b4e725 arangod`triagens::rest::HttpServerJob::work() + 185
    frame #53: 0x0000000102b39b1d arangod`triagens::rest::DispatcherThread::handleJob(triagens::rest::Job*) + 173
    frame #54: 0x0000000102b3990e arangod`triagens::rest::DispatcherThread::run() + 190
    frame #55: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #56: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #57: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #58: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #59: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #60: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #21: tid = 0x1ff239, 0x00007fff8cd344d8 libsystem_malloc.dylib`get_tiny_free_size + 73
    frame #0: 0x00007fff8cd344d8 libsystem_malloc.dylib`get_tiny_free_size + 73
    frame #1: 0x00007fff8cd319cf libsystem_malloc.dylib`tiny_malloc_from_free_list + 771
    frame #2: 0x00007fff8cd30715 libsystem_malloc.dylib`szone_malloc_should_clear + 292
    frame #3: 0x00007fff8cd305b1 libsystem_malloc.dylib`malloc_zone_malloc + 71
    frame #4: 0x00007fff8cd2f0dc libsystem_malloc.dylib`malloc + 42
    frame #5: 0x0000000102ca9cbc arangod`TRI_Allocate(TRI_memory_zone_s*, unsigned long long, bool) + 31
    frame #6: 0x0000000102cd3beb arangod`TRI_CopyToBlob(TRI_memory_zone_s*, TRI_blob_s*, TRI_blob_s const*) + 38
    frame #7: 0x0000000102ca4456 arangod`TRI_CopyJson(TRI_memory_zone_s*, TRI_json_t const*) + 50
    frame #8: 0x0000000102a248cd arangod`triagens::aql::AqlValue::extractObjectMember(triagens::arango::AqlTransaction*, TRI_document_collection_t const*, char const*, bool, triagens::basics::StringBuffer&) const + 265
    frame #9: 0x0000000102a9a83d arangod`triagens::aql::Expression::executeSimpleExpression(triagens::aql::AstNode const*, TRI_document_collection_t const**, triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, bool) + 1415
    frame #10: 0x0000000102a9a3d0 arangod`triagens::aql::Expression::executeSimpleExpression(triagens::aql::AstNode const*, TRI_document_collection_t const**, triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, bool) + 282
    frame #11: 0x0000000102a9a4db arangod`triagens::aql::Expression::executeSimpleExpression(triagens::aql::AstNode const*, TRI_document_collection_t const**, triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, bool) + 549
    frame #12: 0x0000000102a9a533 arangod`triagens::aql::Expression::executeSimpleExpression(triagens::aql::AstNode const*, TRI_document_collection_t const**, triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, bool) + 637
    frame #13: 0x0000000102a9a4db arangod`triagens::aql::Expression::executeSimpleExpression(triagens::aql::AstNode const*, TRI_document_collection_t const**, triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, bool) + 549
    frame #14: 0x0000000102a9a0bb arangod`triagens::aql::Expression::execute(triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, TRI_document_collection_t const**) + 471
    frame #15: 0x0000000102a38bd0 arangod`triagens::aql::CalculationBlock::executeExpression(triagens::aql::AqlItemBlock*) + 222
    frame #16: 0x0000000102a38dad arangod`triagens::aql::CalculationBlock::doEvaluation(triagens::aql::AqlItemBlock*) + 283
    frame #17: 0x0000000102a38ef6 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 36
    frame #18: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #19: 0x0000000102a36b98 arangod`triagens::aql::FilterBlock::getBlock(unsigned long, unsigned long) + 356
    frame #20: 0x0000000102a36c40 arangod`triagens::aql::FilterBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 134
    frame #21: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #22: 0x0000000102a4695d arangod`triagens::aql::ExecutionBlock::getSome(unsigned long, unsigned long) + 15
    frame #23: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #24: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #25: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #26: 0x0000000102a37253 arangod`triagens::aql::ReturnBlock::getSome(unsigned long, unsigned long) + 25
    frame #27: 0x0000000102af599e arangod`triagens::aql::SubqueryBlock::executeSubquery() + 96
    frame #28: 0x0000000102af5863 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 83
    frame #29: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #30: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #31: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #32: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #33: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #34: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #35: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #36: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #37: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #38: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #39: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #40: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #41: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #42: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #43: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #44: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #45: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #46: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #47: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #48: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #49: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #50: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #51: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #52: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #53: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #54: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #55: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #56: 0x0000000102a37253 arangod`triagens::aql::ReturnBlock::getSome(unsigned long, unsigned long) + 25
    frame #57: 0x0000000102ad88e0 arangod`triagens::aql::Query::execute(triagens::aql::QueryRegistry*) + 844
    frame #58: 0x0000000102b80e56 arangod`triagens::arango::RestCursorHandler::processQuery(TRI_json_t const*) + 320
    frame #59: 0x0000000102b80554 arangod`triagens::arango::RestCursorHandler::createCursor() + 130
    frame #60: 0x0000000102b804b0 arangod`triagens::arango::RestCursorHandler::execute() + 60
    frame #61: 0x0000000102b4e725 arangod`triagens::rest::HttpServerJob::work() + 185
    frame #62: 0x0000000102b39b1d arangod`triagens::rest::DispatcherThread::handleJob(triagens::rest::Job*) + 173
    frame #63: 0x0000000102b3990e arangod`triagens::rest::DispatcherThread::run() + 190
    frame #64: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #65: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #66: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #67: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #68: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #69: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #22: tid = 0x29f166, 0x0000000102ca3876 arangod`TRI_CopyToJson(TRI_memory_zone_s*, TRI_json_t*, TRI_json_t const*) + 14
    frame #0: 0x0000000102ca3876 arangod`TRI_CopyToJson(TRI_memory_zone_s*, TRI_json_t*, TRI_json_t const*) + 14
    frame #1: 0x0000000102ca3908 arangod`TRI_CopyToJson(TRI_memory_zone_s*, TRI_json_t*, TRI_json_t const*) + 160
    frame #2: 0x0000000102ca4456 arangod`TRI_CopyJson(TRI_memory_zone_s*, TRI_json_t const*) + 50
    frame #3: 0x0000000102a248cd arangod`triagens::aql::AqlValue::extractObjectMember(triagens::arango::AqlTransaction*, TRI_document_collection_t const*, char const*, bool, triagens::basics::StringBuffer&) const + 265
    frame #4: 0x0000000102a9a83d arangod`triagens::aql::Expression::executeSimpleExpression(triagens::aql::AstNode const*, TRI_document_collection_t const**, triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, bool) + 1415
    frame #5: 0x0000000102a9a3d0 arangod`triagens::aql::Expression::executeSimpleExpression(triagens::aql::AstNode const*, TRI_document_collection_t const**, triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, bool) + 282
    frame #6: 0x0000000102a9a4db arangod`triagens::aql::Expression::executeSimpleExpression(triagens::aql::AstNode const*, TRI_document_collection_t const**, triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, bool) + 549
    frame #7: 0x0000000102a9a533 arangod`triagens::aql::Expression::executeSimpleExpression(triagens::aql::AstNode const*, TRI_document_collection_t const**, triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, bool) + 637
    frame #8: 0x0000000102a9a4db arangod`triagens::aql::Expression::executeSimpleExpression(triagens::aql::AstNode const*, TRI_document_collection_t const**, triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, bool) + 549
    frame #9: 0x0000000102a9a0bb arangod`triagens::aql::Expression::execute(triagens::arango::AqlTransaction*, triagens::aql::AqlItemBlock const*, unsigned long, std::__1::vector<triagens::aql::Variable const*, std::__1::allocator<triagens::aql::Variable const*> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, TRI_document_collection_t const**) + 471
    frame #10: 0x0000000102a38bd0 arangod`triagens::aql::CalculationBlock::executeExpression(triagens::aql::AqlItemBlock*) + 222
    frame #11: 0x0000000102a38dad arangod`triagens::aql::CalculationBlock::doEvaluation(triagens::aql::AqlItemBlock*) + 283
    frame #12: 0x0000000102a38ef6 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 36
    frame #13: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #14: 0x0000000102a36b98 arangod`triagens::aql::FilterBlock::getBlock(unsigned long, unsigned long) + 356
    frame #15: 0x0000000102a36c40 arangod`triagens::aql::FilterBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 134
    frame #16: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #17: 0x0000000102a4695d arangod`triagens::aql::ExecutionBlock::getSome(unsigned long, unsigned long) + 15
    frame #18: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #19: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #20: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #21: 0x0000000102a37253 arangod`triagens::aql::ReturnBlock::getSome(unsigned long, unsigned long) + 25
    frame #22: 0x0000000102af599e arangod`triagens::aql::SubqueryBlock::executeSubquery() + 96
    frame #23: 0x0000000102af5863 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 83
    frame #24: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #25: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #26: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #27: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #28: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #29: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #30: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #31: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #32: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #33: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #34: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #35: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #36: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #37: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #38: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #39: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #40: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #41: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #42: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #43: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #44: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #45: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #46: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #47: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #48: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #49: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #50: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #51: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #52: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #53: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #54: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #55: 0x0000000102a37253 arangod`triagens::aql::ReturnBlock::getSome(unsigned long, unsigned long) + 25
    frame #56: 0x0000000102ad88e0 arangod`triagens::aql::Query::execute(triagens::aql::QueryRegistry*) + 844
    frame #57: 0x0000000102b80e56 arangod`triagens::arango::RestCursorHandler::processQuery(TRI_json_t const*) + 320
    frame #58: 0x0000000102b80554 arangod`triagens::arango::RestCursorHandler::createCursor() + 130
    frame #59: 0x0000000102b804b0 arangod`triagens::arango::RestCursorHandler::execute() + 60
    frame #60: 0x0000000102b4e725 arangod`triagens::rest::HttpServerJob::work() + 185
    frame #61: 0x0000000102b39b1d arangod`triagens::rest::DispatcherThread::handleJob(triagens::rest::Job*) + 173
    frame #62: 0x0000000102b3990e arangod`triagens::rest::DispatcherThread::run() + 190
    frame #63: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #64: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #65: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #66: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #67: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #68: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #23: tid = 0x29f1fd, 0x00007fff8cd345e5 libsystem_malloc.dylib`tiny_free_list_remove_ptr + 258
    frame #0: 0x00007fff8cd345e5 libsystem_malloc.dylib`tiny_free_list_remove_ptr + 258
    frame #1: 0x00007fff8cd32956 libsystem_malloc.dylib`szone_free_definite_size + 1480
    frame #2: 0x0000000102ca3770 arangod`TRI_DestroyJson(TRI_memory_zone_s*, TRI_json_t*) + 67
    frame #3: 0x0000000102ca37d1 arangod`TRI_FreeJson(TRI_memory_zone_s*, TRI_json_t*) + 18
    frame #4: 0x0000000102b9dfdd arangod`triagens::basics::Json::~Json() + 45
    frame #5: 0x0000000102a22af7 arangod`triagens::aql::AqlValue::destroy() + 131
    frame #6: 0x0000000102a20380 arangod`triagens::aql::AqlItemBlock::destroy() + 76
    frame #7: 0x0000000102ab2def arangod`triagens::aql::AqlItemBlock::~AqlItemBlock() + 15
    frame #8: 0x0000000102a36b80 arangod`triagens::aql::FilterBlock::getBlock(unsigned long, unsigned long) + 332
    frame #9: 0x0000000102a36c40 arangod`triagens::aql::FilterBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 134
    frame #10: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #11: 0x0000000102a4695d arangod`triagens::aql::ExecutionBlock::getSome(unsigned long, unsigned long) + 15
    frame #12: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #13: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #14: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #15: 0x0000000102a37253 arangod`triagens::aql::ReturnBlock::getSome(unsigned long, unsigned long) + 25
    frame #16: 0x0000000102af599e arangod`triagens::aql::SubqueryBlock::executeSubquery() + 96
    frame #17: 0x0000000102af5863 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 83
    frame #18: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #19: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #20: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #21: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #22: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #23: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #24: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #25: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #26: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #27: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #28: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #29: 0x0000000102af5829 arangod`triagens::aql::SubqueryBlock::getSome(unsigned long, unsigned long) + 25
    frame #30: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #31: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #32: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #33: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #34: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #35: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #36: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #37: 0x0000000102a38ee1 arangod`triagens::aql::CalculationBlock::getSome(unsigned long, unsigned long) + 15
    frame #38: 0x0000000102a46ded arangod`triagens::aql::ExecutionBlock::getBlock(unsigned long, unsigned long) + 43
    frame #39: 0x0000000102a4709d arangod`triagens::aql::ExecutionBlock::getOrSkipSome(unsigned long, unsigned long, bool, triagens::aql::AqlItemBlock*&, unsigned long&) + 135
    frame #40: 0x0000000102a469c7 arangod`triagens::aql::ExecutionBlock::getSomeWithoutRegisterClearout(unsigned long, unsigned long) + 43
    frame #41: 0x0000000102a37253 arangod`triagens::aql::ReturnBlock::getSome(unsigned long, unsigned long) + 25
    frame #42: 0x0000000102ad88e0 arangod`triagens::aql::Query::execute(triagens::aql::QueryRegistry*) + 844
    frame #43: 0x0000000102b80e56 arangod`triagens::arango::RestCursorHandler::processQuery(TRI_json_t const*) + 320
    frame #44: 0x0000000102b80554 arangod`triagens::arango::RestCursorHandler::createCursor() + 130
    frame #45: 0x0000000102b804b0 arangod`triagens::arango::RestCursorHandler::execute() + 60
    frame #46: 0x0000000102b4e725 arangod`triagens::rest::HttpServerJob::work() + 185
    frame #47: 0x0000000102b39b1d arangod`triagens::rest::DispatcherThread::handleJob(triagens::rest::Job*) + 173
    frame #48: 0x0000000102b3990e arangod`triagens::rest::DispatcherThread::run() + 190
    frame #49: 0x0000000102cd3ed7 arangod`triagens::basics::Thread::runMe() + 129
    frame #50: 0x0000000102cd3e46 arangod`triagens::basics::Thread::startThread(void*) + 14
    frame #51: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #52: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #53: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #54: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #24: tid = 0x3f0d0d, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c1daa1 arangod`TRI_CleanupVocBase(void*) + 1077
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #25: tid = 0x3f0d0e, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c22ea2 arangod`TRI_CompactorVocBase(void*) + 5034
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #26: tid = 0x3f0e5f, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c1daa1 arangod`TRI_CleanupVocBase(void*) + 1077
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #27: tid = 0x3f0e60, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c22ea2 arangod`TRI_CompactorVocBase(void*) + 5034
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #28: tid = 0x95438b, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c1daa1 arangod`TRI_CleanupVocBase(void*) + 1077
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #29: tid = 0x95438c, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c22ea2 arangod`TRI_CompactorVocBase(void*) + 5034
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #30: tid = 0x106aaa1, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c1daa1 arangod`TRI_CleanupVocBase(void*) + 1077
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13

  thread #31: tid = 0x106aaa2, 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff945b6f5e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff9daa173d libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x0000000102ca6ddc arangod`TRI_TimedWaitCondition(TRI_condition_s*, unsigned long long) + 115
    frame #3: 0x0000000102c22ea2 arangod`TRI_CompactorVocBase(void*) + 5034
    frame #4: 0x0000000102cd54df arangod`ThreadStarter(void*) + 46
    frame #5: 0x00007fff9daa09b1 libsystem_pthread.dylib`_pthread_body + 131
    frame #6: 0x00007fff9daa092e libsystem_pthread.dylib`_pthread_start + 168
    frame #7: 0x00007fff9da9e385 libsystem_pthread.dylib`thread_start + 13
@dothebart
Copy link
Contributor

If I see that correctly, you're running 4 queries, at least one of them is a join.
None of your queries was using an index, so all of them are doing full table scans.
For sure you wouldn't want to do something like that in production - with neither RDBMS nor document store.
You can use the db._explain() command to analyze your queries and learn whether they are able to utilize indices or not. In general its advised to have one prominent filter that delivers a good selectivity on an indexed attribute.

Another possible pittfall is that you're doing a join with non existing attributes - which then in term will all become true, so you'll get the square of the two collections, which takes a lot of computing time and memory.

@markterm
Copy link
Author

Thanks for the prompt reply.

As I was running ad-hoc queries, I'm sure you're right that I got it executing four 'pathological' queries simultaneously which meant any further queries would be queued behind them until they timed out.

I do want to be able to run ad-hoc queries on a slave instance and it's always possible I'll send some bad ones. One way I should be able to mitigate this is by making the timeout more aggressive - I see there is a config setting for arangosh, but is it possible to set the request timeout when making HTTP AQL calls via the Cursor API either on a per-request or at a global server level? I can't find a setting in the documentation.

Also it looks like the 'Cancel' button when running a query in the aadvark web UI doesn't stop the execution - I suggest that would be a good idea.

@dothebart dothebart mentioned this issue Nov 19, 2015
Closed
@dothebart
Copy link
Contributor

Will be part of 2.8 - Thanks for reporting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
1 Question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@markterm @dothebart