Skip to content

Conversation

nyannyacha
Copy link
Contributor

@nyannyacha nyannyacha commented Jun 4, 2024

What kind of change does this PR introduce?

Enhancement

Description

Clarify which memory limit kind caused termination to be requested and expose the memory metrics used to decide to terminate into the event record.

Sample (MemCheck / MemCheckState::check)

2024-06-04T03:20:14.726849Z DEBUG sb-supervisor base::rt_worker::worker_ctx: memory limit triggered: isolate: 6745b2ae-075e-4a5d-8414-912411de8aaf, detail: MemoryLimitDetailMemCheck { captured: 158894854 }    
2024-06-04T03:20:14.728807Z ERROR sb-supervisor base::rt_worker::supervisor::strategy_per_worker: memory limit reached for the worker: isolate: 6745b2ae-075e-4a5d-8414-912411de8aaf    
2024-06-04T03:20:14.731119Z ERROR          main base::rt_worker::worker_pool: failed to send request to user worker: request has been cancelled by supervisor    
2024-06-04T03:20:14.829827Z TRACE ThreadId(18) base::deno_runtime: name: Some("sb-iso-6745b2ae-075e-4a5d-8414-912411de8aaf"), thread_id: ThreadId(18), accumulated_cpu_time: 4413ms, malloced: 136.59MiB    
WorkerRequestCancelled: request has been cancelled by supervisor
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:77:21)
    at async callWorker (file:///home/deno/examples/main/index.ts:108:14)
    at async respond (ext:sb_core_main_js/js/http.js:162:14) {
  name: "WorkerRequestCancelled"
}
2024-06-04T03:20:14.846946Z DEBUG ThreadId(18) base::rt_worker::worker: CPU time used: 4413ms    
{
  timestamp: "2024-06-04T03:20:14.852Z",
  event_type: "Shutdown",
  event: [Object: null prototype] {
    reason: [Object: null prototype] {
      Memory: [Object: null prototype] {
        limited_by: "mem_check",
        captured: 158894854
      }
    },
    cpu_time_used: 4413,
    memory_used: [Object: null prototype] {
      total: 142947754,
      heap: 142933184,
      external: 14570,
      mem_check_captured: 143230146
    }
  },
  metadata: [Object: null prototype] {
    service_path: "./examples/serve",
    execution_id: "6745b2ae-075e-4a5d-8414-912411de8aaf"
  }
}

Sample (V8 / near_heap_callback)

2024-06-04T03:17:18.796863Z DEBUG ThreadId(17) base::rt_worker::worker_ctx: memory limit triggered: isolate: c8a88599-3535-4185-85f4-4efe7d7e1842, detail: MemoryLimitDetailV8 { current: 154140672, initial: 154140672 }    
2024-06-04T03:17:18.797096Z ERROR sb-supervisor base::rt_worker::supervisor::strategy_per_worker: memory limit reached for the worker: isolate: c8a88599-3535-4185-85f4-4efe7d7e1842    
2024-06-04T03:17:18.798519Z ERROR          main base::rt_worker::worker_pool: failed to send request to user worker: request has been cancelled by supervisor    
2024-06-04T03:17:18.803128Z TRACE ThreadId(17) base::deno_runtime: name: Some("sb-iso-c8a88599-3535-4185-85f4-4efe7d7e1842"), thread_id: ThreadId(17), accumulated_cpu_time: 4985ms, malloced: 8.93MiB    
2024-06-04T03:17:18.814446Z DEBUG ThreadId(17) base::rt_worker::worker: CPU time used: 4985ms    
WorkerRequestCancelled: request has been cancelled by supervisor
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:77:21)
    at async callWorker (file:///home/deno/examples/main/index.ts:108:14)
    at async respond (ext:sb_core_main_js/js/http.js:162:14) {
  name: "WorkerRequestCancelled"
}
{
  timestamp: "2024-06-04T03:17:18.819Z",
  event_type: "Shutdown",
  event: [Object: null prototype] {
    reason: [Object: null prototype] {
      Memory: [Object: null prototype] {
        limited_by: "v8",
        current: 154140672,
        initial: 154140672
      }
    },
    cpu_time_used: 4985,
    memory_used: [Object: null prototype] {
      total: 153479114,
      heap: 153464544,
      external: 14570,
      mem_check_captured: 153763754
    }
  },
  metadata: [Object: null prototype] {
    service_path: "./examples/serve",
    execution_id: "c8a88599-3535-4185-85f4-4efe7d7e1842"
  }
}

@nyannyacha nyannyacha requested a review from laktek June 4, 2024 03:40
@nyannyacha nyannyacha merged commit 15407a7 into supabase:main Jun 7, 2024
@nyannyacha nyannyacha deleted the fix-detailed-mem-metric branch June 7, 2024 04:16
Copy link

github-actions bot commented Jun 7, 2024

🎉 This PR is included in version 1.53.4 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants