Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions locale/en/docs/guides/diagnostics/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,16 @@ layout: docs.hbs

# Diagnostics Guide

These guides were created in the [Diagnostics Working Group](https://github.com/nodejs/diagnostics)
with the objective to provide a guidance when diagnosing an issue
in the user application.
The documentation project is organized based on user journey.
Those journeys are a coherent set of step-by-step procedures,
that a user follows for problem determination of reported issues.
These guides were created by the [Diagnostics Working Group][] with the
objective of providing guidance when diagnosing an issue in a user's
application.

The documentation project is organized based on user journey. Those journeys
are a coherent set of step-by-step procedures that a user can follow to
root-cause their issues.

This is the available set of diagnostics guides:

* [Memory](/en/docs/guides/diagnostics/memory)

[Diagnostics Working Group]: https://github.com/nodejs/diagnostics)
30 changes: 9 additions & 21 deletions locale/en/docs/guides/diagnostics/memory/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,10 @@ In this document you can learn about how to debug memory related issues.
* [My process runs out of memory](#my-process-runs-out-of-memory)
* [Symptoms](#symptoms)
* [Side Effects](#side-effects)
* [Debugging](#debugging)
* [My process utilizes memory inefficiently](#my-process-utilizes-memory-inefficiently)
* [Symptoms](#symptoms-1)
* [Side Effects](#side-effects-1)
* [Debugging](#debugging-1)
* [Debugging](#debugging)

## My process runs out of memory

Expand All @@ -29,28 +28,17 @@ efficient way of finding a memory leak is essential.
The user observes continuously increasing memory usage _(can be fast or slow,
over days or even weeks)_ then sees the process crashing and restarting by the
process manager. The process is maybe running slower than before and the
restarts make certain requests to fail _(load balancer responds with 502)_.
restarts cause some requests to fail _(load balancer responds with 502)_.

### Side Effects

* Process restarts due to the memory exhaustion and request are dropped on the
floor
* Process restarts due to the memory exhaustion and requests are dropped
on the floor
* Increased GC activity leads to higher CPU usage and slower response time
* GC blocking the Event Loop causing slowness
* Increased memory swapping slows down the process (GC activity)
* May not have enough available memory to get a Heap Snapshot

### Debugging

To debug a memory issue we need to be able to see how much space our specific
type of objects take, and what variables retain them to get garbage collected.
For the effective debugging we also need to know the allocation pattern of our
variables over time.

* [Using Heap Profiler](/en/docs/guides/diagnostics/memory/using-heap-profiler/)
* [Using Heap Snapshot](/en/docs/guides/diagnostics/memory/using-heap-snapshot/)
* [GC Traces](/en/docs/guides/diagnostics/memory/using-gc-traces)

## My process utilizes memory inefficiently

### Symptoms
Expand All @@ -63,12 +51,12 @@ garbage collector activity.
* An elevated number of page faults
* Higher GC activity and CPU usage

### Debugging
## Debugging

To debug a memory issue we need to be able to see how much space our specific
type of objects take, and what variables retain them to get garbage collected.
For the effective debugging we also need to know the allocation pattern of our
variables over time.
Most memory issues can be solved by determining how much space our specific
type of objects take and what variables are preventing them from being garbage
collected. It can also help to know the allocation pattern of our program over
time.

* [Using Heap Profiler](/en/docs/guides/diagnostics/memory/using-heap-profiler/)
* [Using Heap Snapshot](/en/docs/guides/diagnostics/memory/using-heap-snapshot/)
Expand Down
95 changes: 51 additions & 44 deletions locale/en/docs/guides/diagnostics/memory/using-gc-traces.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ one thing it's that when GC is running, your code is not.
You may want to know how often and how long the garbage collection is running,
and what is the outcome.

## Runnig with garbage collection traces
## Running with garbage collection traces

You can see traces for garbage collection in console output of your process
using the `--trace_gc` flag.
Expand All @@ -21,10 +21,9 @@ $ node --trace_gc app.js
```

You might want to avoid getting traces from the entire lifetime of your
process running on a server. In that case, set the flag from within the process,
and switch it off once the need for tracing is over.

Here's how to print GC events to stdout for one minute.
process. In that case, you can set the flag from within the process, and switch
it off once the need for tracing is over. For example, here's how to print GC
events to stdout for one minute:

```js
const v8 = require('v8');
Expand All @@ -34,15 +33,15 @@ setTimeout(() => { v8.setFlagsFromString('--notrace_gc'); }, 60e3);

### Examining a trace with `--trace_gc`

Obtained traces of garbage collection looks like the following lines.
The output traces look like the following:

```
[19278:0x5408db0] 44 ms: Scavenge 2.3 (3.0) -> 1.9 (4.0) MB, 1.2 / 0.0 ms (average mu = 1.000, current mu = 1.000) allocation failure

[23521:0x10268b000] 120 ms: Mark-sweep 100.7 (122.7) -> 100.6 (122.7) MB, 0.15 / 0.0 ms (average mu = 0.132, current mu = 0.137) deserialize GC in old space requested
```

This is how to interpret the trace data (for the second line):
Let's look at the second line. Here is how to interpret the trace data:

<table>
<tr>
Expand All @@ -59,33 +58,39 @@ This is how to interpret the trace data (for the second line):
</tr>
<tr>
<td>120</td>
<td>Time since the process start in ms</td>
<td>Time since the thread start in ms</td>
</tr>
<tr>
<td>Mark-sweep</td>
<td>Type / Phase of GC</td>
</tr>
<tr>
<td>100.7</td>
<td>Heap used before GC in MB</td>
<td>Heap used before GC in MiB</td>
</tr>
<tr>
<td>122.7</td>
<td>Total heap before GC in MB</td>
<td>Total heap before GC in MiB</td>
</tr>
<tr>
<td>100.6</td>
<td>Heap used after GC in MB</td>
<td>Heap used after GC in MiB</td>
</tr>
<tr>
<td>122.7</td>
<td>Total heap after GC in MB</td>
<td>Total heap after GC in MiB</td>
</tr>
<tr>
<td>0.15 / 0.0 <br/>
(average mu = 0.132, current mu = 0.137)</td>
<td>0.15</td>
<td>Time spent in GC in ms</td>
</tr>
<tr>
<td>0.0</td>
<td>Time spent in GC callbacks in ms</td>
</tr>
<tr>
<td>(average mu = 0.132, current mu = 0.137)</td>
<td>Mutator utilization (from 0-1)</td>
<tr>
<td>deserialize GC in old space requested</td>
<td>Reason for GC</td>
Expand All @@ -104,8 +109,8 @@ const { PerformanceObserver } = require('perf_hooks');
const obs = new PerformanceObserver((list) => {
const entry = list.getEntries()[0];
/*
The entry would be an instance of PerformanceEntry containing
metrics of garbage collection.
The entry is an instance of PerformanceEntry containing
metrics of a single garbage collection event.
For example:
PerformanceEntry {
name: 'gc',
Expand All @@ -117,7 +122,7 @@ const obs = new PerformanceObserver((list) => {
*/
});

// Subscribe notifications of GCs
// Subscribe to notifications of GCs
obs.observe({ entryTypes: ['gc'] });

// Stop subscription
Expand Down Expand Up @@ -178,39 +183,41 @@ For more information, you can refer to
## Examples of diagnosing memory issues with trace option:

A. How to get context of bad allocations
1. Suppose we observe that the old space is continously increasing.
2. But due to heavy gc, the heap maximum is not hit, but the process is slow.
1. Suppose we observe that the old space is continuously increasing.
2. But due to heavy GC, the heap maximum is not hit, but the process is slow.
3. Review the trace data and figure out how much is the total heap before and
after the gc.
4. Reduce `--max-old-space-size` such that the total heap is closer to the
limit.
5. Allow the program to run, hit the out of memory.
after the GC.
4. Reduce [`--max-old-space-size`][] such that the total heap is closer to the
limit.
5. Allow the program to run and run out-of-memory.
6. The produced log shows the failing context.

B. How to assert whether there is a memory leak when heap growth is observed
1. Suppose we observe that the old space is continously increasing.
2. Due to heavy gc, the heap maximum is not hit, but the process is slow.
1. Suppose we observe that the old space is continuously increasing.
2. Due to heavy GC, the heap maximum is not hit, but the process is slow.
3. Review the trace data and figure out how much is the total heap before and
after the gc.
4. Reduce `--max-old-space-size` such that the total heap is closer to the
limit.
after the GC.
4. Reduce [`--max-old-space-size`][] such that the total heap is closer to the
limit.
5. Allow the program to run, see if it hits the out of memory.
6. If it hits OOM, increment the heap size by ~10% or so and repeat few times.
If the same pattern is observed, it is indicative of a memory leak.
7. If there is no OOM, then freeze the heap size to that value - A packed heap
reduces memory footprint and compaction latency.

C. How to assert whether too many gcs are happening or too many gcs are causing
an overhead
1. Review the trace data, specifically around time between consecutive gcs.
2. Review the trace data, specifically around time spent in gc.
3. If the time between two gc is less than the time spent in gc, the
application is severely starving.
4. If the time between two gcs and the time spent in gc are very high, probably
the application can use a smaller heap.
5. If the time between two gcs are much greater than the time spent in gc,
application is relatively healthy.
6. If it hits out-of-memory, increment the heap size by ~10% or so and repeat
few times. If the same pattern is observed, it is indicative of a memory
leak.
7. If there is no out-of-memory error, then freeze the heap size to that value.
A packed heap reduces memory footprint and compaction latency.

C. How to assert whether too many GCs are happening or too many GCs are causing
an overhead
1. Review the trace data, specifically around time between consecutive GCs.
2. Review the trace data, specifically around time spent in GC.
3. If the time between two GC is less than the time spent in GC, the
application is severely starving.
4. If the time between two GCs and the time spent in GC is very high,
the application can probably use a smaller heap.
5. If the time between two GCs is much greater than the time spent in GC,
the application is relatively healthy.

[performance hooks]: https://nodejs.org/api/perf_hooks.html
[PerformanceEntry]: https://nodejs.org/api/perf_hooks.html#perf_hooks_class_performanceentry
[PerformanceObserver]: https://nodejs.org/api/perf_hooks.html#perf_hooks_class_performanceobserver
[`--max-old-space-size`]: https://nodejs.org/api/cli.html#--max-old-space-sizesize-in-megabytes
[performance hooks]: https://nodejs.org/api/perf_hooks.html
57 changes: 26 additions & 31 deletions locale/en/docs/guides/diagnostics/memory/using-heap-profiler.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,28 +5,24 @@ layout: docs.hbs

# Using Heap Profiler

To debug a memory issue we need to be able to see how much space our specific
type of objects take, and what variables retain them to get garbage collected.
For the effective debugging we also need to know the allocation pattern of our
variables over time.

The heap profiler acts on top of V8 towards to bring snapshots of memory over
time. In this document, we will cover the memory profiling using:
The heap profiler acts on top of V8 to capture allocations over time. In this
document, we will cover memory profiling using:

1. Allocation Timeline
2. Sampling Heap Profiler

Unlike heap dump that was cover in the [Using Heap Snapshot][],
the idea of using real-time profiling is to understand allocations in a given
time frame.
Unlike heap dumps which were covered in the [Using Heap Snapshot][] guide, the
idea of using real-time profiling is to understand allocations over a period of
time.

## Heap Profiler - Allocation Timeline

Heap Profiler is similar to the Sampling Heap Profiler, except it will trace
every allocation. It has higher overhead than the Sampling Heap Profiler so
it’s not recommended to use in production.

> You can use [@mmarchini/observe][] to do it programmatically.
> You can use [@mmarchini/observe][] to start and stop the profiler
> programmatically.

### How To

Expand All @@ -36,44 +32,43 @@ Start the application:
node --inspect index.js
```

> `--inspect-brk` is an better choice for scripts.
> `--inspect-brk` is a better choice for scripts.

Connect to the dev-tools instance in chrome and then:

* Select `memory` tab
* Select `Allocation instrumentation timeline`
* Start profiling
* Select the `Memory` tab.
* Select `Allocation instrumentation timeline`.
* Start profiling.

![heap profiler tutorial step 1][heap profiler tutorial 1]

After it, the heap profiling is running, it is strongly recommended to run
samples in order to identify memory issues, for this example, we will use
`Apache Benchmark` to produce load in the application.

> In this example, we are assuming the heap profiling under web application.
Once the heap profiling is running, it is strongly recommended to run samples
in order to identify memory issues. For example, if we were heap profiling a
web application, we could use `Apache Benchmark` to produce load:

```console
$ ab -n 1000 -c 5 http://localhost:3000
```

Hence, press stop button when the load expected is complete
Then, press stop button when the load is complete:

![heap profiler tutorial step 2][heap profiler tutorial 2]

Then look at the snapshot data towards to memory allocation.
Finally, look at the snapshot data:

![heap profiler tutorial step 3][heap profiler tutorial 3]

Check the [usefull links](#usefull-links) section for futher information
Check the [useful links](#useful-links) section for futher information
about memory terminology.

## Sampling Heap Profiler

Sampling Heap Profiler tracks memory allocation pattern and reserved space
over time. As it’s sampling based it has a low enough overhead to use it in
Sampling Heap Profiler tracks the memory allocation pattern and reserved space
over time. Since it is sampling based its overhead is low enough to use in
production systems.

> You can use the module [`heap-profiler`][] to do it programmatically.
> You can use the module [`heap-profiler`][] to start and stop the heap
> profiler programatically.

### How To

Expand All @@ -87,15 +82,15 @@ $ node --inspect index.js

Connect to the dev-tools instance and then:

1. Select `memory` tab
2. Select `Allocation sampling`
3. Start profiling
1. Select the `Memory` tab.
2. Select `Allocation sampling`.
3. Start profiling.

![heap profiler tutorial 4][heap profiler tutorial 4]

Produce some load and stop the profiler. It will generate a summary with
allocation based in the stacktrace, you can lookup to the functions with more
heap allocations in a timespan, see the example below:
allocation based on their stacktraces. You can focus on the functions with more
heap allocations, see the example below:

![heap profiler tutorial 5][heap profiler tutorial 5]

Expand Down
Loading