Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
270 changes: 270 additions & 0 deletions docs/core/diagnostics/debug-deadlock.md

Large diffs are not rendered by default.

174 changes: 174 additions & 0 deletions docs/core/diagnostics/debug-highcpu.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
---
title: Debug high CPU usage - .NET Core
description: A tutorial that walks you through debugging high CPU usage in .NET Core.
ms.topic: tutorial
ms.date: 07/20/2020
---

# Debug high CPU usage in .NET Core

**This article applies to: ✔️** .NET Core 3.1 SDK and later versions

In this tutorial, you'll learn how to debug an excessive CPU usage scenario. Using the provided example [ASP.NET Core web app](https://docs.microsoft.com/samples/dotnet/samples/diagnostic-scenarios) source code repository, you can cause a deadlock intentionally. The endpoint will experience a hang and thread accumulation. You'll learn how you can use various tools to diagnose this scenario with several key pieces of diagnostics data.

In this tutorial, you will:

> [!div class="checklist"]
>
> - Investigate high CPU usage
> - Determine CPU usage with [dotnet-counters](dotnet-counters.md)
> - Use [dotnet-trace](dotnet-trace.md) for trace generation
> - Profile performance in PerfView
> - Diagnose and solve excessive CPU usage

## Prerequisites

The tutorial uses:

- [.NET Core 3.1 SDK](https://dotnet.microsoft.com/download/dotnet-core) or a later version.
- [Sample debug target](https://docs.microsoft.com/samples/dotnet/samples/diagnostic-scenarios) to trigger the scenario.
- [dotnet-trace](dotnet-trace.md) to list processes and generate a profile.
- [dotnet-counters](dotnet-counters.md) to monitor cpu usage.

## CPU counters

Before attempting to collect diagnostics data, you need to observe a high CPU condition. Run the [sample application](https://docs.microsoft.com/samples/dotnet/samples/diagnostic-scenarios) using the following command from the project root directory.

```dotnetcli
dotnet run
```

To find the process ID, use the following command:

```dotnetcli
dotnet-trace ps
```

Take note of the process ID from your command output. Our process ID was `22884`, but yours will be different. To check the current CPU usage, use the [dotnet-counters](dotnet-counters.md) tool command:

```dotnetcli
dotnet-counters monitor --refresh-interval 1 -p 22884
```

The `refresh-interval` is the number of seconds between the counter polling CPU values. The output should be similar to the following:

```console
Press p to pause, r to resume, q to quit.
Status: Running

[System.Runtime]
% Time in GC since last GC (%) 0
Allocation Rate / 1 sec (B) 0
CPU Usage (%) 0
Exception Count / 1 sec 0
GC Heap Size (MB) 4
Gen 0 GC Count / 60 sec 0
Gen 0 Size (B) 0
Gen 1 GC Count / 60 sec 0
Gen 1 Size (B) 0
Gen 2 GC Count / 60 sec 0
Gen 2 Size (B) 0
LOH Size (B) 0
Monitor Lock Contention Count / 1 sec 0
Number of Active Timers 1
Number of Assemblies Loaded 140
ThreadPool Completed Work Item Count / 1 sec 3
ThreadPool Queue Length 0
ThreadPool Thread Count 7
Working Set (MB) 63
```

With the web app running, immediately after startup, the CPU isn't being consumed at all and is reported at `0%`. Navigate to the `api/diagscenario/highcpu` route with `60000` as the route parameter:

[https://localhost:5001/api/diagscenario/highcpu/60000](https://localhost:5001/api/diagscenario/highcpu/60000)

Now, rerun the [dotnet-counters](dotnet-counters.md) command. To monitor just the `cpu-usage`, specify `System.Runtime[cpu-usage]` as part of the command.

```dotnetcli
dotnet-counters monitor System.Runtime[cpu-usage] -p 22884 --refresh-interval 1
```

You should see an increase in CPU usage as shown below:

```console
Press p to pause, r to resume, q to quit.
Status: Running

[System.Runtime]
CPU Usage (%) 25
```

Throughout the duration of the request, the CPU usage will hover around 25% . Depending on the host machine, expect varying CPU usage.

> [!TIP]
> To visualize an even higher CPU usage, you can exercise this endpoint in multiple browser tabs simultaneously.

At this point, you can safely say the CPU is running higher than you expect.

## Trace generation

When analyzing a slow request, you need a diagnostics tool that can provide insights into what the code is doing. The usual choice is a profiler, and there are different profiler options to choose from.

### [Linux](#tab/linux)

The `perf` tool can be used to generate .NET Core app profiles. Exit the previous instance of the [sample debug target](https://docs.microsoft.com/samples/dotnet/samples/diagnostic-scenarios).

Set the `COMPlus_PerfMapEnabled` environment variable to cause the .NET Core app to create a `map` file in the `/tmp` directory. This `map` file is used by `perf` to map CPU address to JIT-generated functions by name. For more information, see [Write perf map](../run-time-config/debugging-profiling.md#write-perf-map).

Run the [sample debug target](https://docs.microsoft.com/samples/dotnet/samples/diagnostic-scenarios) in the same terminal session.

```dotnetcli
export COMPlus_PerfMapEnabled=1
dotnet run
```

Exercise the high CPU API (<https://localhost:5001/api/diagscenario/highcpu/60000>) endpoint again. While it's running within the 1-minute request, run the `perf` command with your process ID:

```bash
sudo perf record -p 2266 -g
```

The `perf` command starts the performance collection process. Let it run for about 20-30 seconds, then press <kbd>Ctrl+C</kbd> to exit the collection process. You can use the same `perf` command to see the output of the trace.

```bash
sudo perf report -f
```

You can also generate a _flame-graph_ by using the following commands:

```bash
git clone --depth=1 https://github.com/BrendanGregg/FlameGraph
sudo perf script | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl > flamegraph.svg
```

This command generates a `flamegraph.svg` that you can view in the browser to investigate the performance problem:

[![Flame graph SVG image](media/flamegraph.jpg)](media/flamegraph.jpg#lightbox)

### [Windows](#tab/windows)

On Windows, you can use the [dotnet-trace](dotnet-trace.md) tool as a profiler. Using the previous [sample debug target](https://docs.microsoft.com/samples/dotnet/samples/diagnostic-scenarios), exercise the high CPU (<https://localhost:5001/api/diagscenario/highcpu/60000>) endpoint again. While it's running within the 1-minute request, use the `collect` command as follows:

```dotnetcli
dotnet-trace collect -p 22884 --providers Microsoft-DotNETCore-SampleProfiler
```

Let [dotnet-trace](dotnet-trace.md) run for about 20-30 seconds, and then press the <kbd>Enter</kbd> to exit the collection. The result is a `nettrace` file located in the same folder. The `nettrace` files are a great way to use existing analysis tools on Windows.

Open the `nettrace` with [`PerfView`](https://github.com/microsoft/perfview/blob/master/documentation/Downloading.md) as shown below.

[![PerfView image](media/perfview.jpg)](media/perfview.jpg#lightbox)

---

## See also

- [dotnet-trace](dotnet-trace.md) to list processes
- [dotnet-counters](dotnet-counters.md) to check managed memory usage
- [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file
- [dotnet/diagnostics](https://github.com/dotnet/diagnostics/tree/master/documentation/tutorial)

## Next steps

> [!div class="nextstepaction"]
> [Debug a deadlock in .NET Core](debug-deadlock.md)
27 changes: 14 additions & 13 deletions docs/core/diagnostics/debug-memory-leak.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@ description: Learn how to debug a memory leak in .NET Core.
ms.topic: tutorial
ms.date: 04/20/2020
---
# Tutorial: Debug a memory leak in .NET Core

**This article applies to:** ✔️ .NET Core 3.0 SDK and later versions
# Debug a memory leak in .NET Core

**This article applies to:** ✔️ .NET Core 3.1 SDK and later versions

This tutorial demonstrates the tools to analyze a .NET Core memory leak.

Expand All @@ -24,7 +25,7 @@ In this tutorial, you will:

The tutorial uses:

- [.NET Core 3.0 SDK](https://dotnet.microsoft.com/download/dotnet-core) or a later version.
- [.NET Core 3.1 SDK](https://dotnet.microsoft.com/download/dotnet-core) or a later version.
- [dotnet-trace](dotnet-trace.md) to list processes.
- [dotnet-counters](dotnet-counters.md) to check managed memory usage.
- [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file.
Expand Down Expand Up @@ -96,7 +97,7 @@ Focusing on this line:

You can see that the managed heap memory is 4 MB right after startup.

Now, hit the URL `http://localhost:5000/api/diagscenario/memleak/20000`.
Now, hit the URL `https://localhost:5001/api/diagscenario/memleak/20000`.

Observe that the memory usage has grown to 30 MB.

Expand Down Expand Up @@ -127,7 +128,7 @@ Complete

Once the dump is collected, you should have sufficient information to diagnose the failed process. If the failed process is running on a production server, now it's the ideal time for short-term remediation by restarting the process.

In this tutorial, you're now done with the [Sample debug target](https://docs.microsoft.com/samples/dotnet/samples/diagnostic-scenarios/) and you can close it. Navigate to the terminal that started the server and press `Control-C`.
In this tutorial, you're now done with the [Sample debug target](https://docs.microsoft.com/samples/dotnet/samples/diagnostic-scenarios/) and you can close it. Navigate to the terminal that started the server, and press <kbd>Ctrl+C</kbd>.

### Analyze the core dump

Expand Down Expand Up @@ -226,14 +227,14 @@ In this tutorial, you started a sample web server. This server should have been

You can also delete the dump file that was created.

## Next steps

Congratulations on completing this tutorial.
## See also

We're still publishing more diagnostic tutorials. You can read the draft versions on the [dotnet/diagnostics](https://github.com/dotnet/diagnostics/tree/master/documentation/tutorial) repository.
- [dotnet-trace](dotnet-trace.md) to list processes
- [dotnet-counters](dotnet-counters.md) to check managed memory usage
- [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file
- [dotnet/diagnostics](https://github.com/dotnet/diagnostics/tree/master/documentation/tutorial)

This tutorial covered the basics of key .NET diagnostic tools. For advanced usage, see the following reference documentation:
## Next steps

* [dotnet-trace](dotnet-trace.md) to list processes.
* [dotnet-counters](dotnet-counters.md) to check managed memory usage.
* [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file.
> [!div class="nextstepaction"]
> [Debug high CPU in .NET Core](debug-highcpu.md)
12 changes: 6 additions & 6 deletions docs/core/diagnostics/dotnet-trace.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,12 @@ The `dotnet-trace` tool:

## Commands

| Command |
| ----------------------------------------------------------- |
| [dotnet-trace collect](#dotnet-trace-collect) |
| [dotnet-trace convert](#dotnet-trace-convert) |
| [dotnet-trace ps](#dotnet-trace-ps) |
| [dotnet-trace list-profiles](#dotnet-trace-list-profiles) |
| Command |
|-----------------------------------------------------------|
| [dotnet-trace collect](#dotnet-trace-collect) |
| [dotnet-trace convert](#dotnet-trace-convert) |
| [dotnet-trace ps](#dotnet-trace-ps) |
| [dotnet-trace list-profiles](#dotnet-trace-list-profiles) |

## dotnet-trace collect

Expand Down
10 changes: 9 additions & 1 deletion docs/core/diagnostics/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Diagnostics tools overview - .NET Core
description: An overview of the tools and techniques available to diagnose .NET Core applications.
ms.date: 12/17/2019
ms.date: 07/16/2020
ms.topic: overview
#Customer intent: As a .NET Core developer I want to find the best tools to help me diagnose problems so that I can be productive.
---
Expand Down Expand Up @@ -42,3 +42,11 @@ The [dotnet-dump](dotnet-dump.md) tool is a way to collect and analyze Windows a
### Debug a memory leak

[Tutorial: Debug a memory leak](debug-memory-leak.md) walks through finding a memory leak. The [dotnet-counters](dotnet-counters.md) tool is used to confirm the leak and the [dotnet-dump](dotnet-dump.md) tool is used to diagnose the leak.

### Debug high CPU usage

[Tutorial: Debug high CPU usage](debug-highcpu.md) walks you through investigating high CPU usage. It uses the [dotnet-counters](dotnet-counters.md) tool to confirm the high CPU usage. It then walks you through using [Trace for performance analysis utility (`dotnet-trace`)](dotnet-trace.md) or Linux `perf` to collect and view CPU usage profile.

### Debug deadlock

[Tutorial: Debug deadlock](debug-deadlock.md) shows you how to use the [dotnet-dump](dotnet-dump.md) tool to investigate threads and locks.
Binary file added docs/core/diagnostics/media/flamegraph.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/core/diagnostics/media/perfview.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions docs/core/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,10 @@ items:
items:
- name: Debug a memory leak
href: diagnostics/debug-memory-leak.md
- name: Debug high CPU usage
href: diagnostics/debug-highcpu.md
- name: Debug deadlock
href: diagnostics/debug-deadlock.md
- name: .NET Core additional tools
items:
- name: Overview
Expand Down