Skip to content

Commit

Permalink
Dev support oneflow insight (#10370)
Browse files Browse the repository at this point in the history
本Pr实现了OneFlow
Insight模块,相关issue:Oneflow-Inc/OneTeam#2162

当我们需要profiling cuda
kernel执行时间/瓶颈分析时,通常会基于nvidia提供的nsys指令,生成对应的profile文件(早期的.qdrep以及现在的.nsys-rep)并用Nvidia的GUI软件Nsight
Systems来可视化分析、查看。

在nsys生成profile文件的同时,还会生成平台无关的数据信息,记录在.sqlite文件中,OneFlow
Insight模块就可以通过解析.sqlite,来生成符合Google Chrome Trace
Event格式的JSON文件,使得可以直接通过Chrome或者Edge浏览器,通过`chrome://tracing/` 或
`edge://tracing/`来解析和渲染此JSON文件,从而进行可视化分析、查看,效果如下:
<img width="1320" alt="image"
src="https://github.com/Oneflow-Inc/oneflow/assets/28823622/cbfab9bc-47bd-474c-8f39-e145348db17d">

---------

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
  • Loading branch information
Flowingsun007 and oneflow-ci-bot committed Dec 29, 2023
1 parent a0f2122 commit 82c965b
Show file tree
Hide file tree
Showing 4 changed files with 557 additions and 0 deletions.
36 changes: 36 additions & 0 deletions python/oneflow/utils/insight/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# OneFlow Insight

## Overview

OneFlow Insight is a module designed for profiling CUDA kernel execution time and bottleneck analysis. Typically, this is done using the nsys command provided by Nvidia, which generates corresponding profile files (formerly .qdrep and now .nsys-rep). These files can be visualized and analyzed using Nvidia's GUI software, Nsight Systems.

In addition to generating profile files, nsys also produces platform-independent data information recorded in a .sqlite file. The OneFlow Insight module can parse this .sqlite file to generate a JSON file formatted according to the Google Chrome Trace Event standard. This allows for direct visualization and analysis through Chrome or Edge browsers using chrome://tracing/ or edge://tracing/ (supported by trace-event-profiling-tool, see:https://www.chromium.org/developers/how-tos/trace-event-profiling-tool/).


## Usage

1. Generate profile files using the following nsys command:

```bash
nsys profile --export=sqlite -o profile_data
```

This will produce .nsys-rep files along with a .sqlite file.

2. Use OneFlow Insight to parse the .sqlite file and generate a JSON file:

```bash
python3 sqlite_to_google_trace_event.py --input 'profile_data.sqlite' -o trace.json
```

3. Open Chrome or Edge browser and navigate to chrome://tracing/ or edge://tracing/.

4. Load the generated trace.json file for visualizing and analyzing the profiling data.

## Visualization Example

![OneFlow Insight Visualization](trace.json.png)

The above image demonstrates the visualization capabilities using Chrome or Edge browser with the generated JSON file.

Feel free to explore and gain insights into your CUDA kernel execution performance!
3 changes: 3 additions & 0 deletions python/oneflow/utils/insight/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
sqlite3
argparse
traceback
Loading

0 comments on commit 82c965b

Please sign in to comment.