【Op Profiling】Add operator run time profiling feature

[lchdl]

PR types

New features

PR changes

APIs

Description

• Added op profiling support.
• Added two Python APIs (for dev):
  1. measure_program_real_op_cost
  2. check_if_op_supports_runtime_profiling
     use
     from paddle.distributed.auto_parallel.static.cost import measure_program_real_op_cost
     and
     from paddle.distributed.auto_parallel.static.cost import check_if_op_supports_runtime_profiling
     to import them.
• To obtain operator (op) run time measured in microseconds (us), use:
  op.dist_attr.run_time_us, or
  program.global_block().ops[i].dist_attr.run_time_us
  to obtain op run time from a profiled program object.
• NOTE: currently run time profiling feature is not supported for communication ops (such as "c_*", "send*", "recv*") since their run times are mainly determined by other ops. To check if an op supports run time profiling feature, use:
  check_if_op_supports_runtime_profiling(op)

---

• 添加算子级运行时间性能分析功能，来对任意program对象进行op级性能剖析。
• 添加了两个面向开发者的Python端API：
  1. measure_program_real_op_cost
  2. check_if_op_supports_runtime_profiling
     请通过
     from paddle.distributed.auto_parallel.static.cost import measure_program_real_op_cost
     from paddle.distributed.auto_parallel.static.cost import check_if_op_supports_runtime_profiling
     来导入这两个API.
• 对任意op使用
  op.dist_attr.run_time_us
  获取其具体运行时间（微秒），或者使用
  program.global_block().ops[i].dist_attr.run_time_us
  来从已被profile的program对象中获取op实际运行时间
• 对任意op使用check_if_op_supports_runtime_profiling(op)来检查其是否支持runtime profiling feature，当前该特性暂不支持所有与通信有关的算子

PCard-71568

[From00]

相同逻辑的代码拷贝太多了，考虑下代码复用。

[From00]

utils.py里添加的代码需要优化，许多实现过于复杂。如果添加的代码太多，可以直接写成一个独立模块。

[From00]

utils.py新增的功能，代码行数比较多，建议拆分成一个独立的模块

[From00]

LGTM