Support Multi-Stream, Single-Thread in New Executor

[Aurelius84]

PR types

New features

PR changes

Others

Describe

1. 描述

Support Multi-Stream, Single-Thread in New Executor

For Program or Graph topology:

• kernel 执行和Memcpy 在同一个stream中
• Host 端按拓扑序依次拉起 Kernel
• D2H / CPU kernel 会导致 Host 端阻塞

In this PR:

• kernel 执行单独一个 stream 流， Memcpy 单独一个 stream 流，借助 event 控制同步

2. 为什么引入h2d/d2h 算子？

本PR 新增了两个细粒度的数据拷贝的算子，目的是为了更加精细化的进行Op的管理和调度。

What's Next?

• Integrate thread pool to implement multi-thread.

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[wanghuancoder]

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__EVENT.html#group__CUDART__EVENT_1g7b317e07ff385d85aa656204b971a042
cuda官方文档中表示，对Event的初始化，使用如下flag对于我们的性能比较好：
cudaEventDisableTiming: Specifies that the created event does not need to record timing data. Events created with this flag specified and the cudaEventBlockingSync flag not specified will provide the best performance when used with cudaStreamWaitEvent() and cudaEventQuery().
是否把CudaEvent中的flag修改一下？现在是cudaEventDefault。

[Aurelius84]

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__EVENT.html#group__CUDART__EVENT_1g7b317e07ff385d85aa656204b971a042
cuda官方文档中表示，对Event的初始化，使用如下flag对于我们的性能比较好：
cudaEventDisableTiming: Specifies that the created event does not need to record timing data. Events created with this flag specified and the cudaEventBlockingSync flag not specified will provide the best performance when used with cudaStreamWaitEvent() and cudaEventQuery().
是否把CudaEvent中的flag修改一下？现在是cudaEventDefault。

是的，我这里已经做了处理。代码逻辑在这里：

auto cuda_event = std::make_shared<platform::CudaEvent>(
          platform::get_cuda_flags(false, false, false));

[wanghuancoder]

LGTM

[lanxianghit]

LGTM for new c++ operators