Add different pattern to show op in timeline

[zhuzilin]

This is a PR from JIZHI, the AI platform in Tencent.

When using timeline in tensorflow, we often observe large blanks in the "/job" row, which should be showing the consecutive execution of operators.

The following are timelines for transformer and transformer using XLA:
Transformer:

XLA:

The timeline is somehow confusing and those blanks may leads to misunderstanding that there are gpus hanging freely.

The reason for this issue is that the "/job" row only shows the scheduling time for each op. As a result, the async kernels may not start to compute by the time the scheduling of its op is over. For the transformer example, the matmul in embedding takes long and blocks other kernels, which results in the large blank in the middle. And for XLA, those fused kernels are scheduled early but have to wait for execution, which results in the large gap between the ending in "/job" and "/stream:all".

Therefore, we propose 2 new pattern to show the op execution time.

• "gpu" pattern will align op with the execution span of all its kernels
• "all" pattern will only change the ending time of the op to the ending of its last kernel.

we added a new argument op_time to function generate_chrome_trace_format to let user select how op execution time will be shown. The default value is "schedule" which behaves the same as before. And other possible values are "gpu" and "all" as explained above.

  def generate_chrome_trace_format(self, show_dataflow=True, show_memory=False, op_time="schedule"):

The result of "gpu" pattern is:
Transformer:

XLA:

And the result of "all" pattern is
Transformer:

XLA:

Notice that the above illustrations have only shown part of the "all" pattern since it may induce large parallel (many op are waiting to be executed at the same time.)

Additionally, the kernel name of an XLA fusion kernel is not parsed correctly, and because we do not have plans on changing the C++ part of the profiler, we reparsed it using the timeline_label attribute provided in RunMetadata.

Thank you for your time on this review.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[zhuzilin]

@googlebot I signed it!

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[yifeif]

Thanks for the PR @zhuzilin. @prb12, would you mind taking a look?

[xinan-jiang]

@prb12 do you see this pull request?

[yifeif]

@xinan-jiang, sorry, looks like @prb12 no longer works on TF anymore. @qiuminxu do you know if anybody on the performance side can take a look at this PR? Thank you!

[qiuminxu]

Thanks for sending the PR.

[zhuzilin]

Thanks for sending the PR.

Thank you for your review. I have modified the file according to the advice. Could you have a second look? Thank you!

[zhuzilin]

@google-admin @googlebot Could you help me check what's wrong with the cla check on this pr? The same github account and email address are used on #37813 and that pr passed the cla check.

[qiuminxu]

Can you run the check again and see if it passes?

[zhuzilin]

@googlebot I signed it!

[zhuzilin]

@qiuminxu Could you help me check what's the migration error in the check "import/copybara"? Thank you!

[qiuminxu]

@yifeif I couldn't see the details of the test error, can someone help take a look?

[gbaned]

@mihaimaruseac Can you please take a look on cla/google test failure? Thanks!