[Profiler] iterate frontend function events for profiler post processing #124596

zejun-chen · 2024-04-22T06:04:50Z

The function_events in _parse_kineto_results is used to contain all function events from the result. It contains 2 kinds of events. One is frontend function events whose correlation id is 0, for example, aten::add, aten::mul. They are on the top level of the profile results. The other is the backend events, which are associated with the frontend events and its correlation id is > 0, for example, at::native::vectorized_elementwise_kernel, it should be the backend event of a frontend element-wise op. They have the device execution duration for the related frontend op.

In the following post processing code, the frontend function events should be iterated to find its correlated backend events in device_corr_map, instead of iterating all function events, because device_corr_map is designed as a dict, whose key is the id of the frontend function event.

pytorch/torch/autograd/profiler.py

Lines 543 to 560 in 3af1244

    
           for fe in function_events: 
        
               if ( 
        
                   fe.device_type == DeviceType.CPU 
        
                   and not fe.is_async 
        
                   and fe.id in device_corr_map 
        
               ): 
        
                   for f_evt in device_corr_map[fe.id]: 
        
                       if f_evt.device_type == DeviceType.CUDA: 
        
                           fe.append_kernel( 
        
                               f_evt.name, 
        
                               f_evt.device_index, 
        
                               f_evt.time_range.end - f_evt.time_range.start, 
        
                           ) 
        
                       elif f_evt.device_type == DeviceType.CPU: 
        
                           # make sure that 'thread' of a CPU Kineto (e.g. Device Runtime) event is associated 
        
                           # with the 'thread' of the corresponding linked PyTorch event to properly track 
        
                           # parents and children 
        
                           f_evt.thread = fe.thread

pytorch/torch/autograd/profiler.py

Lines 537 to 540 in 3af1244

    
           if corr_id > 0: 
        
               if corr_id not in device_corr_map: 
        
                   device_corr_map[corr_id] = [] 
        
               device_corr_map[corr_id].append(fe)

whose correlation id is 0, then post processing is to iterate the frontend function events Signed-off-by: Chen, Zejun <zejun.chen@intel.com>

pytorch-bot · 2024-04-22T06:04:53Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/124596

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 53af78e with merge base b198423 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge) (gh)
RuntimeError: profiler/test_profiler 1/1 failed
trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge) (gh)
RuntimeError: profiler/test_profiler 1/1 failed

This comment was automatically generated by Dr. CI and updates every 15 minutes.

zejun-chen · 2024-04-22T13:09:35Z

Hi, @aaronenyeshi

Could you help review the PR?

Thank you. 👍

fwenguang · 2024-04-23T03:02:18Z

It looks like we've encountered the same issue. If this patch is merged, then the PR #124389 can be closed.

zejun-chen · 2024-04-23T03:12:21Z

It looks like we've encountered the same issue. If this patch is merged, then the PR #124389 can be closed.

Woohoo! We are solving the same issue. Thank you for help. Both PR are ok to me.
@aaronenyeshi You can review and comment both 2 PRs.

Thank you

aaronenyeshi

Both implementations are good, let's go and merge this since its easier reading the comments and variable names.

aaronenyeshi · 2024-04-23T14:11:38Z

@pytorchbot merge

pytorchmergebot · 2024-04-23T14:13:46Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…ing (pytorch#124596) The `function_events` in `_parse_kineto_results` is used to contain all function events from the result. It contains 2 kinds of events. One is frontend function events whose correlation id is 0, for example, `aten::add`, `aten::mul`. They are on the top level of the profile results. The other is the backend events, which are associated with the frontend events and its correlation id is > 0, for example, `at::native::vectorized_elementwise_kernel`, it should be the backend event of a frontend element-wise op. They have the device execution duration for the related frontend op. In the following post processing code, the **frontend function events** should be iterated to find its correlated backend events in `device_corr_map`, instead of iterating all function events, because `device_corr_map` is designed as a dict, whose key is the id of the frontend function event. https://github.com/pytorch/pytorch/blob/3af12447f85dfede191a113c052e58fa7b21a8b3/torch/autograd/profiler.py#L543-L560 https://github.com/pytorch/pytorch/blob/3af12447f85dfede191a113c052e58fa7b21a8b3/torch/autograd/profiler.py#L537-L540 Pull Request resolved: pytorch#124596 Approved by: https://github.com/aaronenyeshi

[Profiler] divide all the function events to frontend event lists,

53af78e

whose correlation id is 0, then post processing is to iterate the frontend function events Signed-off-by: Chen, Zejun <zejun.chen@intel.com>

zejun-chen requested a review from aaronenyeshi as a code owner April 22, 2024 06:04

zejun-chen changed the title ~~[Profiler] divide all the function events to frontend event lists, whose correlation id is 0~~ [Profiler] iterate frontend function events for profiler post processing Apr 22, 2024

pytorchbot added the open source label Apr 22, 2024

colesbury added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Apr 22, 2024

aaronenyeshi approved these changes Apr 23, 2024

View reviewed changes

aaronenyeshi added release notes: profiler release notes category topic: improvements topic category labels Apr 23, 2024

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Apr 23, 2024

aaronenyeshi mentioned this pull request Apr 23, 2024

fix appending kernel to an unrelated runtime event #124389

Closed

pytorchmergebot added the merging label Apr 23, 2024

pytorchmergebot added the Merged label Apr 23, 2024

pytorchmergebot closed this in 25a2d18 Apr 23, 2024

pytorchmergebot removed the merging label Apr 23, 2024

fwenguang mentioned this pull request Apr 24, 2024

[Profiler] Maybe append a kernel to an unrelated event. #124388

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Profiler] iterate frontend function events for profiler post processing #124596

[Profiler] iterate frontend function events for profiler post processing #124596

zejun-chen commented Apr 22, 2024 •

edited

pytorch-bot bot commented Apr 22, 2024 •

edited

zejun-chen commented Apr 22, 2024

fwenguang commented Apr 23, 2024

zejun-chen commented Apr 23, 2024 •

edited

aaronenyeshi left a comment

aaronenyeshi commented Apr 23, 2024

pytorchmergebot commented Apr 23, 2024

	for fe in function_events:
	if (
	fe.device_type == DeviceType.CPU
	and not fe.is_async
	and fe.id in device_corr_map
	):
	for f_evt in device_corr_map[fe.id]:
	if f_evt.device_type == DeviceType.CUDA:
	fe.append_kernel(
	f_evt.name,
	f_evt.device_index,
	f_evt.time_range.end - f_evt.time_range.start,
	)
	elif f_evt.device_type == DeviceType.CPU:
	# make sure that 'thread' of a CPU Kineto (e.g. Device Runtime) event is associated
	# with the 'thread' of the corresponding linked PyTorch event to properly track
	# parents and children
	f_evt.thread = fe.thread

	if corr_id > 0:
	if corr_id not in device_corr_map:
	device_corr_map[corr_id] = []
	device_corr_map[corr_id].append(fe)

[Profiler] iterate frontend function events for profiler post processing #124596

[Profiler] iterate frontend function events for profiler post processing #124596

Conversation

zejun-chen commented Apr 22, 2024 • edited

pytorch-bot bot commented Apr 22, 2024 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/124596

✅ You can merge normally! (2 Unrelated Failures)

zejun-chen commented Apr 22, 2024

fwenguang commented Apr 23, 2024

zejun-chen commented Apr 23, 2024 • edited

aaronenyeshi left a comment

Choose a reason for hiding this comment

aaronenyeshi commented Apr 23, 2024

pytorchmergebot commented Apr 23, 2024

Merge started

zejun-chen commented Apr 22, 2024 •

edited

pytorch-bot bot commented Apr 22, 2024 •

edited

zejun-chen commented Apr 23, 2024 •

edited