New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use libkineto in profiler #46470
Closed
Closed
Use libkineto in profiler #46470
Changes from 1 commit
Commits
Show all changes
91 commits
Select commit
Hold shift + click to select a range
a4d4124
Use libkineto in profiler
ilia-cher e27f74c
Update on "Use libkineto in profiler"
ilia-cher 5c3833e
Update on "Use libkineto in profiler"
ilia-cher 662431b
Update on "Use libkineto in profiler"
ilia-cher ea956aa
Update on "Use libkineto in profiler"
ilia-cher 7dfdbc9
Update on "Use libkineto in profiler"
ilia-cher 6725778
Update on "Use libkineto in profiler"
ilia-cher e9a219b
Update on "Use libkineto in profiler"
ilia-cher 49a9fee
Update on "Use libkineto in profiler"
ilia-cher 8edb346
Update on "Use libkineto in profiler"
ilia-cher f288623
Update on "Use libkineto in profiler"
ilia-cher 979cdfa
Update on "Use libkineto in profiler"
ilia-cher c8cbeb0
Update on "Use libkineto in profiler"
ilia-cher 226089c
Update on "Use libkineto in profiler"
ilia-cher 266b75f
Update on "Use libkineto in profiler"
ilia-cher 6958eac
Update on "Use libkineto in profiler"
ilia-cher 97e5070
Update on "Use libkineto in profiler"
ilia-cher 8d111d2
Update on "Use libkineto in profiler"
ilia-cher bfb0360
Update on "Use libkineto in profiler"
ilia-cher 1ff1a12
Update on "Use libkineto in profiler"
ilia-cher b3b69d8
Update on "Use libkineto in profiler"
ilia-cher 2faeb8a
Update on "Use libkineto in profiler"
ilia-cher 67c890d
Update on "Use libkineto in profiler"
ilia-cher ed8babe
Update on "Use libkineto in profiler"
ilia-cher ffc11fd
Update on "Use libkineto in profiler"
ilia-cher fe76b84
Update on "Use libkineto in profiler"
ilia-cher 76ee80c
Update on "Use libkineto in profiler"
ilia-cher 5761ea2
Update on "Use libkineto in profiler"
ilia-cher dde5ec3
Update on "Use libkineto in profiler"
ilia-cher 3a25bd2
Update on "Use libkineto in profiler"
ilia-cher 6023998
Update on "Use libkineto in profiler"
ilia-cher 0bc66a6
Update on "Use libkineto in profiler"
ilia-cher aa2d09e
Update on "Use libkineto in profiler"
ilia-cher 91718ac
Update on "Use libkineto in profiler"
ilia-cher 1556a7c
Update on "Use libkineto in profiler"
ilia-cher 4a0fec9
Update on "Use libkineto in profiler"
ilia-cher bb6396a
Update on "Use libkineto in profiler"
ilia-cher 60b5dee
Update on "Use libkineto in profiler"
ilia-cher e1a5480
Update on "Use libkineto in profiler"
ilia-cher 38a37dd
Update on "Use libkineto in profiler"
ilia-cher c6c6039
Update on "Use libkineto in profiler"
ilia-cher 17767d1
Update on "Use libkineto in profiler"
ilia-cher 3537e9d
Update on "Use libkineto in profiler"
ilia-cher 043dcd2
Update on "Use libkineto in profiler"
ilia-cher aa17339
Update on "Use libkineto in profiler"
ilia-cher 9262f92
Update on "Use libkineto in profiler"
ilia-cher 8371b33
Update on "Use libkineto in profiler"
ilia-cher 67d4acb
Update on "Use libkineto in profiler"
ilia-cher 9f1d24f
Update on "Use libkineto in profiler"
ilia-cher e864205
Update on "Use libkineto in profiler"
ilia-cher 380b874
Update on "Use libkineto in profiler"
ilia-cher 165bb7c
Update on "Use libkineto in profiler"
ilia-cher 445b8c1
Update on "Use libkineto in profiler"
ilia-cher 7c317f5
Update on "Use libkineto in profiler"
ilia-cher c904443
Update on "Use libkineto in profiler"
ilia-cher 1f600f8
Update on "Use libkineto in profiler"
ilia-cher 5aacc1c
Update on "Use libkineto in profiler"
ilia-cher 651f556
Update on "Use libkineto in profiler"
ilia-cher 9997011
Update on "Use libkineto in profiler"
ilia-cher cfd0424
Update on "Use libkineto in profiler"
ilia-cher 30114d8
Update on "Use libkineto in profiler"
ilia-cher 27e4e9c
Update on "Use libkineto in profiler"
ilia-cher bde96f6
Update on "Use libkineto in profiler"
ilia-cher b1a0292
Update on "Use libkineto in profiler"
ilia-cher b7fda07
Update on "Use libkineto in profiler"
ilia-cher 459df8e
Update on "Use libkineto in profiler"
ilia-cher 09a4762
Update on "Use libkineto in profiler"
ilia-cher cafee0f
Update on "Use libkineto in profiler"
ilia-cher 39ff2b3
Update on "Use libkineto in profiler"
ilia-cher 5502837
Update on "Use libkineto in profiler"
ilia-cher 7c2017b
Update on "Use libkineto in profiler"
ilia-cher 525e5b5
Update on "Use libkineto in profiler"
ilia-cher 1f50e4b
Update on "Use libkineto in profiler"
ilia-cher f70a95c
Update on "Use libkineto in profiler"
ilia-cher 5fed8be
Update on "Use libkineto in profiler"
ilia-cher 2494879
Update on "Use libkineto in profiler"
ilia-cher 4f401ff
Update on "Use libkineto in profiler"
ilia-cher c689e6b
Update on "Use libkineto in profiler"
ilia-cher 4a5632f
Update on "Use libkineto in profiler"
ilia-cher 6d0e7ab
Update on "Use libkineto in profiler"
ilia-cher 95b686f
Update on "Use libkineto in profiler"
ilia-cher d98a5fb
Update on "Use libkineto in profiler"
ilia-cher cb7367e
Update on "Use libkineto in profiler"
ilia-cher 5ad0a34
Update on "Use libkineto in profiler"
ilia-cher d6bd96e
Update on "Use libkineto in profiler"
ilia-cher 0c4faaa
Update on "Use libkineto in profiler"
ilia-cher ab754e1
Update on "Use libkineto in profiler"
ilia-cher aee38e8
Update on "Use libkineto in profiler"
ilia-cher 671785f
Update on "Use libkineto in profiler"
ilia-cher 8fde042
Update on "Use libkineto in profiler"
ilia-cher ca6cb73
Update on "Use libkineto in profiler"
ilia-cher File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -364,32 +364,56 @@ def __init__( | |
use_cuda=False, | ||
record_shapes=False, | ||
profile_memory=False, | ||
with_stack=False): | ||
with_stack=False, | ||
use_kineto=False): | ||
self.enabled = enabled | ||
self.use_cuda = use_cuda | ||
self.function_events = None | ||
if not self.enabled: | ||
return | ||
self.use_cuda = use_cuda | ||
self.function_events = None | ||
self.entered = False | ||
self.record_shapes = record_shapes | ||
self.profile_memory = profile_memory | ||
self.with_stack = with_stack | ||
self.use_kineto = use_kineto | ||
|
||
self.profiler_kind = None | ||
self.kineto_activities = [] | ||
if self.use_kineto: | ||
if self.use_cuda: | ||
self.profiler_kind = torch.autograd.ProfilerState.KINETO | ||
self.kineto_activities = [ | ||
torch.autograd.ProfilerActivity.CPU, | ||
# uses CUPTI | ||
torch.autograd.ProfilerActivity.CUDA_RUNTIME, | ||
torch.autograd.ProfilerActivity.CUDA] | ||
else: | ||
# intially we're not using Kineto for CPU only case | ||
ilia-cher marked this conversation as resolved.
Show resolved
Hide resolved
|
||
self.profiler_kind = torch.autograd.ProfilerState.CPU | ||
elif self.use_cuda: | ||
# legacy CUDA mode | ||
self.profiler_kind = torch.autograd.ProfilerState.CUDA | ||
else: | ||
self.profiler_kind = torch.autograd.ProfilerState.CPU | ||
self.kineto_activities = set(self.kineto_activities) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit - it could be a set from the very beginning (and use |
||
|
||
if self.profiler_kind == torch.autograd.ProfilerState.KINETO: | ||
assert torch.autograd.kineto_available() | ||
ilia-cher marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
self.config = torch.autograd.ProfilerConfig( | ||
self.profiler_kind, | ||
self.record_shapes, | ||
self.profile_memory, | ||
self.with_stack) | ||
|
||
def __enter__(self): | ||
if not self.enabled: | ||
return | ||
if self.entered: | ||
raise RuntimeError("autograd profiler traces are not reentrant") | ||
self.entered = True | ||
profiler_kind = torch.autograd.ProfilerState.CUDA if self.use_cuda \ | ||
else torch.autograd.ProfilerState.CPU | ||
|
||
config = torch.autograd.ProfilerConfig( | ||
profiler_kind, | ||
self.record_shapes, | ||
self.profile_memory, | ||
self.with_stack) | ||
torch.autograd._enable_profiler(config) | ||
torch.autograd._prepare_profiler(self.config, self.kineto_activities) | ||
torch.autograd._enable_profiler(self.config) | ||
return self | ||
|
||
def __exit__(self, exc_type, exc_val, exc_tb): | ||
|
@@ -732,7 +756,7 @@ class FunctionEvent(FormattedTimesMixin): | |
def __init__( | ||
self, id, node_id, name, thread, cpu_start, cpu_end, fwd_thread=None, input_shapes=None, | ||
stack=None, scope=0, cpu_memory_usage=0, cuda_memory_usage=0, is_async=False, | ||
is_remote=True, sequence_nr=-1): | ||
is_remote=True, sequence_nr=-1, device_id=-1): | ||
self.id: int = id | ||
self.node_id: int = node_id | ||
self.name: str = name | ||
|
@@ -751,6 +775,7 @@ def __init__( | |
self.is_async: bool = is_async | ||
self.is_remote: bool = is_remote | ||
self.sequence_nr: int = sequence_nr | ||
self.device_id: int = device_id | ||
ilia-cher marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
def append_kernel(self, name, device, start, end): | ||
self.kernels.append(Kernel(name, device, Interval(start, end))) | ||
|
@@ -802,15 +827,21 @@ def self_cpu_time_total(self): | |
|
||
@property | ||
def cuda_time_total(self): | ||
if self.device_id >= 0: | ||
ilia-cher marked this conversation as resolved.
Show resolved
Hide resolved
|
||
return self.cpu_interval.elapsed_us() | ||
return sum(kinfo.interval.elapsed_us() for kinfo in self.kernels) | ||
|
||
@property | ||
def self_cuda_time_total(self): | ||
if self.device_id >= 0: | ||
return self.cuda_time_total - sum([child.cuda_time_total for child in self.cpu_children]) | ||
return sum(kinfo.interval.elapsed_us() for kinfo in self.kernels) - \ | ||
sum([child.cuda_time_total for child in self.cpu_children]) | ||
|
||
@property | ||
def cpu_time_total(self): | ||
if self.device_id >= 0: | ||
return 0 | ||
ilia-cher marked this conversation as resolved.
Show resolved
Hide resolved
|
||
return self.cpu_interval.elapsed_us() | ||
|
||
@property | ||
|
@@ -1045,6 +1076,7 @@ def adjusted_time(cuda_record, cuda_records_map): | |
is_async=is_async, | ||
is_remote=is_remote_event, | ||
sequence_nr=start.sequence_nr(), | ||
device_id=start.device_id(), | ||
) | ||
# note: async events have only cpu total time | ||
if not is_async and start.has_cuda(): | ||
|
@@ -1180,7 +1212,9 @@ def build_table( | |
has_input_shapes = any( | ||
[(event.input_shapes is not None and len(event.input_shapes) > 0) for event in events]) | ||
|
||
MAX_NAME_COLUMN_WIDTH = 55 | ||
name_column_width = max([len(evt.key) for evt in events]) + 4 | ||
name_column_width = min(name_column_width, MAX_NAME_COLUMN_WIDTH) | ||
|
||
DEFAULT_COLUMN_WIDTH = 12 | ||
|
||
|
@@ -1288,8 +1322,11 @@ def append(s): | |
continue | ||
else: | ||
event_limit += 1 | ||
name = evt.key | ||
if len(name) >= MAX_NAME_COLUMN_WIDTH-3: | ||
name = name[:(MAX_NAME_COLUMN_WIDTH-3)] + "..." | ||
row_values = [ | ||
evt.key, # Name | ||
name, | ||
# Self CPU total, 0 for async events. % | ||
format_time_share(evt.self_cpu_time_total, | ||
self_cpu_time_total), | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you technically don't need a field on
self
- you can just checkself.kineto_activities
being non-empty below