Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add @count util to torch, use it to track benchmark stats #93013

Closed
wants to merge 6 commits into from
Closed

Add @count util to torch, use it to track benchmark stats #93013

wants to merge 6 commits into from

Conversation

voznesenskym
Copy link
Contributor

@voznesenskym voznesenskym commented Jan 25, 2023 •
edited

@voznesenskym
wip stats
4a1d2ff 
lint

[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Jan 25, 2023 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/93013

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 41c25ce:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added release notes: fx release notes category labels Jan 25, 2023
voznesenskym added a commit that referenced this pull request Jan 25, 2023
@voznesenskym
wip stats
e959714 
lint

ghstack-source-id: 5d95fb66493fe3e8dfa9fefd0e32d81a5146676d
Pull Request resolved: #93013
@voznesenskym
Update on "wip stats"
07a21b3 
lint

cc mlazos soumith yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen chunyuan-w XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire

[ghstack-poisoned]
voznesenskym added a commit that referenced this pull request Jan 25, 2023
@voznesenskym
wip stats
7b70431 
lint

ghstack-source-id: 9bfc70d6a5fc3288cc749648356d730ed90199da
Pull Request resolved: #93013

rm unused
@voznesenskym
Update on "wip stats"
f8aacfc 
lint

cc mlazos soumith yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen chunyuan-w XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire

[ghstack-poisoned]
voznesenskym added a commit that referenced this pull request Jan 25, 2023
@voznesenskym
wip stats
ba66da2 
lint

ghstack-source-id: b6cc5d1dbdda5c8d696255260043ca5b39a7b075
Pull Request resolved: #93013

rm unused

stats fix
@voznesenskym
Update on "wip stats"
8956546 
lint

cc mlazos soumith yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen chunyuan-w XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire

[ghstack-poisoned]
voznesenskym added a commit that referenced this pull request Jan 25, 2023
@voznesenskym
wip stats
8344bb6 
lint

ghstack-source-id: ce037814c385b03ae7180b9af7cbd0d2d11ed20e
Pull Request resolved: #93013

rm unused

stats fix

Undo
@voznesenskym voznesenskym changed the title wip stats Add @count util to torch, use it to track benchmark stats Jan 25, 2023
@voznesenskym
Update on "Add @count util to torch, use it to track benchmark stats"
dc33cc4 
lint

cc mlazos soumith yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen chunyuan-w XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire

[ghstack-poisoned]
voznesenskym added a commit that referenced this pull request Jan 25, 2023
@voznesenskym
wip stats
98e5c66 
lint

ghstack-source-id: fcd668ba8597fcacd87e79177e5312d63bfceb17
Pull Request resolved: #93013

rm unused

stats fix

Undo

undo
@ezyang
Copy link
Contributor

ezyang commented Jan 25, 2023

This seems fine but:

  1. It is duplicative with the existing compiler counters in Dynamo, and
  2. The context manager is needlessly costly when you can do a less elegant direct increment call inside the body of the function you want to count, which will only be a bytecode

@voznesenskym
Copy link
Contributor Author

This seems fine but:

  1. It is duplicative with the existing compiler counters in Dynamo, and

These cant be trivially imported without a bunch of import shuffling, and there is no simple counter here as I could see

  1. The context manager is needlessly costly when you can do a less elegant direct increment call inside the body of the function you want to count, which will only be a bytecode

This is not a context manager, just a decorator, its functionally equivalent to doing it within the body of the function, right?

ezyang
ezyang reviewed Jan 25, 2023
View reviewed changes
benchmarks/dynamo/common.py Outdated

print_time_report()
stats = f"STATS: call_* op count: {op_count}"
for key, value in simple_call_counter.items():
stats = f"{stats} | {key}:{value}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: " | ".join(f"{key}:{value}" for key, value in simple_call_counter.items()), mostly to get rid of the N^2 reinterpolation and it's about the same length of code nayway

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it doesn't matter, I just reflexively hate n^2 code

ezyang
ezyang reviewed Jan 25, 2023
View reviewed changes
torch/utils/_stats.py
@functools.wraps(fn)
def wrapper(*args, **kwargs):
if fn.__qualname__ not in simple_call_counter:
simple_call_counter[fn.__qualname__] = 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One downside to the QOL fix of avoiding having to type qualname is it makes it less easy to grep the codebase to answer the question "given this stat, what line of code am I actually measuring hits for"

@voznesenskym
Update on "Add @count util to torch, use it to track benchmark stats"
41c25ce 
<img width="1333" alt="image" src="https://user-images.githubusercontent.com/4755252/214687911-f766f072-c162-4298-9aed-c889f1375336.png">

cc mlazos soumith yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen chunyuan-w XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire

[ghstack-poisoned]
voznesenskym added a commit that referenced this pull request Jan 25, 2023
@voznesenskym
wip stats
357f496 
lint

ghstack-source-id: 406e8b4342702845d8564073468eb20a84011228
Pull Request resolved: #93013

rm unused

stats fix

Undo

undo

Feedback and lint
@voznesenskym
Copy link
Contributor Author

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 25, 2023
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@facebook-github-bot facebook-github-bot deleted the gh/voznesenskym/43/head branch June 8, 2023 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ezyang ezyang ezyang approved these changes

@albanD albanD Awaiting requested review from albanD

@antoniojkim antoniojkim Awaiting requested review from antoniojkim

@bdhirsh bdhirsh Awaiting requested review from bdhirsh

@Chillee Chillee Awaiting requested review from Chillee

@jbschlosser jbschlosser Awaiting requested review from jbschlosser

@miladm miladm Awaiting requested review from miladm

@SherlockNoMad SherlockNoMad Awaiting requested review from SherlockNoMad

@wconstab wconstab Awaiting requested review from wconstab

Assignees
No one assigned
Labels
ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: dynamo release notes: fx release notes category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@voznesenskym @ezyang @pytorchmergebot