[logging] dynamo_timed the synchronize in CachingAutotuner make_launchers

[masnesral]

Stack from ghstack (oldest at bottom):

• -> [logging] dynamo_timed the synchronize in CachingAutotuner make_launchers #157747

Summary: There's some evidence that some very long compile times are actually attributable to the sync. This should make it easier to say for sure.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157747

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a8098df with merge base 28aae93 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[aorenste]

This is already recorded in ODS - do we want it here too? (One advantage of being here is that it'll be in the perfetto trace)

[masnesral]

This is already recorded in ODS - do we want it here too? (One advantage of being here is that it'll be in the perfetto trace)

I was only shooting for perfetto. But I didn't know it was already recorded in ODS. Where is that happening? Is there some underlying counter that captures all cuda.synchronize?

[aorenste]

This is already recorded in ODS - do we want it here too? (One advantage of being here is that it'll be in the perfetto trace)

I was only shooting for perfetto. But I didn't know it was already recorded in ODS. Where is that happening? Is there some underlying counter that captures all cuda.synchronize?

It's in c10/cuda/CUDAFunctions.cpp::device_synchronize():

STATIC_SCOPED_WAIT_COUNTER(pytorch.wait_counter.cuda_device_synchronize);

all STATIC_SCOPED_WAIT_COUNTERS automatically get logged to ods.

[masnesral]

@aorenste , ah thanks. This PR is in response to https://fburl.com/workplace/it2q0qzs. I thought it would be easier to see how much overhead there is in this specific cuda synchronize.

[masnesral]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[pytorchmergebot]

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

[masnesral]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[pytorchmergebot]

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

[masnesral]

@pytorchbot rebase

[pytorchmergebot]

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

[pytorchmergebot]

Successfully rebased gh/masnesral/216/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/157747)

[masnesral]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here