[CI] Collect inductor max-autotune performance every Sunday

[desertfire]

Stack from ghstack (oldest at bottom):

• -> [CI] Collect inductor max-autotune performance every Sunday #99387

cc @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99387

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 Failures

As of commit 1affb96:

BROKEN TRUNK - The following jobs failed but were present on the merge base 7ff1f3f:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench_dynamic, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
• cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[malfet]

Instead of creating a whole new workflow, wouldn't it be better to just condition one of the jobs to run only on Sunday, something like if: github.event.schedule == '0 0 * * 0'

[huydhn]

Do you also need to add inductor-A100-max-autotune-weekly to https://github.com/pytorch/pytorch/blob/main/.github/workflows/upload-torch-dynamo-perf-stats.yml#L5 to get the performance results sent to Rockset?

[desertfire]

Yes, we will need that, but it might need some twist on the UI. Weekly dots vs. daily lines.

[desertfire]

@huydhn , I will leave the upload script as is for now and let's work out the details on how to distinguish the data from nightly and weekly.

[huydhn]

Instead of creating a whole new workflow, wouldn't it be better to just condition one of the jobs to run only on Sunday, something like if: github.event.schedule == '0 0 * * 0'

+1. This is a less-known fact that there can be more than one schedule on the workflow, i.e. https://github.com/pytorch/pytorch/blob/main/.github/workflows/periodic.yml#L5-L6, and the schedule can be used as a condition https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule.

Although this is fine too as I have a feeling that combine everything in the same test config matrix wouldn't be that easy

[desertfire]

Instead of creating a whole new workflow, wouldn't it be better to just condition one of the jobs to run only on Sunday, something like if: github.event.schedule == '0 0 * * 0'

We may need a separate workflow name to make sure dashboard can treat the data differently? cc @huydhn
Another reason is user will be able to manually start a max-autotune run with this separate workflow.

[huydhn]

We may need a separate workflow name to make sure dashboard can treat the data differently? cc @huydhn Another reason is user will be able to manually start a max-autotune run with this separate workflow.

The second part about manual dispatch makes sense. About the first part, this sounds like something to be fixed on my part as I don't use the workflow name at the moment anywhere in the dashboard. It means that inductor_with_cudagraphs running with Torchbench for example wouldn't know if the records come with or without max autotune enabled. So I agree that having a different name would be needed if the rest of the configs is the same (same set of compilers, same suites, same tests)

[desertfire]

@pytorchbot merge -f "broken trunk; this PR only affects perf test"

[pytorchmergebot]

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here