Fuzzing benchmark for FFT operators

[peterbell10]

Stack from ghstack:

• Fuzzing benchmark for FFT operators #47872 Fuzzing benchmark for FFT operators
• Fix complex tensors and missing data in benchmark utility #47871 Fix complex tensors and missing data in benchmark utility

Differential Revision: D25237499

[dr-ci]

💊 CI failures summary and remediations

As of commit c547000 (more details on the Dr. CI page):

---

• 1/1 failures possibly* introduced in this PR
  • 1/1 non-CircleCI failure(s)

---

codecov.io: 1 failed

• Failed: codecov/patch

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 14 times.

[mruberry]

Hey @peterbell10!

I've asked @ngimel to suggest a reviewer for this and #47871 since she's more familiar with benchmarking.

[robieta]

At a high level this looks good. I would suggest moving SpectralOpFuzzer to torch/utils/benchmark/op_fuzzers and moving the benchmark script itself to torch/utils/benchmark/examples. I have a TODO to extend the op microbenchmarks to use more fuzzed values so having a canned fuzzer for spectral ops would be quite helpful. (As well as just generally good to have.)

The one question I have on the fuzzer is that you are restricting shapes to those that factor nicely; do you think it makes sense to also test some more pathological shapes? (Albeit a small fraction of the time.)

[peterbell10]

moving the benchmark script itself to torch/utils/benchmark/examples

Shouldn't it be somewhere in the benchmarks/ folder, or is that being phased out?

The one question I have on the fuzzer is that you are restricting shapes to those that factor nicely; do you think it makes sense to also test some more pathological shapes? (Albeit a small fraction of the time.)

In general when comparing two FFT libraries this is definitely worth doing. But in my case I want to benchmark the pytorch interface code and how it calls the underlying FFT library, not necessarily the FFT library itself. For my purposes, using non-regular sizes just increases the benchmark runtime.

Perhaps it could still be useful as a parameter to the SpectralOpFuzzer. e.g. probability_regular=0.8 similar to FuzzedTensor's probability_contiguous.

[robieta]

Shouldn't it be somewhere in the benchmarks/ folder, or is that being phased out?

I think the main thing is not having ad-hoc code in the operator benchmarks folder. @ngimel It seems like a lot of these one-off scripts have popped up recently. Any strong preferences for where they should live?

Perhaps it could still be useful as a parameter to the SpectralOpFuzzer. e.g. probability_regular=0.8 similar to FuzzedTensor's probability_contiguous.

SGTM.

[codecov]

Codecov Report

Merging #47872 (c547000) into gh/peterbell10/26/base (d47770f) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@                    Coverage Diff                     @@
##           gh/peterbell10/26/base   #47872      +/-   ##
==========================================================
- Coverage                   81.14%   81.11%   -0.04%     
==========================================================
  Files                        1839     1841       +2     
  Lines                      198447   198537      +90     
==========================================================
- Hits                       161036   161034       -2     
- Misses                      37411    37503      +92     

[peterbell10]

@ngimel, @robieta any new thoughts on where to put the benchmark files, or is examples the best place for now?

[mruberry]

@ngimel, @robieta any new thoughts on where to put the benchmark files, or is examples the best place for now?

Just fyi that the team is on holiday this week, so it may take a few (working) days to get a response.

[ngimel]

I don't have a strong preference, we can leave the code in examples.

[robieta]

LGTM. I'll start merging the stack. Thanks for your patience.

[facebook-github-bot]

@robieta merged this pull request in 74d6a61.