[Regression] Local performance regression

[hjjq]

python ./scripts/regression/run.py -h at the repo root for usage.
Currently only some operator shapes and models, with 3090 latency data, are added. More operator/shapes/models/GPUs to be added.
Tolerance to report speedup/slowdown is set at +-8%. May adjust in the future accordingly.

[yaoyaoding]

Hi @hjjq, thanks!

Just one suggestion. Can we store the shapes to profile in a seperate json file instead of using the regression_data.json, which also records the profile data for previous runs.

• configs.json: store all the shape information that we want to benchmark. (This information can also be written in the python source code directly).
• regression_data.json: (optional) store the benchmark data for previous run, if have. We can generate one for the first run.

[hjjq]

Hi @yaoyaoding , I generated this current version of regression_data.json with a local run on my 3090.
What I did was similar to what triton did, where the latency numbers are stored in the repository together with the shapes.
I can create configs.json as you described, but regression_data.json will still be needed in order to compare against a reference run, and the reference latencies need to be updated in the repo every time we achieve a speedup (It will always contain the all-time-high peformance that Hidet can achieve on a particular GPU.)

[yaoyaoding]

I see, that makes sense.