Fix benchmark (and migrate to ASV)

[yfukai]

Fix the benchmark and migrate to ASV

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.35%. Comparing base (06b4588) to head (b5bc04c).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #199   +/-   ##
=======================================
  Coverage   88.35%   88.35%           
=======================================
  Files          54       54           
  Lines        3856     3856           
  Branches      665      665           
=======================================
  Hits         3407     3407           
  Misses        268      268           
  Partials      181      181           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yfukai]

TODOs:

• Review asv config
• Fix workflow

[yfukai]

The benchmark artifact is still not valid but at least the benchmark task is running!

[JoOkuma]

@yfukai this is amazing!

I was able to access the artifacts they look good to me.

I left a minor comment regarding the show_progress.

Do you think we could reduce the number of samples it run the benchmark?
It took 34 min to run the whole thing.

The github summary is super useful, do you think it could be formatted in markdown?

Thanks a bunch for this PR.

[yfukai]

todo:

• organize the report
• further shorten the run for CI, looking at the env

[yfukai]

Hi @JoOkuma, I believe this is good to be merged! I gave up on asv publish since the result from the asv continuous is not really helpful. Instead I generated the comparison table in the summary. Feedbacks are welcome!

[JoOkuma]

Thanks a lot @yfukai, the ratio in the summary is super helpful!