-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[test] Continuous Performance Regression Tests #948
Comments
https://www.cnblogs.com/younggun/articles/1814989.html |
By "regression" I mean to detect performance regression (e.g. a new change caused the performance, as measured by our benchmark tests, to drop by 50%). In contrast, what we have currently in the CI are just unit tests. They are used to verify if the system is not fundamentally broken. |
Thank for clarify this, so we want to verify the functionability not broken, also want to verify the performance not broken? Not sure how Travis CI could do this, currently we can only do this by |
Yep
I'm no expert on this either. But such kind of regression tests are actually quite common, so I guess Travis must have a way to run some command, then produce a few timing numbers. I suggest we don't worry too much about this issue. We may prioritize this when Taichi is more mature. For now I'm simply creating an issue so that we don't forget :) |
I searched the web and found no info about relation between Travis and CPRT... A stright-forward attempt can be: |
Ah, the naming could be performance tests, benchmark (BM) tests... I think the terms are pretty confusing here. Yeah, I think having a file to store the historical BM data is a good way to get things on going (Usually this would be stored in some database for ease of query, but obviously we'd then have to pay for that...) I think this can be even simpler -- configure the bot so that it posts the BM data on each PR. For example: pingcap/tidb#17101 (comment) |
Cool! But I guess we will pay money for that. not sure if @yuanming-hu like this...
When clicked, it runs ti benchmark and update misc/benchmark.txt in a commit [skip ci] update benchmark just like the [skip ci] enforce code format .Or we can trigger this when user pushes [benchmark] do benchmark for me like the [format] currently does.Then the reviewers can check the Files changed page to see if the performance increased or decreased.
|
I think it's better to be funded, rather than paying out of our own pockets, even if we are very enthusiastic on this...
Yeah, i think this can be a good start.. The good thing about having a report on the PR is that people can actively look into it, though. But again, these are all fancy stuffs, which we don't need urgently |
Anyway, before these fancy stuffs, we must set up |
Great idea. Thanks for proposing this.
We can find a computer in our lab for benchmark purposes. We need a machine with consistent hardware otherwise the performance comparisons won't make much sense. I guess Travis will just randomly pick an available VM slot, whose hardware capability fluctuates. Our group also has some free Google cloud accounts. I'll think about this.
We actually already have some basic benchmarks: https://github.com/taichi-dev/taichi/tree/master/benchmarks. https://github.com/taichi-dev/taichi/blob/master/benchmarks/run.py can trigger these benchmarks:
These can be reused. For example, https://github.com/taichi-dev/taichi/blob/master/benchmarks/mpm2d.py should be able to detect the performance issue in introduced in #937. How to automatically summarize the benchmark results and display on GitHub is worth discussions. I haven't got a chance to systematically work on performance issues though. |
Currently, I'm just setting
If you want something like this, we can just let this command set |
So currently
versus:
Although the second have more statements, but it's actually more efficient than the first. Also consider vector division:
versus:
And not to mention loop unroll. So what we want is Time Performance, instead of Size Performance. I think it's good to add SP, but TP is more important for Regression Test, since sometimes we want to sacrifice SP for TP like #944. |
Yes, but we may need to solve this issue first before adding the regression test of time performance:
|
Oh, I see, so we can first setup SP CPRT as a 练手 for TP CPRT before this issue is solved? |
Concisely describe the proposed feature
I think it will be great if we can have a CI pipeline to run some benchmarks as regression tests. This way we can easily detect problems like #937 (comment).
The text was updated successfully, but these errors were encountered: