PTL supports running tests multiple times #1908
Conversation
test/fw/bin/pbs_benchpress
Outdated
| @@ -145,6 +145,8 @@ def usage(): | |||
| 'fails\n'] | |||
| msg += ['--timeout=<seconds>: duration after which no test suites are ' | |||
| 'run\n'] | |||
| msg += ['--repeat-tests=<int>: number of times test is going to repeat\n'] | |||
There was a problem hiding this comment.
How about --repeat-count=<count>: repeat given all tests <count> times\n?
test/fw/bin/pbs_benchpress
Outdated
| @@ -145,6 +145,8 @@ def usage(): | |||
| 'fails\n'] | |||
| msg += ['--timeout=<seconds>: duration after which no test suites are ' | |||
| 'run\n'] | |||
| msg += ['--repeat-tests=<int>: number of times test is going to repeat\n'] | |||
| msg += ['--repeat-tests-delay=<seconds>: delay between two repetition\n'] | |||
There was a problem hiding this comment.
I think --repeat-delay will be a better name
| test(result) | ||
| for i in range(self.repetition): | ||
| self.logger.info("====================================") | ||
| self.logger.info("tests are running (" + str(i + 1) + ") time") |
There was a problem hiding this comment.
self.logger.info("Running tests %d time" % i + 1)
There was a problem hiding this comment.
I don't think you need both this message and the below one. We only need to be told once that the tests were repeated N times.
| self.logger.info("====================================") | ||
| self.logger.info("tests are running (" + str(i + 1) + ") time") | ||
| self.logger.info("====================================") | ||
| if i is not 0: |
There was a problem hiding this comment.
I suggest not to use is not (py3.8 gives a warning for it), also you can do if i != 0: here
| time.sleep(self.repetition_delay) | ||
| test(result) | ||
| self.logger.info("=============================================") | ||
| self.logger.info(" All tests are repeated " + |
There was a problem hiding this comment.
self.logger.info("All Tests are repeated %d times" % self.repeat_count)
Also shouldn't we print this message only if repeat count is > 0?
There was a problem hiding this comment.
Actually repeat_count = 1 by default, so it should be > 1.
| self.logger = logging.getLogger(__name__) | ||
| self.result = None | ||
| self.repetition = repetition |
There was a problem hiding this comment.
how about repeat_count as a variable name? and for below repeat_delay
|
Also, PR title and commit-msg can better like |
birbalsain
changed the title
|
Thanks @hirenvadalia @bhroam , I have done the changes. |
Describe Bug or Feature
Test Framework should allow testers to run a test for multiple times. Tester can use delay between two repetition.
Describe Your Change
Introduces new pbs_benchpress options --repeat_tests= for how many times test will run and
--repeat_tests_delay for delay= delay between repeated test.
Each repetition includes all submitted tests. next repetition starts after finishing of all tests of first repetition.
If submits three tests then in first repetition all three tests run and after finishing all three tests, next repetition starts and again all three tests run.
At the end we get message " All tests are repeated times"
Link to Design Doc
https://pbspro.atlassian.net/wiki/spaces/PD/pages/1386938446/Test+Framework+should+allow+testers+to+run+a+test+for+multiple+times
Attach Test and Valgrind Logs/Output
repetition_mixed_testsuite.txt
repetition_multiple_tests.txt
no_repetition.txt
json_report.txt