Limit the number of test steps/results that can be created #2322
Conversation
Codecov Report
@@ Coverage Diff @@
## master #2322 +/- ##
=======================================
Coverage 94.97% 94.98%
=======================================
Files 155 155
Lines 15189 15216 +27
=======================================
+ Hits 14426 14453 +27
Misses 763 763
|
|
This results in a test failure (not incomplete) with followed by a backtrace. Not the nicest outcome but I guess acceptable. I'm not even sure whether we should consider this an incomplete because this actually is a test mistake (test produces too many results). In my local test I've noticed that the limit of 10000 is actually nothing. The web UI can still handle it quite well and it doesn't look that huge. In fact, I'm afraid we'd run into that limit in production jobs and would fail those jobs quite needlessly. So I'll do some more local testing to determine a more sensible limit. My findings so far:
|
I would favor "incomplete". When we implemented the "reason" (mostly you did, of course) we decided that incomplete with specific reasons is fine even if the instance admins are not the right target audience but test maintainers. Think of examples like "missing asset", "unable to find schedule file", etc. So can it be made an "incomplete"?
good point. I would be ok with anything in the range 10k-100k as a default :) |
Ok, I suppose that makes sense. And the answer to the question is of course yet. Then a simple "die" doesn't suffice anymore of course.
|
|
I've just changed the limit to 50000 which is hopefully high enough to still cover valid production use cases and low enough to now crash the browser for most users (likely depends on the available RAM). Since finding a sensible limit that fits all use cases is likely not possible it is good that this can still be changed via the job setting |
A worker change is required for passing the "incomplete" result to the web UI. Otherwise the job just ends up "failed" (still with the proper reason). Related ticket: https://progress.opensuse.org/issues/129068
|
I retriggered GHA CI jobs which passed now but we are missing codecov check results even though there is a coverage message comment. Trying rebasing. @Mergifyio rebase |
|
huh, seems like mergify was just slower than I expected :D All good |
| @@ -399,15 +412,21 @@ sub save_test_result ($self) { | |||
| return $result; | |||
| } | |||
|
|
|||
| sub _increment_test_count ($self, $max = $bmwqemu::vars{MAX_TEST_STEPS} // 50) { | |||
There was a problem hiding this comment.
I thought that should be 50000 and not 50? Or am I missing something?
There was a problem hiding this comment.
seems like a problem in rebasing happened here. I will create a fix
Related ticket: https://progress.opensuse.org/issues/129068