[feat] Evaluate and log all performance values before asserting them #334
Conversation
One can create multiple Objectives for a singe test case (see example below). This commit improves the logging of the values. It add the tag to the log and it logs all values. (before it was all up to reference is not reached). Thus we can keep better track of all the values observed during the tests.
Reference example:
p_names = {'creation', 'stat', 'removal'}
for p_name in p_names:
self.perf_patterns[p_name] = sn.extractsingle(r'^\s+File '+p_name+
'\s+:\s+(?P<'+p_name+'>\S+) ', self.stdout, p_name, float)
self.reference = {
'kupe:compute' : {
'creation' : (7747, -(2*223.1)/7747, None),
'stat' : (16527, -(2*558.2)/16527,None),
'removal' : (7355, -(2*173.1)/7355, None)
},
}
|
Can I test this patch? |
Codecov Report
@@ Coverage Diff @@
## master #334 +/- ##
==========================================
+ Coverage 91.28% 91.31% +0.02%
==========================================
Files 67 67
Lines 8018 8022 +4
==========================================
+ Hits 7319 7325 +6
+ Misses 699 697 -2
Continue to review full report at Codecov.
|
| key = '%s:%s' % (self._current_partition.fullname, tag) | ||
| ref, low_thres, high_thres = self.reference[key] | ||
| evaluate(assert_reference(value[tag], ref, low_thres, | ||
| high_thres)) |
There was a problem hiding this comment.
I can't understand very well the purpose of this nested loop here.
There was a problem hiding this comment.
It is not a nested loop. I needed to move the evaluation to an additional loop. The reason was that if one of the early reference comparison throw the assert the other values were not written to the log file. In this way first all values are written and then the comparison is performed.
There was a problem hiding this comment.
You are right, for some reason I overlooked the alignment! I see what you are trying to do.
|
@jenkins-cscs retry all |
|
|
||
| with os_ext.change_dir(self._stagedir): | ||
| # first check and print all values with references | ||
| value = {} |
There was a problem hiding this comment.
Better call this perf_values. I'd also change a bit the comment to "Evaluate all performance variables first, so as to log them, then assert them".
| (tag, self._current_partition.fullname) | ||
| ) | ||
| evaluate(assert_reference(value, ref, low_thres, high_thres)) | ||
| for tag, expr in self.perf_patterns.items(): |
There was a problem hiding this comment.
There is no need to go over the perf_patterns again here. Just over the evaluated perf_values.
Also, leave a blank line after the previous for loop.
| (value, self.reference[key]) | ||
| '%s, value: %s, reference: %s' % | ||
| (tag, value[tag], self.reference[key]) | ||
| ) |
There was a problem hiding this comment.
For the logging, I suggest waiting for #213 to be merged (perhaps today).
vkarak
changed the title
vkarak
changed the title
|
Closing this. Replaced by #359. |
4 participants
One can create multiple Objectives for a singe test case (see example below). This commit improves the logging of the values. It add the tag to the log and it logs all values. (before it was all up to reference is not reached). Thus we can keep better track of all the values observed during the tests.
Reference example:
p_names = {'creation', 'stat', 'removal'}
for p_name in p_names:
self.perf_patterns[p_name] = sn.extractsingle(r'^\s+File '+p_name+
'\s+:\s+(?P<'+p_name+'>\S+) ', self.stdout, p_name, float)