Add segment performance check #935
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
|
||
| def _create_bins_and_metrics(self, batch_data: t.List[t.Tuple], dataset): | ||
| """Return dict of bins for each property in format | ||
| {property_name: [{start: val, stop: val, count: x, metrics: {name: metric...}}, ...], ...}""" |
There was a problem hiding this comment.
| {property_name: [{start: val, stop: val, count: x, metrics: {name: metric...}}, ...], ...}""" | |
| {property_name: [{start: val, stop: val, count: x, metrics: {name: metric...}}], ...}""" |
There was a problem hiding this comment.
this is not true, since this is a list with multiple items
There was a problem hiding this comment.
what other item is there after the metric?
|
View / edit / reply to this conversation on ReviewNB noamzbr commented on 2022-02-28T14:27:43Z Line #5. coco_data = coco.load_dataset(train=False, object_type='VisionData') We should link to an explanation of what is the meaning of each property |
|
View / edit / reply to this conversation on ReviewNB noamzbr commented on 2022-03-01T12:10:21Z * the image segment performance |
|
View / edit / reply to this conversation on ReviewNB noamzbr commented on 2022-03-01T12:10:23Z The check helps to detect segments of your data that are under-performing based on the basic properties of the image. For example, by default the check would show how the performance depends on brightness, area and other such properties. Identifying you models' weak segments might help to address specific issues and improve the overall performance of the model.
|
|
View / edit / reply to this conversation on ReviewNB noamzbr commented on 2022-03-01T12:10:24Z Line #8. prediction_formatter = DetectionPredictionFormatter(coco.yolo_prediction_formatter) Perhaps let's start now with improving our notebooks by writing the explicit formatter here? |
|
View / edit / reply to this conversation on ReviewNB noamzbr commented on 2022-03-01T12:10:25Z The check has a default condition which can be defined. The condition calculates for each property & metric the mean score and then looks at the ratio between the lowest segment score and the mean score. If this ratio is less than defined threshold, the condition fails.\ The purpose of the condition is to catch properties segments that are significantly worse than the mean - which might indicate a problem.
Also, add something specific to the result we're seeing here - "in this case, the condition has identified..." |
|
View / edit / reply to this conversation on ReviewNB ItayGabbay commented on 2022-03-01T14:49:27Z which AP is this? maybe call it AveragePrecision@0.5..0.95, and AverageRecall@100? |
resolve #917
Open problems: