mea metrics #166
mea metrics #166
Conversation
msokoloff1
commented
Jun 10, 2021
msokoloff1
commented
- Add code for datarow level iou scores
…into ms/mea-metrics
…into ms/mea-metrics
…belbox-python into ms/mea-metrics
…belbox-python into ms/mea-metrics
…belbox-python into ms/mea-metrics
labelbox/data/metrics/iou.py
Outdated
| ious = [iou for iou in ious if iou is not None] | ||
| if not ious: | ||
| return None | ||
| return 100 * np.mean(ious) |
There was a problem hiding this comment.
I wasn't sure where to put this * 100. I kind of feel like this should just be a float that is bound between 0 and 1. Any thoughts?
There was a problem hiding this comment.
fyi I reverted this back ( removed the 100 ) but I still want to discuss.
There was a problem hiding this comment.
I feel better for some reason with values like this as within [0, 1].
But whatever is typical in this domain would also be a good way to decide.
There was a problem hiding this comment.
I think 0 to 1 makes the most sense
| @retry.Retry(deadline=10.) | ||
| def url_to_numpy(mask_url: str) -> np.ndarray: | ||
| """ Downloads an image and converts to a numpy array """ | ||
| return np.array(Image.open(BytesIO(requests.get(mask_url).content))) |
There was a problem hiding this comment.
Are these resources closed automatically?
|
|
||
| def mask_iou(predictions: List[NDMask], labels: List[NDMask]) -> float: | ||
| """ | ||
| Creates prediction and label binary mask for all features with the same feature scheama id. |
There was a problem hiding this comment.
Does this method check for same feature schema id's, or is that something assumed from the input?
There was a problem hiding this comment.
It is not checked here. Technically it could accept any ids. because it is just calculating the iou between any set of masks.
labelbox/data/metrics/iou.py
Outdated
| label_content : one row from the bulk label export - `project.export_labels()` | ||
| ndjsons: Model predictions in the ndjson format specified here (https://docs.labelbox.com/data-model/en/index-en#annotations) | ||
| include_classifications: Whether or not to factor top level classifications into the iou score. | ||
| include_subclassifications: Whether or not to factor in subclassifications into the iou score |
There was a problem hiding this comment.
Are include_classifications, include_subclassifications independent, or should there be some validation on their relation?
There was a problem hiding this comment.
They are independent.
labelbox/data/metrics/iou.py
Outdated
| prediction, label = predictions[0], labels[0] | ||
|
|
||
| if type(prediction) != type(label): | ||
| raise Type |
There was a problem hiding this comment.
nope! that is bug haha
awu-labelbox
left a comment
awu-labelbox
left a comment
There was a problem hiding this comment.
Just one comment, but otherwise lgtm!
|
Please don't delete this branch! There is testing code floating around that relies on it. |
