Change eval_semantic_segmentation #217

yuyu2172 · 2017-05-30T03:53:18Z

No description provided.

yuyu2172 · 2017-05-30T14:19:53Z

I think the return values of eval_* should be reconsidered.

There are three types of return values

mean of Metric (e.g. mIoU mAP)
Metric (e.g. IoU, AP for each class)
intermediate values (e.g. confusion matrix, prec, rec)

In my opinion, returning a tuple of (metric, intermediate1, ..., intermedite n) is good.
So in that case, eval_semantic_segmentation returns iou, confusion and
eval_detection_voc returns ap, prec, rec.

yuyu2172 · 2017-05-30T14:20:45Z

I will fix the interface of this function after its design is agreed upon.

Hakuyume · 2017-06-01T01:50:28Z

I think there are two groups of metrics/intermediate values.
First group is values that computed for each class. For example, AP, prec and rec (detection) and IoU ( (semantic segmentation).
Second group is values that computed for the entire classes. For example, confusion of semantic segmentation. This is not an mean value of any class-wise values. I guess your proposal can not treat this group. How do you think of it?

yuyu2172 · 2017-06-01T11:01:21Z

Updated interface.

Hakuyume · 2017-06-01T23:57:15Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+    gt_labels = iter(gt_labels)
+
+    confusion = np.zeros((n_class, n_class), dtype=np.int64)
+    while True:


You can write this loop as follows.

pred_labels = iter(pred_labels) gt_labels = iter(gt_labels) for pred_label, gt_label in six.moves.zip(pred_labels, gt_labels): ... # Check if all iterators are same length. for iter_ in (pred_labels, gt_labels): # If one of them has more items, we can pop more item from it. # This code assumes any iterator does not contain None as its item. if next(iter_, None) is not None: raise ValueError('Length of input iterables need to be same.')

Hakuyume · 2017-06-02T00:05:38Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+    """Collect confusion matrix.
+
+    Args:
+        pred_label (iterable of numpy.ndarray): A collection of predicted


pred_label -> pred_labels

Hakuyume · 2017-06-02T00:05:52Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+            is :math:`(H, W)`. :math:`H` and :math:`W`
+            are height and width of the label.
+            For example, This is an array of shape :math:`(N, H, W)`.
+        gt_label (iterable of numpy.ndarray): A collection of ground


gt_label -> gt_labels

Hakuyume · 2017-06-02T00:14:16Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+            labels. The shape of a label array
+            is :math:`(H, W)`. :math:`H` and :math:`W`
+            are height and width of the label.
+            For example, This is an array of shape :math:`(N, H, W)`.


This example is not good because this function can take images whose shapes are different each other. (N, H, W) sounds that all images should have the same shape. How about [label0, label1, ...], where label0.shape = (H0, W0), label1.shape = (H1, W1) .. ?

Hakuyume · 2017-06-02T00:15:24Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+            truth label. The shape of a ground truth label array is
+            :math:`(H, W)`. The corresponding prediction label should
+            have the same shape.
+            A pixel with value "-1" will be ignored during evaluation.


"-1" -> :obj:-1

Hakuyume · 2017-06-02T00:20:49Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+        A confusion matrix. Its shape is :math:`(n\_class, n\_class)`.
+        The :math:`(i, j)` th element corresponds to the number of times
+        a pixel with ground truth class :math:`i` is predicted
+        to be class :math:`j`.


How about the number of pixels that is labelled as class i in ground truth and predicted as class j?

Hakuyume · 2017-06-02T00:23:48Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+    The definition of Intersection over Union (IoU) is as follows.
+
+    * :math:`\\text{IoU of the i-th class} =  \
+        \\frac{N_{ii}}{\\sum_{j=1}^k N_{ij} + \\sum_{j=1}^k N_{ji} - N_{ii}}`


Perhaps, we need denote the definition of N_{ij}.

Hakuyume · 2017-06-02T00:27:53Z

examples/segnet/train.py

-         'main/loss', 'main/mean_iou', 'main/mean_accuracy',
-         'validation/main/loss',
-         'validation/main/mean_iou', 'validation/main/mean_accuracy']),
+         'main/loss']),


Why did you remove validation/main/loss?

Hakuyume · 2017-06-02T00:29:44Z

tests/evaluations_tests/test_eval_semantic_segmentation_iou.py

+def _pred_iterator():
+    pred_labels = np.repeat([[[1, 1, 0], [0, 0, 1]]], 2, axis=0)
+    for pred_label in pred_labels:
+        yield pred_label


You can use iter() to make an iterator from an iterable.

yuyu2172 · 2017-06-02T02:38:12Z

Thanks. I fixed the code.

Hakuyume · 2017-06-02T04:46:03Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+            are height and width of the label.
+            For example, this is a list of labels
+            :obj:`[label_0, label_1, ...]`, where
+            :obj:`label_i.shape= (H_i, W_i)`.


:obj:label_i.shape= (H_i, W_i) -> :obj:label_i.shape = (H_i, W_i) (space)

Hakuyume · 2017-06-02T04:56:57Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+        mask = (gt_label >= 0) & (gt_label < n_class)
+        confusion += np.bincount(
+            n_class * gt_label[mask].astype(int) +
+            pred_label[mask], minlength=n_class**2).reshape(n_class, n_class)


I think we can expand the confusion matrix dynamically. n_class can be removed.

n_class = 0 ... lb_max = gt_label.max() if lb_max >= n_class: expanded_confusion = np.zeros((lb_max + 1, lb_max + 1)) expanded_confusion[0:n_class, 0:n_class] = confusion n_class = lb_max + 1 confusion = expanded_confusion

This expansion may affect performance. But I think it will be ignorable because neural networks are much slower.
How do you think of it?

That code seems simple enough.
I left it as is because I thought the code gets too complex.

I will try it.

Hakuyume · 2017-06-02T05:40:23Z

docs/source/reference/evaluations.rst

+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autofunction:: eval_semantic_segmentation_iou
+
+calc_semantic_segmentation_confusion


This is not an alphabetical order. I guess you put these functions here because they are helper functions of eval_semantic_segmentation_iou. How about nesting these items to show the hierarchy more explicitly?

Hakuyume · 2017-06-02T07:10:15Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+    """Collect a confusion matrix.
+
+    The number of classes :math:`n\_class` is computed as the maximum
+    class id among :obj:`pred_labels` and :obj:`gt_labels`.


It sounds n_class = max(pred_labels, gt_labels). as -> from? Perhaps, an equation like n_class = max(pred_labels, gt_labels) + 1 will be helpful.

Hakuyume · 2017-06-02T07:10:38Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+            is :math:`(H, W)`. :math:`H` and :math:`W`
+            are height and width of the label.
+        gt_labels (iterable of numpy.ndarray): A collection of ground
+            truth label. The shape of a ground truth label array is


label -> labels

Hakuyume · 2017-06-02T07:11:29Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+        # Dynamically expand the confusion matrix if necessary.
+        lb_max = np.max((pred_label, gt_label))
+        if lb_max >= n_class:
+            expanded_confusion = np.zeros


What is this line?

Hakuyume · 2017-06-02T07:12:45Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+        lb_max = np.max((pred_label, gt_label))
+        if lb_max >= n_class:
+            expanded_confusion = np.zeros
+            expanded_confusion = np.zeros((lb_max + 1, lb_max + 1))


Perhaps it is better to use dtype=np.int64 option. You are using this option at L.36 but it will be overwritten by expansion.

Hakuyume · 2017-06-02T07:14:25Z

chainercv/evaluations/eval_semantic_segmentation_iou.py

+            confusion = expanded_confusion
+
+        # Count statistics from valid pixels.
+        mask = (gt_label >= 0) & (gt_label < n_class)


(gt_label < n_class) is always true because n_class > lb_max = np.max((pred_label, gt_label)).

Hakuyume

LGTM

yuyu2172 added 3 commits May 30, 2017 12:52

change eval_semantic_segmentation

3474058

change interface to accept iterator

8b92dc8

add iterator to test case

7430dab

yuyu2172 assigned Hakuyume May 30, 2017

Hakuyume mentioned this pull request Jun 1, 2017

Improve eval_detection_voc #232

Merged

change interface

ec2ac61

yuyu2172 force-pushed the eval-semantic-segmentation-change branch from d07751c to ec2ac61 Compare June 1, 2017 10:58

fix doc

5f7a47f

yuyu2172 added 2 commits June 1, 2017 20:03

add ignore labels to the test

9ce7523

fix pixelwise_softmax_classifier

e45a60b

yuyu2172 mentioned this pull request Jun 1, 2017

Add SemanticSegmentationEvaluator #238

Merged

Hakuyume reviewed Jun 1, 2017

View reviewed changes

Hakuyume reviewed Jun 2, 2017

View reviewed changes

yuyu2172 force-pushed the eval-semantic-segmentation-change branch from 5cf0add to a390dfc Compare June 2, 2017 02:33

reflect comment on eval_semantic_segmentation_iou

babf7c9

yuyu2172 force-pushed the eval-semantic-segmentation-change branch from a390dfc to babf7c9 Compare June 2, 2017 02:36

yuyu2172 added 2 commits June 2, 2017 11:36

fix train.py

ee4bd80

fix test

569e66f

Hakuyume reviewed Jun 2, 2017

View reviewed changes

yuyu2172 added 3 commits June 2, 2017 14:31

dynamically calcualte n_class

f6c9288

fix doc

35d6efb

fix doc

41198b3

yuyu2172 force-pushed the eval-semantic-segmentation-change branch from 9f45e1c to 41198b3 Compare June 2, 2017 05:37

Hakuyume reviewed Jun 2, 2017

View reviewed changes

make hierarchy inside evaluations

5db4c7e

Hakuyume reviewed Jun 2, 2017

View reviewed changes

fix code

916d298

yuyu2172 force-pushed the eval-semantic-segmentation-change branch from 1be7061 to 4bfbcdd Compare June 2, 2017 07:31

fix doc

c4ec9ec

yuyu2172 force-pushed the eval-semantic-segmentation-change branch from 4bfbcdd to c4ec9ec Compare June 2, 2017 07:33

fix valuerror

e82eede

Hakuyume approved these changes Jun 2, 2017

View reviewed changes

Hakuyume merged commit 7990fc2 into chainer:master Jun 2, 2017

yuyu2172 mentioned this pull request Jun 7, 2017

eval_semantic_segmentation is wrong #215

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change eval_semantic_segmentation #217

Change eval_semantic_segmentation #217

yuyu2172 commented May 30, 2017

yuyu2172 commented May 30, 2017

yuyu2172 commented May 30, 2017

Hakuyume commented Jun 1, 2017

yuyu2172 commented Jun 1, 2017

Hakuyume Jun 1, 2017 •

edited

Loading

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

yuyu2172 commented Jun 2, 2017

Hakuyume Jun 2, 2017

yuyu2172 Jun 2, 2017

Hakuyume Jun 2, 2017 •

edited

Loading

yuyu2172 Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume Jun 2, 2017

Hakuyume left a comment

Change eval_semantic_segmentation #217

Change eval_semantic_segmentation #217

Conversation

yuyu2172 commented May 30, 2017

yuyu2172 commented May 30, 2017

yuyu2172 commented May 30, 2017

Hakuyume commented Jun 1, 2017

yuyu2172 commented Jun 1, 2017

Hakuyume Jun 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuyu2172 commented Jun 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hakuyume Jun 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hakuyume left a comment

Choose a reason for hiding this comment

Hakuyume Jun 1, 2017 •

edited

Loading

Hakuyume Jun 2, 2017 •

edited

Loading