Add chainermn support to apply_to_iterator

[Hakuyume]

import time

from chainer.iterators import SerialIterator
import chainermn
from chainercv.utils import apply_to_iterator

comm = chainermn.create_communicator('naive')


def func(xs):
    time.sleep(1)
    ys = [x * x for x in xs]
    print('{}: {} -> {}'.format(comm.rank, xs, ys))
    return ys


if comm.rank == 0:
    dataset = [
        (3, 'one'), (4, 'one'), (5, 'nine'), (2, 'six'),
        (5, 'three'), (5, 'eight'), (9, 'seven'), (9, 'zero')]
    iter_ = SerialIterator(dataset, batch_size=5, repeat=False, shuffle=False)
    in_values, out_values, rest_values = apply_to_iterator(
        func, iter_, comm=comm)
    for in_, out, rest in zip(in_values[0], out_values[0], rest_values[0]):
        print(in_, out, rest)
else:
    apply_to_iterator(func, None, comm=comm)

with 3 workers

2: [5] -> [25]
1: [5, 2] -> [25, 4]
0: [3, 4] -> [9, 16]
3 9 one
4 16 one
5 25 nine
2 4 six
5 25 three
2: [9] -> [81]
1: [9] -> [81]
0: [5] -> [25]
5 25 eight
9 81 seven
9 81 zero

[Hakuyume]

COCO detection example (note that I reduced the size of dataset)

w/o comm

100 of 100 samples, 4.87 samples/sec
mmAP (all): 0.39965793
mmAP (large): 0.5002278
mmAP (medium): 0.45323664
mmAP (small): 0.2703926

w/ comm (4 workers)

100 of 100 samples, 14.48 samples/sec
mmAP (all): 0.39965793
mmAP (large): 0.5002278
mmAP (medium): 0.45323664
mmAP (small): 0.2703926

[Hakuyume]

CamVid example

w/o comm

233 of 233 samples, 16.07 samples/sec
                    Sky : 0.8801
               Building : 0.6662
                   Pole : 0.1924
                   Road : 0.8720
               Pavement : 0.6394
                   Tree : 0.6242
             SignSymbol : 0.1888
                  Fence : 0.2119
                    Car : 0.6421
             Pedestrian : 0.2704
              Bicyclist : 0.2389
==================================
               mean IoU : 0.4933
 Class average accuracy : 0.6730
Global average accuracy : 0.8255

w/ comm (4 workers, note that 233 % 4 is not 0)

233 of 233 samples, 34.42 samples/sec
                    Sky : 0.8801
               Building : 0.6662
                   Pole : 0.1924
                   Road : 0.8720
               Pavement : 0.6394
                   Tree : 0.6242
             SignSymbol : 0.1888
                  Fence : 0.2119
                    Car : 0.6421
             Pedestrian : 0.2704
              Bicyclist : 0.2389
==================================
               mean IoU : 0.4933
 Class average accuracy : 0.6730
Global average accuracy : 0.8255

[yuyu2172]

I prefer reusing part of eval_coco.py.

[Hakuyume]

I'll do this in another PR. It will also merge eval_voc07and eval_coco like eval_semantic_segmentation.

[yuyu2172]

How about adding assertion here?