How to train with cityscapes dataset?

[Flawless1202]

I use the python scripts convert_cityscapes_to_coco.py and successfully convert the cityscapes dataset to coco. But when I modified the config file faster_rcnn_r50_fpn_1x.py and use the command python tools/train.py configs/faster_rcnn_r50_fpn_1x.py --gpus 2 --work_dir ./out --validate to train, I got the error:

`2018-10-16 11:05:01,156 - INFO - Distributed training: False
2018-10-16 11:05:01,600 - INFO - load model from: modelzoo://resnet50
2018-10-16 11:05:01,825 - WARNING - unexpected key in source state_dict: fc.weight, fc.bias

missing keys in source state_dict: layer3.4.bn1.num_batches_tracked, layer1.0.downsample.1.num_batches_tracked, layer3.5.bn2.num_batches_tracked, layer3.0.bn2.num_batches_tracked, layer4.1.bn3.num_batches_tracked, layer2.1.bn1.num_batches_tracked, layer3.4.bn3.num_batches_tracked, layer1.1.bn2.num_batches_tracked, layer3.3.bn3.num_batches_tracked, layer2.2.bn3.num_batches_tracked, layer3.1.bn1.num_batches_tracked, layer1.2.bn1.num_batches_tracked, layer4.0.bn1.num_batches_tracked, layer2.3.bn2.num_batches_tracked, layer3.1.bn3.num_batches_tracked, layer4.2.bn1.num_batches_tracked, layer3.4.bn2.num_batches_tracked, layer4.0.bn2.num_batches_tracked, layer4.2.bn3.num_batches_tracked, layer2.1.bn3.num_batches_tracked, layer2.0.bn3.num_batches_tracked, layer1.0.bn1.num_batches_tracked, layer1.0.bn3.num_batches_tracked, layer4.1.bn2.num_batches_tracked, layer3.0.bn1.num_batches_tracked, layer1.0.bn2.num_batches_tracked, layer3.2.bn1.num_batches_tracked, layer2.1.bn2.num_batches_tracked, layer4.1.bn1.num_batches_tracked, layer3.0.downsample.1.num_batches_tracked, layer1.2.bn3.num_batches_tracked, layer2.0.downsample.1.num_batches_tracked, layer2.3.bn3.num_batches_tracked, layer3.3.bn2.num_batches_tracked, layer1.1.bn1.num_batches_tracked, layer4.2.bn2.num_batches_tracked, layer3.3.bn1.num_batches_tracked, layer1.1.bn3.num_batches_tracked, layer2.0.bn1.num_batches_tracked, layer3.0.bn3.num_batches_tracked, layer2.3.bn1.num_batches_tracked, layer3.1.bn2.num_batches_tracked, layer4.0.downsample.1.num_batches_tracked, layer2.0.bn2.num_batches_tracked, layer2.2.bn2.num_batches_tracked, layer3.5.bn1.num_batches_tracked, layer3.2.bn3.num_batches_tracked, layer2.2.bn1.num_batches_tracked, layer3.2.bn2.num_batches_tracked, bn1.num_batches_tracked, layer3.5.bn3.num_batches_tracked, layer4.0.bn3.num_batches_tracked, layer1.2.bn2.num_batches_tracked

loading annotations into memory...
Done (t=4.59s)
creating index...
index created!
2018-10-16 11:05:09,360 - INFO - Start running, host: chenkai@Autodrive, work_dir: /home/chenkai/Documents/mmdetection/out
2018-10-16 11:05:09,360 - INFO - workflow: [('train', 1)], max: 12 epochs
Exception ignored in: <bound method _DataLoaderIter.del of <torch.utils.data.dataloader._DataLoaderIter object at 0x7f2cce44c1d0>>
Traceback (most recent call last):
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 399, in del
self._shutdown_workers()
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 378, in _shutdown_workers
self.worker_result_queue.get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 337, in get
return _ForkingPickler.loads(res)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 151, in rebuild_storage_fd
fd = df.detach()
File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
with _resource_sharer.get_connection(self._id) as conn:
File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
c = Client(address, authkey=process.current_process().authkey)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 493, in Client
answer_challenge(c, authkey)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 737, in answer_challenge
response = connection.recv_bytes(256) # reject large message
File "/usr/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "tools/train.py", line 82, in
main()
File "tools/train.py", line 78, in main
logger=logger)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmdet-0.5.0+f29f020-py3.6.egg/mmdet/apis/train.py", line 59, in train_detector
_non_dist_train(model, dataset, cfg, validate=validate)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmdet-0.5.0+f29f020-py3.6.egg/mmdet/apis/train.py", line 117, in _non_dist_train
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmcv-0.2.0-py3.6.egg/mmcv/runner/runner.py", line 349, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmcv-0.2.0-py3.6.egg/mmcv/runner/runner.py", line 255, in train
self.model, data_batch, train_mode=True, **kwargs)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmdet-0.5.0+f29f020-py3.6.egg/mmdet/apis/train.py", line 37, in batch_processor
losses = model(**data)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 123, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 133, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 77, in parallel_apply
raise output
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 53, in _worker
output = module(*input, **kwargs)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmdet-0.5.0+f29f020-py3.6.egg/mmdet/models/detectors/base.py", line 79, in forward
return self.forward_train(img, img_meta, **kwargs)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmdet-0.5.0+f29f020-py3.6.egg/mmdet/models/detectors/two_stage.py", line 111, in forward_train
self.train_cfg.rcnn)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmdet-0.5.0+f29f020-py3.6.egg/mmdet/models/bbox_heads/bbox_head.py", line 73, in get_bbox_target
target_stds=self.target_stds)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmdet-0.5.0+f29f020-py3.6.egg/mmdet/core/bbox/bbox_target.py", line 25, in bbox_target
target_stds=target_stds)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmdet-0.5.0+f29f020-py3.6.egg/mmdet/core/utils/misc.py", line 24, in multi_apply
return tuple(map(list, zip(*map_results)))
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmdet-0.5.0+f29f020-py3.6.egg/mmdet/core/bbox/bbox_target.py", line 62, in proposal_target_single
labels, reg_num_classes)
File "/home/chenkai/.virtualenvs/mmdetection_py36/lib/python3.6/site-packages/mmdet-0.5.0+f29f020-py3.6.egg/mmdet/core/bbox/bbox_target.py", line 75, in expand_target
bbox_targets_expand[i, start:end] = bbox_targets[i, :]
RuntimeError: The expanded size of the tensor (0) must match the existing size (4) at non-singleton dimension 0`

Could you help me to solve this problem? @hellock

[hellock]

We will try the cityscape dataset and update the information here.

[Flawless1202]

@hellock OK， thank you. And the two python scripts I use is here:
my_convert_cityscapes_to_coco.py

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals

import argparse
import h5py
import json
import os
import scipy.misc
import sys
import numpy as np

import instances2dict as cs
# import mmcv

# from matplotlib import pyplot as plt


def parse_args():
    parser = argparse.ArgumentParser(description='Convert dataset')
    parser.add_argument(
        '--dataset', help="cocostuff, cityscapes", default=None, type=str)
    parser.add_argument(
        '--outdir', help="output dir for json files", default=None, type=str)
    parser.add_argument(
        '--datadir', help="data dir for annotations to be converted",
        default=None, type=str)
    if len(sys.argv) == 1:
        parser.print_help()
        sys.exit(1)
    return parser.parse_args()


def polys_to_boxes(polys):
    """Convert a list of polygons into an array of tight bounding boxes."""
    boxes_from_polys = np.zeros((len(polys), 4), dtype=np.float32)
    for i in range(len(polys)):
        poly = polys[i]
        x0 = min(min(p[::2]) for p in poly)
        x1 = max(max(p[::2]) for p in poly)
        y0 = min(min(p[1::2]) for p in poly)
        y1 = max(max(p[1::2]) for p in poly)
        boxes_from_polys[i, :] = [x0, y0, x1, y1]

    return boxes_from_polys


def xyxy_to_xywh(xyxy):
    """Convert [x1 y1 x2 y2] box format to [x1 y1 w h] format."""
    if isinstance(xyxy, (list, tuple)):
        # Single box given as a list of coordinates
        assert len(xyxy) == 4
        x1, y1 = xyxy[0], xyxy[1]
        w = xyxy[2] - x1 + 1
        h = xyxy[3] - y1 + 1
        return (x1, y1, w, h)
    elif isinstance(xyxy, np.ndarray):
        # Multiple boxes given as a 2D ndarray
        return np.hstack((xyxy[:, 0:2], xyxy[:, 2:4] - xyxy[:, 0:2] + 1))
    else:
        raise TypeError('Argument xyxy must be a list, tuple, or numpy array.')


def convert_coco_stuff_mat(data_dir, out_dir):
    """Convert to png and save json with path. This currently only contains
    the segmentation labels for objects+stuff in cocostuff - if we need to
    combine with other labels from original COCO that will be a TODO."""
    sets = ['train', 'val']
    categories = []
    json_name = 'coco_stuff_%s.json'
    ann_dict = {}
    for data_set in sets:
        file_list = os.path.join(data_dir, '%s.txt')
        images = []
        with open(file_list % data_set) as f:
            for img_id, img_name in enumerate(f):
                img_name = img_name.replace('coco', 'COCO').strip('\n')
                image = {}
                mat_file = os.path.join(
                    data_dir, 'annotations/%s.mat' % img_name)
                data = h5py.File(mat_file, 'r')
                labelMap = data.get('S')
                if len(categories) == 0:
                    labelNames = data.get('names')
                    for idx, n in enumerate(labelNames):
                        categories.append(
                            {"id": idx, "name": ''.join(chr(i) for i in data[
                                n[0]])})
                    ann_dict['categories'] = categories
                scipy.misc.imsave(
                    os.path.join(data_dir, img_name + '.png'), labelMap)
                image['width'] = labelMap.shape[0]
                image['height'] = labelMap.shape[1]
                image['file_name'] = img_name
                image['seg_file_name'] = img_name
                image['id'] = img_id
                images.append(image)
        ann_dict['images'] = images
        print("Num images: %s" % len(images))
        with open(os.path.join(out_dir, json_name % data_set), 'wb') as outfile:
            outfile.write(json.dumps(ann_dict))


# for Cityscapes
def getLabelID(self, instID):
    if (instID < 1000):
        return instID
    else:
        return int(instID / 1000)


def convert_cityscapes_instance_only(
        data_dir, out_dir):
    """Convert from cityscapes format to COCO instance seg format - polygons"""
    sets = [
        'gtFine_val',
        # 'gtFine_train',
        # 'gtFine_test',

        # 'gtCoarse_train',
        # 'gtCoarse_val',
        # 'gtCoarse_train_extra'
    ]
    ann_dirs = [
        'gtFine_trainvaltest/gtFine/val',
        # 'gtFine_trainvaltest/gtFine/train',
        # 'gtFine_trainvaltest/gtFine/test',

        # 'gtCoarse/train',
        # 'gtCoarse/train_extra',
        # 'gtCoarse/val'
    ]
    img_dirs = [
        'cityscapes2coco/val',
        # 'cityscapes2coco/train'
    ]
    json_name = 'instancesonly_filtered_%s.json'
    ends_in = '%s_polygons.json'
    img_id = 0
    ann_id = 0
    cat_id = 1
    category_dict = {}

    category_instancesonly = [
        'person',
        'rider',
        'car',
        'truck',
        'bus',
        'train',
        'motorcycle',
        'bicycle',
    ]

    for data_set, ann_dir in zip(sets, ann_dirs):
        print('Starting %s' % data_set)
        ann_dict = {}
        images = []
        annotations = []
        ann_dir = os.path.join(data_dir, ann_dir)
        for root, _, files in os.walk(ann_dir):
            for filename in files:
                if filename.endswith(ends_in % data_set.split('_')[0]):
                    if len(images) % 50 == 0:
                        print("Processed %s images, %s annotations" % (
                            len(images), len(annotations)))
                    json_ann = json.load(open(os.path.join(root, filename)))
                    image = {}
                    image['id'] = img_id
                    img_id += 1

                    image['width'] = json_ann['imgWidth']
                    image['height'] = json_ann['imgHeight']
                    image['file_name'] = filename[:-len(
                        ends_in % data_set.split('_')[0])] + 'leftImg8bit.png'
                    image['seg_file_name'] = filename[:-len(
                        ends_in % data_set.split('_')[0])] + \
                                             '%s_instanceIds.png' % data_set.split('_')[0]
                    images.append(image)

                    fullname = os.path.abspath(os.path.join(root, image['seg_file_name']))
                    # print(fullname)
                    # print(image['file_name'], type(image['file_name']))
                    # img = mmcv.imread(os.path.join(img_dirs[0], image['file_name']))
                    # fig = plt.figure()
                    # ax = fig.add_subplot(111)
                    # plt.imshow(mmcv.bgr2rgb(img))

                    objects = cs.instances2dict_with_polygons([fullname], verbose=False)[fullname]

                    # count = 0
                    for object_cls in objects:

                        if object_cls not in category_instancesonly:
                            continue  # skip non-instance categories

                        for obj in objects[object_cls]:
                            if obj['contours'] == []:
                                print('Warning: empty contours.')
                                continue  # skip non-instance categories

                            len_p = [len(p) for p in obj['contours']]
                            if min(len_p) <= 4:
                                print('Warning: invalid contours.')
                                continue  # skip non-instance categories

                            ann = {}
                            ann['id'] = ann_id
                            ann_id += 1
                            ann['image_id'] = image['id']
                            ann['segmentation'] = obj['contours']

                            if object_cls not in category_dict:
                                category_dict[object_cls] = cat_id
                                cat_id += 1
                            ann['category_id'] = category_dict[object_cls]
                            ann['iscrowd'] = 0
                            ann['area'] = obj['pixelCount']
                            ann['bbox'] = xyxy_to_xywh(
                                polys_to_boxes(
                                    [ann['segmentation']])).tolist()[0]
                            # x, y, w, h = ann['bbox']
                            # rect = plt.Rectangle((x, y), w, h, linewidth=1,edgecolor='r',facecolor='none')
                            # ax.add_patch(rect)

                            annotations.append(ann)

                            # count+=1
                    # print("total:{} objects.".format(count))
                    # plt.show()

        ann_dict['images'] = images
        categories = [{"id": category_dict[name], "name": name} for name in
                      category_dict]
        ann_dict['categories'] = categories
        ann_dict['annotations'] = annotations
        print("Num categories: %s" % len(categories))
        print("Num images: %s" % len(images))
        print("Num annotations: %s" % len(annotations))
        with open(os.path.join(out_dir, json_name % data_set), 'w') as outfile:
            outfile.write(json.dumps(ann_dict))


if __name__ == '__main__':
    args = parse_args()
    if args.dataset == "cityscapes_instance_only":
        convert_cityscapes_instance_only(args.datadir, args.outdir)
    elif args.dataset == "cocostuff":
        convert_coco_stuff_mat(args.datadir, args.outdir)
    else:
        print("Dataset not supported: %s" % args.dataset)

instances2dict.py

#!/usr/bin/python
#
# Convert instances from png files to a dictionary
#

from __future__ import print_function, absolute_import, division
import os, sys

import numpy as np

# Cityscapes imports
from cityscapesscripts.evaluation.instance import *
from cityscapesscripts.helpers.csHelpers import *
import cv2


def instances2dict_with_polygons(imageFileList, verbose=False):
    # print(imageFileList)
    imgCount     = 0
    instanceDict = {}

    if not isinstance(imageFileList, list):
        imageFileList = [imageFileList]

    if verbose:
        print("Processing {} images...".format(len(imageFileList)))

    for imageFileName in imageFileList:
        # Load image
        img = Image.open(imageFileName)

        # Image as numpy array
        imgNp = np.array(img)

        # Initialize label categories
        instances = {}
        for label in labels:
            instances[label.name] = []

        # Loop through all instance ids in instance image
        for instanceId in np.unique(imgNp):
            if instanceId < 1000:
                continue

            instanceObj = Instance(imgNp, instanceId)
            instanceObj_dict = instanceObj.toDict()

            # instances[id2label[instanceObj.labelID].name].append(instanceObj.toDict())
            if id2label[instanceObj.labelID].hasInstances:
                mask = (imgNp == instanceId).astype(np.uint8)

                # print(cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE))

                _, contour, hier = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

                polygons = [c.reshape(-1).tolist() for c in contour]
                instanceObj_dict['contours'] = polygons

            instances[id2label[instanceObj.labelID].name].append(instanceObj_dict)

        imgKey = os.path.abspath(imageFileName)
        instanceDict[imgKey] = instances
        imgCount += 1

        if verbose:
            print("\rImages Processed: {}".format(imgCount), end=' ')
            sys.stdout.flush()

    if verbose:
        print("")

    # print(instanceDict)
    return instanceDict

def main(argv):
    fileList = []
    if (len(argv) > 2):
        for arg in argv:
            if ("png" in arg):
                fileList.append(arg)
    instances2dict(fileList, True)

if __name__ == "__main__":
    main(sys.argv[1:])

And the script instances2dict.py is modified on the basis of this.

[Flawless1202]

@hellock Oh I have already done with this two scripts but I could not get good result, maybe because I only use 2 1080ti to train. I am very looking forward to your result.

[OceanPang]

Hi @Flawless1202 , if you got a sub-optimal results and the annotations have been checked. Please refer to #25, 2 1080Ti may need lr = 0.005.

Experiments on cityscapes will be updated as soon as we are free, but maybe not in this month.

[Flawless1202]

@OceanPang Oh I forgot it, thanks a lot. I will try again.

[yinglang]

I

hi, Flawless1202, Have you train cityscapes successfully? do you mind to tell me the perfermance that coco style? very appeciate to you.

[yinglang]

@Flawless1202

[Flawless1202]

@yinglang hi, I have not trained the cityscapes but trained a cityscapes style dataset called Lost and Found. I think there is no problem to convert it to COCO style. And the performance is satisfy.

[yinglang]

ok, thanks @Flawless1202

[michaelisc]

Cityscapes was added in #1037. Follow the instructions in the README.

@hellock @OceanPang I guess this can be closed then.

[abhigoku10]

@michaelisc @hellock @Flawless1202 when i try to visualize cityscapes annotation i am getting below results . how to obtain the polygon area of the cityscapes dataset into json accurately

[xvjiarui]

Hi @abhigoku10
Do you refer to the json file like gtFine/**/*gtFine_polygons.json?