Add cityscapes dataset #1037

michaelisc · 2019-07-22T19:22:11Z

Dataset class and config files for cityscapes.

Annotation format:
The annotations have to be converted into the coco format following the example of maskrcnn-benchmark. The results are evaluated using the pycocotools.

Current status: working
The configs and dataset class are final and working. The results in config/cityscapes/README.md are reproducible and I am happy to provide the checkpoints.

Open points:

I still have to take the time to go over the annotation conversion code again and clean it up. If I remember correctly there were some bugs in the maskrcnn-benchmark version, that I wanted to fix.
The provided configs use multi-scale training to reproduce the results from the Mask R-CNN paper. This might however be suboptimal, as it differs from how the coco models are configured.
I have no way of hosting the checkpoints and my setup differs slightly from that of the modelzoo models.

hellock · 2019-07-23T12:44:32Z

Nice work! Thanks for providing the results on cityscapes. It is okay to adopt multi-scale training since it is almost a convention for this dataset. We can retrain the model with 8 GPUs and compare with your single-gpu results (@yhcao6 please help to verify that). If you figure out the bug in the conversion script, welcome to fix & provide a copy in tools/convert_datasets/.

BTW, there are some linting errors, could you fix them?

michaelisc · 2019-07-24T10:08:00Z

When retraining the model on 8 gpus you can use the learning rate from the config file, it is set for 8 gpus. I downscaled it by a factor of 8 (see the linear scaling rule pull request) for training on 1 gpu.

The linting errors should be fixed.

yhcao6 · 2019-07-24T14:06:34Z

When I run the code, I met some problems, please have a look to see if there are something wrong.

The first error show KeyError: 'None is not in the dataset registry'. So that I add @DATASETS.register_module before class CityscapesDataset(CocoDataset): in cityscapes.py , this resolve the problem.
When I setup the dataset as install.md and ran the program, it report the error
FileNotFoundError: img file does not exist: data/cityscapes/train/cologne_000102_000019_leftImg8bit.png
Then I found the actual image path is data/cityscapes/train/cologne/cologne_000102_000019_leftImg8bit.png. I wonder if I did something wrong.

michaelisc · 2019-07-24T17:45:35Z

@yhcao6 I think the first issue came up, because you changed the dataset handling in the meantime and I did not check the code again after merging. I added '@DATASETS.register_module' to the 'CityscapesDatasets' definition in the commit above.

I think the second error is exactly what I hit too. Either the additional folders have to be added to the filepaths in the annotation or the files have to be extracted from the folders. I decided to do the ladder using 'cp' and some '*'s. However I forgot the exact command and have to try it out again.

michaelisc · 2019-07-24T20:43:11Z

@yhcao6 the right command is: 'mv train/*/* train/'. I will try to look at the conversion scripts soon but am pretty busy the next two weeks. If it works with their scripts and the only thing you have to do is this file moving I can quickly write either a '.sh' function or add this into the conversion code.

yhcao6 · 2019-07-24T23:52:32Z

Report 8 gpu results here. I didn't change anything, but fix the dataset registry and move image.
Faster R50: 36.6
Mask R50: 38.3/33.1

hellock · 2019-07-25T03:08:50Z

Report 8 gpu results here. I didn't change anything, but fix the dataset registry and move image.
Faster R50: 36.6
Mask R50: 38.3/33.1

We can run it for 3 times and take the mid value. Then the model zoo can be updated.

yhcao6 · 2019-07-25T15:16:30Z

I ran the code three times, the fluctuation seems really large,
faster r50: 36.6, 35.8, 36.0
mask r50: 38.3/33.1, 37.2/32.0, 37.4/32.5
Is this normal?

michaelisc · 2019-07-25T18:25:16Z

I guess this is a result of the extremely small data set size (only 5000 images). Any suggestions how to handle this? Finetuning from a Coco checkpoint might reduce the fluctuations (and should result in better performance) but I did not have time yet to implement this.

yhcao6 · 2019-07-27T04:08:31Z

I add benchmark for these two models. Please have a look to see if there is any problem.

hellock · 2019-07-27T08:14:22Z

configs/cityscapes/README.md

+
+## Environment
+
+### Hardware


This may be removed since we tested different hardwares and the performances are comparable.

hellock · 2019-07-27T08:15:06Z

configs/cityscapes/README.md

+
+## Common settings
+
+- All baselines were trained using 1 GPU with a batch size of 2 (2 images per GPU) using the [linear scaling rule](https://arxiv.org/abs/1706.02677) to scale the learning rate. The learning rate in the configs is set for a batch size of 16 to match the default of the coco models.


8 GPUs now.

Can you just add in your setup descriptions?

hellock · 2019-07-27T08:16:09Z

configs/cityscapes/README.md

+
+|    Backbone     |  Style  | Lr schd | Scale    | Pretraining | Mem (GB) | Train time (s/iter) | Inf time (fps) | box AP | mask AP | Download |
+| :-------------: | :-----: | :-----: | :---:    | :---------: | :----:   | :----:              | :----:           | :----: | :-----: | :------: |
+|    R-50-FPN     | pytorch |   1x    | 800-1024 | Backbone    | 4.9      | 0.609               | 2.5            | 37.4  |  32.5   | [model](https://open-mmlab.s3.ap-northeast-2.amazonaws.com/mmdetection/models/cityscapes/mask_rcnn_r50_fpn_1x_city_20190727-9b3c56a5.pth) |


The inference time seems abnormal.

Just caused by the image resolution.

hellock · 2019-07-27T08:17:07Z

configs/cityscapes/README.md

+
+|    Backbone     |  Style  | Lr schd | Scale    | Pretraining | Mem (GB) | Train time (s/iter) | Inf time (fps) | box AP | Download |
+| :-------------: | :-----: | :-----: | :---:    | :---------: | :----:   | :----:              | :----:          | :----: | :------: |
+|    R-50-FPN     | pytorch |   1x    | 800-1024 | Backbone    | 4.9      | 0.345               | 8.8            | 36.0   | [model](https://open-mmlab.s3.ap-northeast-2.amazonaws.com/mmdetection/models/cityscapes/faster_rcnn_r50_fpn_1x_city_20190727-7b9c0534.pth) |


Better to use Y/N for pretraining.

I was trying to specify wether we used coco pretraining. Probably we should just remove this field until we have coco pretrained models and mention somewhere else, that we trained from scratch.

Yes we can just remove this column. In this repo, pretraining usually means using the pretrained model from ImageNet and scratch means no pretrained weights for backbones. If we further add models pretrained from COCO and finetuned on CityScapes, we need to clearly indicate that.

Yes. Let's think about that once we have the finetuned models.

hellock · 2019-07-27T09:42:19Z

@michaelisc Let me know if you have any other comments? This PR looks good to me.

michaelisc · 2019-07-27T10:12:12Z

@yhcao6 did you end up moving all images into one folder or change the annotations to go thorough the foilders? I think we should mention this processing step somewhere.

Apart from that it looks good to me!

michaelisc · 2019-07-27T10:13:30Z

INSTALL.md

 │   ├── VOCdevkit
 │   │   ├── VOC2007
 │   │   ├── VOC2012

 ```
+The cityscapes annotations have to be converted into the coco format using the [cityscapesScripts](https://github.com/mcordts/cityscapesScripts) toolbox.
+We plan to provide an easy to use conversion script. For the moment we recommend following the instructions provided in the 
+[maskrcnn-benchmark](https://github.com/facebookresearch/maskrcnn-benchmark/tree/master/maskrcnn_benchmark/data) toolbox.


@yhcao6 I think this is where the description about the file handling would have to go.

I just move all images into one folder. You can add it into readme.

michaelisc

Looks fine to me

Update information how to arrange cityscapes data.

michaelisc

Everything should be fine now.

michaelisc · 2019-07-27T12:05:55Z

Ah, I found something I missed. I don't think I have added cityscapes to mmde.core.get_classes. Shouldn't be a big deal though.

hellock · 2019-07-27T12:23:58Z

Ah, I found something I missed. I don't think I have added cityscapes to mmde.core.get_classes. Shouldn't be a big deal though.

It will not affect the training and testing, since the class labels are now obtained from the dataset itself. It may be useful for users who want to do something else though.

michaelisc · 2019-07-27T13:41:00Z

Ah, I found something I missed. I don't think I have added cityscapes to mmde.core.get_classes. Shouldn't be a big deal though.

It will not affect the training and testing, since the class labels are now obtained from the dataset itself. It may be useful for users who want to do something else though.

That's why I did not bump into this earlier. I only ran into it when trying to visualize the results of a different dataset.

hellock · 2019-07-27T13:44:34Z

Yep, so you can add it for potential usage, and I will merge this PR.

michaelisc · 2019-07-27T13:50:28Z

Added the class names. From my side this is ready to be merged.

* added cityscapes * updated configs * removed wip configs * Add initial dataset instructions * Add cityscapes readme * Add explanation for lr scaling * Ensure pep8 conformity * Add CityscapesDataset to the registry * add benchmark * rename config, modify README.md * fix typo * fix typo in config * modify INSTALL.md Update information how to arrange cityscapes data. * Add cityscapes class names

nemonameless · 2019-09-18T09:04:34Z

However, I think your mAP in coco protocol is not in conformity with the protocol of cityscapes evaluation https://github.com/mcordts/cityscapesScripts .Although your mAP is higher than the original MASK RCNN paper, it may be pseudomorph. I have trained cityscapes(only fine) using your code and even get 43.0 mask mAP in val-set in coco protocol, but it is only 30.6 when evaluate the val-set with cityscapesScripts,and only 27.3 on the leaderboard(test-set) of cityscapes.

And could you provide your instancesonly_filtered_gtFine_val.json and instancesonly_filtered_gtFine_train.json? Thanks.

I follow the maskrcnn-benchmark and get jsons, but when testing your trained model(https://open-mmlab.s3.ap-northeast-2.amazonaws.com/mmdetection/models/cityscapes/mask_rcnn_r50_fpn_1x_city_20190727-9b3c56a5.pth), I onlt get only 23.9 bbox mAP and 19.9 mask mAP in coco protocol.
Thanks. @michaelisc @yhcao6 @hellock

dhananjaisharma10 · 2019-11-10T02:35:06Z

When I run the code, I met some problems, please have a look to see if there are something wrong.

The first error show KeyError: 'None is not in the dataset registry'. So that I add @DATASETS.register_module before class CityscapesDataset(CocoDataset): in cityscapes.py , this resolve the problem.

When I setup the dataset as install.md and ran the program, it report the error
FileNotFoundError: img file does not exist: data/cityscapes/train/cologne_000102_000019_leftImg8bit.png
Then I found the actual image path is data/cityscapes/train/cologne/cologne_000102_000019_leftImg8bit.png. I wonder if I did something wrong.

Hi! I am trying to train on the Cityscapes dataset as well. I seem to be running into a similar error:

FileNotFoundError: img file does not exist: /home/Cityscapes/gtFine_trainvaltest/gtFine/train/bochum_000000_037223_leftImg8bit.png

When I checked I found that I did not have a single image with a suffix leftImg8bit.png. I am using gtFine_trainvaltest.zip (241MB) for training. Are you using the same dataset or something else?

The error seems to be originating because the convert_cityscapes_to_coco.py script from maskrcnn-benchmark does something like:

image['file_name'] = filename[:-len(
                        ends_in % data_set.split('_')[0])] + 'leftImg8bit.png'

but never writes an image under that name. Please let me know where I'm going wrong. Thanks!

dhananjaisharma10 · 2019-11-10T05:29:20Z

When I run the code, I met some problems, please have a look to see if there are something wrong.

The first error show KeyError: 'None is not in the dataset registry'. So that I add @DATASETS.register_module before class CityscapesDataset(CocoDataset): in cityscapes.py , this resolve the problem.

When I setup the dataset as install.md and ran the program, it report the error
FileNotFoundError: img file does not exist: data/cityscapes/train/cologne_000102_000019_leftImg8bit.png
Then I found the actual image path is data/cityscapes/train/cologne/cologne_000102_000019_leftImg8bit.png. I wonder if I did something wrong.

Hi! I am trying to train on the Cityscapes dataset as well. I seem to be running into a similar error:
FileNotFoundError: img file does not exist: /home/Cityscapes/gtFine_trainvaltest/gtFine/train/bochum_000000_037223_leftImg8bit.png
When I checked I found that I did not have a single image with a suffix leftImg8bit.png. I am using gtFine_trainvaltest.zip (241MB) for training. Are you using the same dataset or something else?

The error seems to be originating because the convert_cityscapes_to_coco.py script from maskrcnn-benchmark does something like:
image['file_name'] = filename[:-len(
                        ends_in % data_set.split('_')[0])] + 'leftImg8bit.png'
but never writes an image under that name. Please let me know where I'm going wrong. Thanks!

Never mind. I downloaded the leftImg8bit_trainvaltest.zip (11GB) dataset which has the original images. Thank you!

* added cityscapes * updated configs * removed wip configs * Add initial dataset instructions * Add cityscapes readme * Add explanation for lr scaling * Ensure pep8 conformity * Add CityscapesDataset to the registry * add benchmark * rename config, modify README.md * fix typo * fix typo in config * modify INSTALL.md Update information how to arrange cityscapes data. * Add cityscapes class names

…AT, XGB (open-mmlab#1037) * Tabular: Added support for specifying early stopping rounds in GBM, CAT, XGB * addressed comment

michaelisc added 7 commits June 18, 2019 15:55

added cityscapes

ea73b20

updated configs

b313f98

removed wip configs

a08eb25

Add initial dataset instructions

7ccb260

Merge branch 'master' into cityscapes

884805f

Add cityscapes readme

606e0df

Add explanation for lr scaling

deebb0a

Ensure pep8 conformity

e5f1d09

Add CityscapesDataset to the registry

fd814f2

add benchmark

3cabeff

hellock reviewed Jul 27, 2019

View reviewed changes

yhcao6 added 3 commits July 27, 2019 17:07

rename config, modify README.md

9495a37

fix typo

637f055

fix typo in config

cdab6dd

hellock approved these changes Jul 27, 2019

View reviewed changes

michaelisc commented Jul 27, 2019

View reviewed changes

modify INSTALL.md

3c01641

Update information how to arrange cityscapes data.

michaelisc commented Jul 27, 2019

View reviewed changes

Add cityscapes class names

76d4176

hellock merged commit 1c28e66 into open-mmlab:master Jul 27, 2019

michaelisc mentioned this pull request Sep 26, 2019

How to train with cityscapes dataset? #29

Closed


		## Common settings

		- All baselines were trained using 1 GPU with a batch size of 2 (2 images per GPU) using the [linear scaling rule](https://arxiv.org/abs/1706.02677) to scale the learning rate. The learning rate in the configs is set for a batch size of 16 to match the default of the coco models.

Add cityscapes dataset #1037

Add cityscapes dataset #1037

Conversation

michaelisc commented Jul 22, 2019

hellock commented Jul 23, 2019

michaelisc commented Jul 24, 2019

yhcao6 commented Jul 24, 2019

michaelisc commented Jul 24, 2019

michaelisc commented Jul 24, 2019 • edited

yhcao6 commented Jul 24, 2019

hellock commented Jul 25, 2019

yhcao6 commented Jul 25, 2019

michaelisc commented Jul 25, 2019 • edited

yhcao6 commented Jul 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hellock commented Jul 27, 2019

michaelisc commented Jul 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelisc left a comment

Choose a reason for hiding this comment

michaelisc left a comment

Choose a reason for hiding this comment

michaelisc commented Jul 27, 2019

hellock commented Jul 27, 2019

michaelisc commented Jul 27, 2019

hellock commented Jul 27, 2019

michaelisc commented Jul 27, 2019

nemonameless commented Sep 18, 2019 • edited

dhananjaisharma10 commented Nov 10, 2019 • edited

dhananjaisharma10 commented Nov 10, 2019

michaelisc commented Jul 24, 2019 •

edited

michaelisc commented Jul 25, 2019 •

edited

nemonameless commented Sep 18, 2019 •

edited

dhananjaisharma10 commented Nov 10, 2019 •

edited