DeepLab: Cityscapes splits refactoring

[ruslo]

Description

Refactoring, follow-up for #8866

Type of change

• Bug fix (non-breaking change which fixes an issue)

Tests

python deeplab/train.py \
    --logtostderr \
    --training_number_of_steps=100 \
    --train_split="train_fine" \
    --model_variant="xception_65" \
    --fine_tune_batch_norm=False \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --train_crop_size="769,769" \
    --train_batch_size=1 \
    --dataset="cityscapes" \
    --tf_initial_checkpoint=${PATH_TO_INITIAL_CHECKPOINT} \
    --train_logdir=${PATH_TO_TRAIN_DIR} \
    --dataset_dir=${PATH_TO_DATASET}

python deeplab/eval.py \
    --logtostderr \
    --eval_split="val_fine" \
    --model_variant="xception_65" \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --eval_crop_size="1025,2049" \
    --dataset="cityscapes" \
    --max_number_of_evaluations=1 \
    --checkpoint_dir=${PATH_TO_CHECKPOINT} \
    --eval_logdir=${PATH_TO_EVAL_DIR} \
    --dataset_dir=${PATH_TO_DATASET}

python deeplab/vis.py \
    --logtostderr \
    --vis_split="val_fine" \
    --model_variant="xception_65" \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --vis_crop_size="1025,2049" \
    --dataset="cityscapes" \
    --max_number_of_iterations=1 \
    --colormap_type="cityscapes" \
    --checkpoint_dir=${PATH_TO_CHECKPOINT} \
    --vis_logdir=${PATH_TO_VIS_DIR} \
    --dataset_dir=${PATH_TO_DATASET}

• Ubuntu 18.04.4
• CUDA 10.2
• Tensorflow 1.15.2

Checklist

• I have signed the Contributor License Agreement.
• I have read guidelines for pull request.
• My code follows the coding guidelines.
• I have performed a self code review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.

[aquariusjay]

Thanks, ruslo! It looks great! Much more readable than before.
Just a few suggestions. Please take a look. Thanks!

[aquariusjay]

We still need this, since test_fine does not have label.

[ruslo]

We still need this, since test_fine does not have label.

The files themselves are there, but they don't contain useful information:

> find gtFine/train/ -type f -name '*.json' | wc -l
2975

> find gtFine/test/ -type f -name '*.json' | wc -l
1525

Example of a list of labels from the train set:

> grep label gtFine/train/zurich/zurich_000121_000019_gtFine_polygons.json | sort -u
            "label": "bicycle", 
            "label": "building", 
            "label": "car", 
            "label": "ego vehicle", 
            "label": "license plate", 
            "label": "motorcycle", 
            "label": "out of roi", 
            "label": "person", 
            "label": "pole", 
            "label": "rider", 
            "label": "road", 
            "label": "sidewalk", 
            "label": "sky", 
            "label": "static", 
            "label": "terrain", 
            "label": "traffic light", 
            "label": "traffic sign", 
            "label": "vegetation", 
            "label": "wall", 

Example of a list of labels from the test set:

> grep label gtFine/test/munich/munich_000152_000019_gtFine_polygons.json | sort -u
            "label": "ego vehicle", 
            "label": "out of roi", 

Basically, they are all the same in the test set (as far as I understand):

> openssl sha1 gtFine/test/munich/munich_00000*_000019_gtFine_polygons.json
SHA1(gtFine/test/munich/munich_000000_000019_gtFine_polygons.json)= 349dd446df6a4066cd47bc898b5d72360dad4946
SHA1(gtFine/test/munich/munich_000001_000019_gtFine_polygons.json)= 349dd446df6a4066cd47bc898b5d72360dad4946
SHA1(gtFine/test/munich/munich_000002_000019_gtFine_polygons.json)= 349dd446df6a4066cd47bc898b5d72360dad4946
SHA1(gtFine/test/munich/munich_000003_000019_gtFine_polygons.json)= 349dd446df6a4066cd47bc898b5d72360dad4946
SHA1(gtFine/test/munich/munich_000004_000019_gtFine_polygons.json)= 349dd446df6a4066cd47bc898b5d72360dad4946
SHA1(gtFine/test/munich/munich_000005_000019_gtFine_polygons.json)= 349dd446df6a4066cd47bc898b5d72360dad4946
SHA1(gtFine/test/munich/munich_000006_000019_gtFine_polygons.json)= 349dd446df6a4066cd47bc898b5d72360dad4946
SHA1(gtFine/test/munich/munich_000007_000019_gtFine_polygons.json)= 349dd446df6a4066cd47bc898b5d72360dad4946
SHA1(gtFine/test/munich/munich_000008_000019_gtFine_polygons.json)= 349dd446df6a4066cd47bc898b5d72360dad4946
SHA1(gtFine/test/munich/munich_000009_000019_gtFine_polygons.json)= 349dd446df6a4066cd47bc898b5d72360dad4946

If you add this line to build_cityscapes_data.py:

print('split: {}, images: {}, labels: {}'.format(dataset_split, num_images, num_labels))

Output will be:

split: train_fine, images: 2975, labels: 2975
split: val_fine, images: 500, labels: 500
split: test_fine, images: 1525, labels: 1525

[aquariusjay]

Thanks for double-checking it, ruslo!

[aquariusjay]

Let's change it back to

for dataset_split in ['train_fine', 'val_fine']:

It seems that we need to do more changes in order to support test_fine (e.g., skip reading groundtruth).
Let's change it back, since this should be orthogonal to your this pull request.
Sorry about going back and forth.

[ruslo]

Let's change it back

I can change it back, but as I said, the vis.py is working fine. That's the only place where I can see the test_fine can be used.

[aquariusjay]

Oh, yes, you are right. I totally forgot that Cityscapes has default groundtruth (all ignore labels) for test set.
Thanks for the clarification, ruslo!

[aquariusjay]

You are right. Thanks for double-checking it, ruslo!

[aquariusjay]

Looks great! Thanks for double checking it, ruslo!

[aquariusjay]

You are right. Thanks for double-checking it, ruslo!

[aquariusjay]

Thanks for double-checking it, ruslo!