Found 2 (potential) bugs when training on semantic-only datasets.

[sebastian-woessner]

Thank you for making this amazing work publicly available.

I am using deeplab2 with a semantic-only dataset, i.e. the red channel of the image contains the class label and the other channels are 0.
I followed the instructions, but found 2 issues that appear when using semantic-only datasets:

Issue 1
original:

label = tf.io.decode_image(parsed_tensors[label_key], channels=1)

proposed change:

label = tf.io.decode_image(parsed_tensors[label_key], channels=3)
# Select red channel
label = label[:, :, 0:1]

The original code converts the image to grayscale instead of specifically selecting the red channel, which remaps the class labels.

Issue 2
original:

semantic_label = label // panoptic_label_divisor

proposed change:

semantic_label = label
if dataset_info['is_panoptic_dataset']:
  semantic_label = semantic_label // panoptic_label_divisor

Only divide by the panoptic_label_divisor if the dataset is a panoptic dataset, otherwise the labels of a semantic-only dataset are overwritten.
Do you agree with me regarding these two issues?

[markweberdev]

Thanks Sebastian for your comment! We do not regularly use semantic segmentation, so we're always looking for feedback!

Regarding your 2nd issue, I'd recommend to use a panoptic_label_divisor of 1, if you don't want to rescale your labels. This way, we make sure that there are no other side-effects in the code.

Regarding your 1st issue: Could you provide some more information on how you encoded the labels into tfrecords? If my understanding is correct, the issue is that you encode the image as a 3 channel image, but it's read out as 1 channel which leads to issues?

Best,
Mark

[sebastian-woessner]

Regarding the 1st issue:
I encoded the images as 3 channel images. The problem is that setting channels=1 as arg for tf.io.decode_image() controls the number of channels after decoding rather than the expected number of channels of the input. Thus, a 3 channel image is converted to a 1 channel grayscale image. This leads to an error if users follow the standard practice of encoding images as 3 channel images as in data/build_cityscapes_data.py.
Regarding the 2nd issue:
I agree that setting panoptic_label_divisor to 1 is the best solution. However, the code block before requires panoptic_label_divisor to be greater than 1 if ignore_label is greater than 1.

[aquariusjay]

Hi @sebastian-woessner,

Thank you for reporting the issue.
Most of our codes assume panoptic label format (panoptic = semantic * label_divisor + instance).
As a quick workaround for training pure semantic segmentation datasets, you could (1) set panptic_label_divisor to be a large value (say 1000), which should be compatible with ignore_label 255 (assume you are using PASCAL or Cityscapes), and (2) encode instance identity 0 when converting the dataset.

Cheers,

[sebastian-woessner]

Thank you for your response,

To be clear, the issue that I originally posted is currently not hindering me since I applied the fixes that I proposed above. I opened this issue to highlight 2 problems/bugs which might come up when a semantic-only dataset is used.
The code requires the semantic label to be encoded as a one-channel image and that panoptic_label_divisor is set to 1, unless the label is multiplied by panoptic_label_divisor during encoding. Users who encode the labels as RGB images will run into a “silent” bug.
I am closing the issue since this is not a bug in the code but rather a matter of usage.