How to feed colored input maps?

[code-de]

Hey guys, what an amazing work!

That's probably a dumb question, but how would one use images like these as inputs to the model?

Is there a pre-processing step that we need to take? I can only see that the model uses greyscale val_inst and val_label pictures, and I don't really get how they're obtained.

Many thanks in advance!

[warnotte]

I've the same question here :)

[taesungp]

The input label, when loaded should contain integer values like 0, 1, 2, ..., 182. Each number means the label of that pixel.

There are two ways of visualizing this. One way is treating the numbers as brightness of greyscale image. This visualization will lead to the bottom images you uploaded. But this visualization is hard to recognize because the difference among different labels is so subtle. On the other hand, you can entirely change the color of each label. For example, 0 can be painted red, 1 can be painted green, and so on. The top images you uploaded correspond this. It's customary that we use this kind of coloring of labels in papers.

We have code that can "colorize" greyscale images into colorful ones. Please use Colorize in this code.

It's harder to convert colored label images to greyscale images, because you would have to figure out what class number each color corresponds to. Usually if you got this image from a dataset, the dataset will come with greyscale version of the label map. Is this coco-stuff?

[warnotte]

Ok I see. Thank you very much Taesungp that a good starting solution.

[code-de]

Thanks so much for your reply, @taesungp! I now understand how these grey-scale label maps work (and yep, this is coco-stuff). But here come a few questions:

1. So we can encode labels in the pixel brightness - that would be our label map (right?). How is instance map obtained then? Do we need both to generate an image (and we don't need the ones in val_img)?

2. How can I convert colored label images to greyscale images, if I know to which object each color corresponds (like, I know that green color on my colored image corresponds to "124" greyscale brightness, which is grass in coco-stuff dataset)? Is there any Python implementations?

[ak9250]

I made a google colab notebook that works with testing and training coco-stuff, will add other datasets and how to train a custom dataset soon https://github.com/ak9250/SPADE-colab

[taesungp]

@code-de

1. instance map is made similar to label map. In instance map, pixel value of 0 will mean the first object, 1 the second object, and so on. Therefore, it's possible to visualize instance map just like label map, in either greyscale or colorized version. Of course, the actual values of numbers don't really mean anything. We can swap the order of objects. In fact, using get_edges of models/pix2pix_model.py, we compute the edge from the instance map and just use that instead of the pixel values.

The pretrained models that are trained using instance map will require instance map to generate outputs.

Lastly, in case of COCO-stuff, we created an instance map that combines the boundary of label map and original instance map. COCO dataset comes with instance map. However, the instance map does not contain boundary between different labels. For example, imagine an image of two people holding hands together on grass field under clear sky. The instance map will show the boundary between the two people, but it wouldn't show the boundary between grass field and sky. We augmented the original instance map by combining the boundaries of the instance map and label map. We plan to release this script soon.

2. We do not have a python implementation to do that, because we never needed one. We were always able to start with the greyscale version. But it shouldn't be so hard to do the conversion as long as you know which color corresponds to which label number.

[taesungp]

@ak9250 thank you!

[gabemagee]

Hey all, great work. I'm a little confused about the difference between an instance map and a label map. They both seem to be labeling pixels of an image. Is there a difference in purpose?

[gabemagee]

Oh wait, in an instance map - each object gets it's own pixel value, in a label map, each TYPE of object gets its won pixel value. Is this correct?

[taesungp]

@Maggab1031 yes it is correct. For example, label map alone cannot distinguish between two overlapping objects and one object.

In this image, there are many people, but without instance map it will just be one big blob of just one label.