semantic segmentation evaluation code

[eyaler]

where can i find the code that converts predicted colors to discrete labels for evaluating semantic segmentation (photo->label)?

[junyanz]

This discussion might help.

[eyaler]

i read it, but i could not find (or did not understand) how for a given output y=(r,g,b) find the nearest t=(R,G,B) which belongs to some label. i guess you would need to do something like argmin L1(y,t)

[junyanz]

Exactly, we did a nearest neighbor search in color space.

[qianshangbushang]

and could you offer the code for how to do a nearest neighbor search? i am using gan doing semantic segmentation and the generate images color can not exactly match the raw color , i cant map the result to label. thanks

[WorkingCC]

@junyanz Hi, junyanz.
I have read the papers about pix-pixGAN and CycleGAN. I'm intersted in GAN, and now I am evaluating the classification performmance of label(photo->label) .But I don't kow how to convert color map to label map. Could you offer the code for converting color map to label map? We will cite your research,thanke you ~.

[sagiebenaim]

Would appreciate if you could release the code for converting the color map to a label map. Thanks!

[junyanz]

@phillipi

[ZhangCZhen]

Would appreciate if you could release the code for converting the color map to a label map. Thanks!

[junyanz]

We just assigned the label to its nearest neighbor color in the label dictionary. I found some old code wrote by @phillipi and/or @tinghuiz. You probably want to rewrite it in Python.

--[[
['road', 'sidewalk', 'building', 'wall', 'fence',
'pole', 'traffic light', 'traffic sign', 'vegetation', 'terrain',
'sky', 'person', 'rider', 'car', 'truck',
'bus', 'train', 'motorcycle', 'bicycle']
--]]

-- label ids are index into label_colors minus one
function load_label_map_cityscapes()

	filler = -1000

	label_colors = torch.FloatTensor({{128, 64,128}, -- road
									  {244, 35,232}, -- sidewalk
									  {70, 70, 70},  -- building
									  {102,102,156}, -- wall
									  {190,153,153}, -- fence
									  {153,153,153}, -- pole
									  {250,170, 30}, -- traffic light
									  {220,220,  0}, -- traffic sign
									  {107,142, 35}, -- vegetation
									  {152,251,152}, -- terrain
									  { 70,130,180}, -- sky
									  {220, 20, 60}, -- person
									  {255,  0,  0}, -- rider
									  {  0,  0,142}, -- car
									  {  0,  0, 70}, -- truck
									  {  0, 60,100}, -- bus
									  {  0, 80,100}, -- train
									  {  0,  0,230}, -- motorcycle
									  {119, 11, 32}}) -- bicycle	
	return label_colors
end

-- images are CxXxY (or maybe CxYxX?)
function image_to_labels_cityscapes(img)

	img = img:float():mul(255)

	-- load the labels
	label_colors = load_label_map_cityscapes()
	Nlabels = label_colors:size(1)

	label_colors_img = torch.FloatTensor(Nlabels,img:size(1),img:size(2),img:size(3))
	for i=1, label_colors_img:size(1) do
		-- fill in three color channels
		label_colors_img[i][1]:fill(label_colors[i][1])
		label_colors_img[i][2]:fill(label_colors[i][2])
		label_colors_img[i][3]:fill(label_colors[i][3])
	end

	-- assign label that is closest in color for each output pixel
	dists = torch.FloatTensor(label_colors_img:size(1),img:size(2),img:size(3))
	for i=1, label_colors_img:size(1) do
		dists[i] = torch.add(img,torch.mul(label_colors_img[i],-1)):pow(2):sum(1):squeeze()
	end

	y, ii = torch.min(dists, 1)
	ii = ii[1]
	ii = ii-1
	ii = ii:float():div(255)

	return ii
end

[phillipi]

Can confirm that's the original code we used for the paper. Thanks for digging it up @junyanz.

[ZhangCZhen]

@junyanz Thank you very much for your help, I will study this code carefully and try to reproduce it. This work is really amazing, thanks again !!

[A-Little-Nut]

We just assigned the label to its nearest neighbor color in the label dictionary. I found some old code wrote by @phillipi and/or @tinghuiz. You probably want to rewrite it in Python.

--[[
['road', 'sidewalk', 'building', 'wall', 'fence',
'pole', 'traffic light', 'traffic sign', 'vegetation', 'terrain',
'sky', 'person', 'rider', 'car', 'truck',
'bus', 'train', 'motorcycle', 'bicycle']
--]]

-- label ids are index into label_colors minus one
function load_label_map_cityscapes()

	filler = -1000

	label_colors = torch.FloatTensor({{128, 64,128}, -- road
									  {244, 35,232}, -- sidewalk
									  {70, 70, 70},  -- building
									  {102,102,156}, -- wall
									  {190,153,153}, -- fence
									  {153,153,153}, -- pole
									  {250,170, 30}, -- traffic light
									  {220,220,  0}, -- traffic sign
									  {107,142, 35}, -- vegetation
									  {152,251,152}, -- terrain
									  { 70,130,180}, -- sky
									  {220, 20, 60}, -- person
									  {255,  0,  0}, -- rider
									  {  0,  0,142}, -- car
									  {  0,  0, 70}, -- truck
									  {  0, 60,100}, -- bus
									  {  0, 80,100}, -- train
									  {  0,  0,230}, -- motorcycle
									  {119, 11, 32}}) -- bicycle	
	return label_colors
end

-- images are CxXxY (or maybe CxYxX?)
function image_to_labels_cityscapes(img)

	img = img:float():mul(255)

	-- load the labels
	label_colors = load_label_map_cityscapes()
	Nlabels = label_colors:size(1)

	label_colors_img = torch.FloatTensor(Nlabels,img:size(1),img:size(2),img:size(3))
	for i=1, label_colors_img:size(1) do
		-- fill in three color channels
		label_colors_img[i][1]:fill(label_colors[i][1])
		label_colors_img[i][2]:fill(label_colors[i][2])
		label_colors_img[i][3]:fill(label_colors[i][3])
	end

	-- assign label that is closest in color for each output pixel
	dists = torch.FloatTensor(label_colors_img:size(1),img:size(2),img:size(3))
	for i=1, label_colors_img:size(1) do
		dists[i] = torch.add(img,torch.mul(label_colors_img[i],-1)):pow(2):sum(1):squeeze()
	end

	y, ii = torch.min(dists, 1)
	ii = ii[1]
	ii = ii-1
	ii = ii:float():div(255)

	return ii
end

The code is matlab code? I am puzzled.

[junyanz]

The code is in Lua. If you don't want to use Lua, I recommend that you rewrite it in your preferred language.