where is the alpha mask in the png

[stevedipaola]

Sorry newbie question - I just assumed the saved png has the alpha mask in it (the a of 8 bit each rgba) and it can be accessed in other programs image processing ( photoshop, gimp. pillow) so for instance, in my case, I can set up a process with it to mask out the background and create a new image without the background. Yet when I inspect the outputed png I do not see a transparency 8 bit matte. How does one access the matte as an image file.

[dbolya]

The default behavior is just to overlay the mask on top of the original image. If you want to output and alpha channel as an image instead, that's a simple change. Just replace this block:

yolact/eval.py

Lines 189 to 209 in 13bb0c6

	if args.display_masks and cfg.eval_mask_branch and num_dets_to_consider > 0:
	# After this, mask is of size [num_dets, h, w, 1]
	masks = masks[:num_dets_to_consider, :, :, None]
	# Prepare the RGB images for each mask given their color (size [num_dets, h, w, 1])
	colors = torch.cat([get_color(j, on_gpu=img_gpu.device.index).view(1, 1, 1, 3) for j in range(num_dets_to_consider)], dim=0)
	masks_color = masks.repeat(1, 1, 1, 3) * colors * mask_alpha
	# This is 1 everywhere except for 1-mask_alpha where the mask is
	inv_alph_masks = masks * (-mask_alpha) + 1
	# I did the math for this on pen and paper. This whole block should be equivalent to:
	# for j in range(num_dets_to_consider):
	# img_gpu = img_gpu * inv_alph_masks[j] + masks_color[j]
	masks_color_summand = masks_color[0]
	if num_dets_to_consider > 1:
	inv_alph_cumul = inv_alph_masks[:(num_dets_to_consider-1)].cumprod(dim=0)
	masks_color_cumul = masks_color[1:] * inv_alph_cumul
	masks_color_summand += masks_color_cumul.sum(dim=0)
	img_gpu = img_gpu * inv_alph_masks.prod(dim=0) + masks_color_summand

with

if args.display_masks and cfg.eval_mask_branch and num_dets_to_consider > 0:
        # After this, mask is of size [num_dets, h, w, 1]
        masks = masks[:num_dets_to_consider, :, :, None]
        img_gpu = (masks.sum(dim=0) >= 1).float().expand(-1, -1, 3).contiguous()
    else:
        img_gpu *= 0

and run evaluation with --display_bbox=False and --display_text=False.

---

But I can do you one better! If you instead replace the block with

if args.display_masks and cfg.eval_mask_branch and num_dets_to_consider > 0:
        # After this, mask is of size [num_dets, h, w, 1]
        masks = masks[:num_dets_to_consider, :, :, None]
        img_gpu *= (masks.sum(dim=0) >= 1).float().expand(-1, -1, 3)
    else:
        img_gpu *= 0

everything will already be masked out for you. Though you probably also want a chroma key so actually that might not be very useful now that I think about it. Yeah it's prob better just to use the mask because you'd need both anyway :/

[bayraktare]

Hi, @dbolya thanks for the explanation. In addition to this, I am just trying to mask out all the single instances from the image individually, for example if there are multiple chairs or Tv eetc I will mask out all of them individually or if there are single one of them, I will do the same thing. Every saved image will include one masked object. But it is just doing it for just one object then only gives black frames. My modified version your above code is like this:

if args.display_masks and cfg.eval_mask_branch and num_dets_to_consider > 0:
    masks = masks[:num_dets_to_consider, :, :, None]
    print('maskshape', (masks.shape))
    for i in range(num_dets_to_consider):
        msk = masks[i, :, :, None]
        mask = msk.view(1, 968, 1296, 1)
        print('newmaskshape', (mask.shape))
        img_gpu *= (mask.sum(dim=0) >= 1).float().expand(-1, -1, 3)
        img_numpy = (img_gpu * 255).byte().cpu().numpy()
        cv2.imwrite('.../mask'+str(i)+'.jpg', img_numpy)

Can you see where the problem is?
Thanks...

[dbolya]

@bayraktare That's because of img_gpu *= (mask.sum(dim=0) >= 1).float().expand(-1, -1, 3). You're masking the first instance out, saving that back to the image, and then masking the second instance out of the version of the image that was already masked, so only the overlapping region between the two masks remain. You're getting black frames because that overlap doesn't happen usually.

To fix, just change these lines:

        img_gpu *= (mask.sum(dim=0) >= 1).float().expand(-1, -1, 3)
        img_numpy = (img_gpu * 255).byte().cpu().numpy()

to

        img_gpu_masked = img_gpu * (mask.sum(dim=0) >= 1).float().expand(-1, -1, 3)
        img_numpy = (img_gpu_masked * 255).byte().cpu().numpy()

Untested so let me know if it doesn't work.

[bayraktare]

Yeah @dbolya that runs great. Thanks a lot.
In addition to that, if you add the following line you can also have the background as well.

img_background = img.byte().cpu().numpy() - (img_gpu_masked * 255).byte().cpu().numpy()

[Lucalucashhh]

Hi @dbolya, could you point out if I can only crop the bounding box? Thanks

[Marwen-Bhj]

Yeah @dbolya that runs great. Thanks a lot.
In addition to that, if you add the following line you can also have the background as well.

img_background = img.byte().cpu().numpy() - (img_gpu_masked * 255).byte().cpu().numpy()

@bayraktare I would like to have the background in green color (greenscreen mimicking) instead of the black one, do you have any insight how can I do that ?

[davodogster]

Thanks it worked. Had to change

msk = masks[i, :, :, None]
       mask = msk.view(1, 968, 1296, 1)

to

msk = masks[i, :, :, None]
        mask = msk.view(1,msk.shape[0], msk.shape[1], 1)

To make it work with any input size image

[hci-mkim]

if args.display_masks and cfg.eval_mask_branch and num_dets_to_consider > 0:
    # After this, mask is of size [num_dets, h, w, 1]
    masks = masks[:num_dets_to_consider, :, :, None]
    img_gpu = (masks.sum(dim=0) >= 1).float().expand(-1, -1, 3).contiguous()
    img_gpu = img_gpu*img/255  # Black/White Image * Original Image(Color) / 255
else:
    img_gpu *= 0

I just modified the above code from 189 lines to 209
This is how I get the black background and color human instance image :)