How does the fixed mask generated?

[codaibk]

Hi, Thank for your work.
I want to generate the fixed mask for my image with the mask's size = 1/2 image's size and the mask is on the right side of original image.
so what i should change to get that because so far your code generate random mask in random_bbox and bbox2mask function.
Original

mask

Thanks

[JiahuiYu]

You just need to change random_bbox and bbox2mask function by fixing the return results as your requirements. See lines here where the output represents (top, left, height, width) bounding box.

[codaibk]

Yes. I am trying to change 2 those functions. I already have the bounding box. so do i need to change anything in bbox2mask?
btw, can you explain 2 variables delta_h and delta_w used for what in these lines ?

h = np.random.randint(delta_h//2+1)
w = np.random.randint(delta_w//2+1))
mask[:, bbox[0]+h:bbox[0]+bbox[2]-h,
             bbox[1]+w:bbox[1]+bbox[3]-w, :] = 1.

Thanks

[JiahuiYu]

@codaibk Hi, delta_h and delta_w is a way to random sample the aspect ratio of bounding box in our case. I believe in your case, you can simply set them to 0.

[codaibk]

@JiahuiYu . Hi , Thanks for your reply
About the fixed mask, i saw the result from tensorboard. it seems not all of my mask is correct (The mask is correct in raw_incomplete_predicted_complete but it is wrong in static view in TensorBoard log as shown in the following pict)

• The result with raw_incomplete_predicted_complete
  
• The result with statuc view
  
• do you have any suggestion?
• This my code in random_bbox and bbox2mask function

def random_bbox(config):
    t = tf.constant(0)
    l = tf.constant(256)
    h = tf.constant(config.HEIGHT)
    w = tf.constant(config.WIDTH)
    return (t, l, h, w)
def bbox2mask(bbox, config, name='mask'):
    def npmask(bbox, height, width, delta_h, delta_w):
        mask = np.zeros((1, height, width, 1), np.float32)
        h = 0 
        w = 0
        mask[:, bbox[0]+h:bbox[0]+bbox[2]-h,
             bbox[1]+w:bbox[1]+bbox[3]-w, :] = 1.
        return mask
    with tf.variable_scope(name), tf.device('/cpu:0'):
        img_shape = config.IMG_SHAPES
        height = img_shape[0]
        width = img_shape[1]
        mask = tf.py_func(
            npmask,
            [bbox, height, width,
             config.MAX_DELTA_HEIGHT, config.MAX_DELTA_WIDTH],
            tf.float32, stateful=False)
        mask.set_shape([1] + [height, width] + [1])
    return mask

[codaibk]

I found the problem. Because in build_static_infer_graph function, you assign bbox without calling random_bbox function. It is ok now after i assign random_bbox value for bbox. Here

[xuyifeng-nwpu]

@codaibk Why can not change the mask picture directly?
How to assign random_bbox value for bbox?
Thanks.

[codaibk]

@xuyifeng-nwpu Because in this project, they generate the mask using bounding box. Therefore, you need to change random_bbox and bbox2mask functions to get your expected mask. In here you just need to change these 2 lines by calling random_bbox function : bbox = random_bbox(config)

[xuyifeng-nwpu]

@codaibk
If we test the random_bbox inpainting, do we need change the code --mask examples/center_mask_256.png in the below code?

ImageNet 256x256 input

python test.py --image examples/imagenet/imagenet_patches_ILSVRC2012_val_00000827_input.png --mask examples/center_mask_256.png --output examples/output.png --checkpoint_dir model_logs/release_imagenet_256

[codaibk]

@xuyifeng-nwpu i dont understand what you mean. Do you mean you will have random mask which is not same with this project? if yes, from my understanding, you need to change --mask examples/center_mask_256.png with corresponding mask in your case.

[xuyifeng-nwpu]

@codaibk
When we inpaint the right half part, did the mask picture set as follows?
explain:left black, right white.

The input image and result show as follows:
input:

the output:

My result is not nice. How about your half-inpainting effect?
Do you use qq or wechat? I think you are a Chinese.

[codaibk]

@xuyifeng-nwpu. I am still running my model. it haven't finish yet. I think your mask is correct. Maybe you can ask the author of this paper about your result.
Btw, i am not chinese :)

[xuyifeng-nwpu]

Do you re-train your data? I only use the author's imagenet model. I guess that the large area(half) inpainting of many categories of image maybe have not good result, especially on the edge of the image. If you try the experiment in a few kind of image(such as skirt) , the model may be effective. I look forward to your good results. Tell me please!

[JiahuiYu]

@codaibk Thanks for your help and answers to others! I appreciate!

@xuyifeng-nwpu Have you trained your model on your dataset with your masks?

[xuyifeng-nwpu]

@JiahuiYu
Have you trained your model on your dataset with your masks? NO, now. I will train with my pictures.
The green apple image is selected from imagenet dataset.
The test model is your imagenet model. I guess that the large area inpainting is difficult.

[JiahuiYu]

@xuyifeng-nwpu Thanks for your information. During training, the largest mask width/height is half of image size. Which means if image is 256x256, the maximum of mask during training is 128x128. I guess that's the reason why it fails in your case (128x256). Because it never saw such mask during training.

[JiahuiYu]

@xuyifeng-nwpu But I have seen (in email) one who also tries to inpainting half of an image like yours. The performance is pretty good and I guess this code repo should work for your requirements. Just a reference.

[xuyifeng-nwpu]

@JiahuiYu
Could you tell me whether the good result (in email) generated by your model or other's retrain model?
If retrain model, do the model train by 128x256?
Thanks!

[xuyifeng-nwpu]

@codaibk @JiahuiYu
Hi, I have a question. If i retrain my dataset for right-half inpainting(128x256), should I change the inpaint_model.py? Can you tell me how to change? Sorry ,I am beginner.

Thanks!

[JiahuiYu]

Hi he retrains the model on half masks. To change, you need to understand how masks in this code are generated. This issue has some code that may help.