get_mask cannot return valid mask

[Z-MU-Z]

Hi!
when running the demo,

src_img_path = "data/images/dog_00.jpg"
trg_img_path = "data/images/dog_59.jpg"
result = process_images(src_img_path, trg_img_path)

I found that the get_mask function cannot return a valid mask but an all-1 matrix. Is this a bug?

if DRAW_DENSE:
                if not Anno:
                    mask1 = get_mask(model, aug, img1, category[0])
                    mask2 = get_mask(model, aug, img2, category[-1])

[Z-MU-Z]

I also found that there were warnings when loading the model. Is this the reason?

backbone.feature_extractor.,category_head.clip.clip.,clip_head.clip.clip.
WARNING:fvcore.common.checkpoint:Removed 2483 ignored_state_dict keys from missing_keys
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.2.0.shortcut.weight' to the model due to incompatible shapes: (512, 2560, 1, 1) in the checkpoint but (512, 1280, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.2.0.conv1.weight' to the model due to incompatible shapes: (128, 2560, 1, 1) in the checkpoint but (128, 1280, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.3.0.shortcut.weight' to the model due to incompatible shapes: (512, 1920, 1, 1) in the checkpoint but (512, 1280, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.3.0.conv1.weight' to the model due to incompatible shapes: (128, 1920, 1, 1) in the checkpoint but (128, 1280, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.4.0.shortcut.weight' to the model due to incompatible shapes: (512, 960, 1, 1) in the checkpoint but (512, 640, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.4.0.conv1.weight' to the model due to incompatible shapes: (128, 960, 1, 1) in the checkpoint but (128, 640, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.5.0.shortcut.weight' to the model due to incompatible shapes: (512, 640, 1, 1) in the checkpoint but (512, 320, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.5.0.conv1.weight' to the model due to incompatible shapes: (128, 640, 1, 1) in the checkpoint but (128, 320, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Some model parameters or buffers are not found in the checkpoint:
backbone.feature_projections.2.0.conv1.weight
backbone.feature_projections.2.0.shortcut.weight
backbone.feature_projections.3.0.conv1.weight
backbone.feature_projections.3.0.shortcut.weight
backbone.feature_projections.4.0.conv1.weight
backbone.feature_projections.4.0.shortcut.weight
backbone.feature_projections.5.0.conv1.weight
backbone.feature_projections.5.0.shortcut.weight

[Junyi42]

Thank you for your interest in our research and for bringing these issues to our attention.

For the get_mask function, it is designed to return an all-1 matrix when it fails to detect a valid mask for the specified category. From the information you've provided, it seems that the issue might stem from the incorrect loading of the feature projection layers of the ODISE checkpoint.

Could you please try to modify the model-loading code from

model, aug = load_model(diffusion_ver=VER, image_size=SIZE, num_timesteps=TIMESTEP)

to

model, aug = load_model(diffusion_ver=VER, image_size=SIZE, num_timesteps=TIMESTEP, decoder_only=False).

This adjustment may better align with the U-Net features utilized by ODISE.

Alternatively, you might want to consider other segmentation methods, such as SAM, for obtaining the object mask. Our choice to use ODISE is mainly for convenience, and the mask is primarily for visualization purposes only.

I hope this could help resolve your queries. Please feel free to reach out If you encounter further issues.

[Z-MU-Z]

Thank you for your thoughtful response and helpful suggestions.
Modifying the model loading code to model, aug = load_model(diffusion_ver=VER, image_size=SIZE, num_timesteps=TIMESTEP, decoder_only=False) did resolve the issue.