Colab demo doesn't work #39

blnfb · 2022-01-27T02:32:30Z

I would love to try out this model but I am struggling with installation. The Colab demo does not work either and gives the error:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/content/Mask2Former/mask2former/modeling/pixel_decoder/ops/functions/ms_deform_attn_func.py in <module>()
     21 try:
---> 22     import MultiScaleDeformableAttention as MSDA
     23 except ModuleNotFoundError as e:

ModuleNotFoundError: No module named 'MultiScaleDeformableAttention'

During handling of the above exception, another exception occurred:

ModuleNotFoundError                       Traceback (most recent call last)
7 frames
/content/Mask2Former/mask2former/modeling/pixel_decoder/ops/functions/ms_deform_attn_func.py in <module>()
     27         "\t`sh make.sh`\n"
     28     )
---> 29     raise ModuleNotFoundError(info_string)
     30 
     31 

ModuleNotFoundError: 

Please compile MultiScaleDeformableAttention CUDA op with the following commands:
	`cd mask2former/modeling/pixel_decoder/ops`
	`sh make.sh`


---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

The text was updated successfully, but these errors were encountered:

bowenc0221 · 2022-01-27T05:19:33Z

Have you restarted runtime after installation?

blnfb · 2022-01-27T16:30:47Z

Yes I pressed the 'Restart runtime' button that appeared at the bottom of the cell. But I tried it again and this time I think I restarted after the running the second installation cell and it works for me now.

AK391 · 2022-02-08T17:12:13Z

@bowenc0221 also get this issue when running in colab on cpu

RuntimeError Traceback (most recent call last)
in ()
9 cfg.MODEL.MASK_FORMER.TEST.PANOPTIC_ON = True
10 predictor = DefaultPredictor(cfg)
---> 11 outputs = predictor(im)

16 frames
/content/Mask2Former/mask2former/modeling/pixel_decoder/ops/functions/ms_deform_attn_func.py in forward(ctx, value, value_spatial_shapes, value_level_start_index, sampling_locations, attention_weights, im2col_step)
35 ctx.im2col_step = im2col_step
36 output = MSDA.ms_deform_attn_forward(
---> 37 value, value_spatial_shapes, value_level_start_index, sampling_locations, attention_weights, ctx.im2col_step)
38 ctx.save_for_backward(value, value_spatial_shapes, value_level_start_index, sampling_locations, attention_weights)
39 return output

RuntimeError: Not implemented on the CPU

bowenc0221 · 2022-02-08T17:51:09Z

The current ms_deform_attn cuda kernel only supports GPU. However, there is a pure PyTorch implementation here:

Mask2Former/mask2former/modeling/pixel_decoder/ops/modules/ms_deform_attn.py

Lines 116 to 119 in 012049c

    
           output = MSDeformAttnFunction.apply( 
        
               value, input_spatial_shapes, input_level_start_index, sampling_locations, attention_weights, self.im2col_step) 
        
           # # For FLOPs calculation only 
        
           # output = ms_deform_attn_core_pytorch(value, input_spatial_shapes, sampling_locations, attention_weights)

Maybe you can try ms_deform_attn_core_pytorch instead of MSDeformAttnFunction.apply when using CPU.

AK391 · 2022-02-08T18:34:35Z

@bowenc0221 thanks could the colab be updated to support cpu? I am currently testing it to add the models to https://huggingface.co/spaces as a web demo, similar to

github: https://github.com/facebookresearch/omnivore
Spaces: https://huggingface.co/spaces/akhaliq/omnivore

Spaces is completely free, and I can help setup a Gradio Space. Here are some getting started instructions if you'd prefer to do it yourself: https://huggingface.co/blog/gradio-spaces

bowenc0221 · 2022-02-09T15:45:04Z

@AK391 thanks for adding it to huggingface space!

I've updated the code to support inference on CPU (16c3bee). Could you try it again?

AK391 · 2022-02-09T16:43:36Z

@bowenc0221 thanks working on it now, also the models can be added to the facebook organization on Huggingface: https://huggingface.co/facebook

Would you be interested in becoming part of the Facebook organization on the hub? This allows you to have write access to all model repos of Facebook, which means you can easily upload new checkpoints, write model cards, etc.

Once a model is added it can be made into a Gradio space with a few lines of code

for example in xm_transformer

https://huggingface.co/spaces/facebook/xm_transformer_600m

gr.Interface.load("huggingface/facebook/xm_transformer_600m-es_en-multi_domain").launch()

can join here: https://huggingface.co/welcome

AK391 · 2022-02-09T23:16:29Z

@bowenc0221 inference on cpu is working in colab thanks, I setup the space here https://huggingface.co/spaces/akhaliq/Mask2Former but I get this error

Traceback (most recent call last):
File "setup.py", line 76, in
ext_modules=get_extensions(),
File "setup.py", line 52, in get_extensions
raise NotImplementedError('CUDA_HOME is None. Please set environment variable CUDA_HOME.')
NotImplementedError: CUDA_HOME is None. Please set environment variable CUDA_HOME.

bowenc0221 · 2022-02-09T23:34:17Z

Installing the package requires CUDA (GPUs are not needed). Is CUDA available?

You can use the following command:

FORCE_CUDA=1 CUDA_HOME=/path/to/cuda python setup.py build install

AK391 · 2022-02-09T23:46:58Z

@bowenc0221 thanks, yes I tried that step, see the code here: https://huggingface.co/spaces/akhaliq/Mask2Former/blob/main/app.py#L16

however, cuda is not available

bowenc0221 · 2022-02-21T18:58:35Z

Hey @AK391, sorry for the late reply. I think there is an easy solution to avoid installing the custom op. Could you try to do the following:

Define an environment variable, e.g., HUGGINGFACE_DEMO
Do not import the following when HUGGINGFACE_DEMO=1.

Mask2Former/mask2former/modeling/pixel_decoder/ops/modules/ms_deform_attn.py

Line 24 in 0293b1c

from ..functions import MSDeformAttnFunction

For example:

if os.environ.get('HUGGINGFACE_DEMO', 0):
    MSDeformAttnFunction = None
else:
    from ..functions import MSDeformAttnFunction

AK391 · 2022-02-22T00:24:30Z

@bowenc0221 thanks, I opened a PR here #52 for the Huggingface demo

bh-cai · 2023-04-04T08:57:25Z

@bowenc0221 thanks, I opened a PR here #52 for the Huggingface demo

Excuse me, have you resolved the question above?

lianzheng-research · 2023-12-04T08:00:43Z

The current ms_deform_attn cuda kernel only supports GPU. However, there is a pure PyTorch implementation here:

Mask2Former/mask2former/modeling/pixel_decoder/ops/modules/ms_deform_attn.py

Lines 116 to 119 in 012049c

output = MSDeformAttnFunction.apply(

value, input_spatial_shapes, input_level_start_index, sampling_locations, attention_weights, self.im2col_step)

# # For FLOPs calculation only

# output = ms_deform_attn_core_pytorch(value, input_spatial_shapes, sampling_locations, attention_weights)

Maybe you can try ms_deform_attn_core_pytorch instead of MSDeformAttnFunction.apply when using CPU.

Thanks for your help! But I tested it on a random input and found that there is a significant error between using ms_deform_attn_core_pytorch under CPU mode and using MSDeformAttnFunction.apply under GPU mode.

Result obtrained by using ms_deform_attn_core_pytorch:
tensor([[[[ 1.8436,  2.2416,  2.6247,  ...,  2.7736,  2.7426,  2.1027],
          [ 2.5010,  3.0443,  3.3008,  ...,  2.9831,  2.9631,  2.3863],
          [ 2.9538,  3.5157,  3.5276,  ...,  3.2951,  2.8759,  2.1722],
          ...,
          [ 2.8571,  3.7810,  3.2169,  ...,  3.1239,  3.6230,  3.1102],
          [ 2.9726,  3.7958,  3.5246,  ...,  3.7332,  4.0925,  3.1939],
          [ 2.9812,  3.2457,  3.2969,  ...,  2.9720,  3.0884,  2.6482]],

         [[-3.9825, -3.8449, -4.4274,  ..., -4.3323, -4.5814, -3.9630],
          [-3.8496, -4.2992, -4.9432,  ..., -4.3094, -4.3040, -4.0190],
          [-4.4084, -4.7840, -4.8061,  ..., -4.6905, -4.1271, -3.5728],
          ...,
          [-4.4807, -5.3218, -4.5969,  ..., -4.8324, -5.4793, -4.2268],
          [-4.7795, -5.3368, -4.9663,  ..., -5.6892, -6.0757, -4.5740],
          [-4.5650, -4.6115, -4.4746,  ..., -4.4084, -4.3251, -4.2097]],

         [[-3.0353, -3.5850, -3.8838,  ..., -4.6120, -4.3296, -3.6653],
          [-3.7541, -4.4487, -4.5815,  ..., -4.4298, -4.8221, -3.9200],
          [-4.0843, -5.0629, -5.0538,  ..., -4.8248, -4.4019, -3.8944],
          ...,
          [-3.9654, -5.2787, -4.6193,  ..., -4.8375, -5.6787, -5.3515],
          [-4.3268, -5.5459, -5.3988,  ..., -5.7265, -6.5075, -5.4388],
          [-5.0671, -5.2403, -5.4108,  ..., -4.9114, -5.2505, -4.6957]],

         [[-4.2067, -4.4303, -4.8076,  ..., -4.4385, -5.2699, -5.5285],
          [-4.0580, -4.6923, -5.0086,  ..., -4.0468, -4.5669, -4.8352],
          [-5.1472, -5.4393, -5.1526,  ..., -4.5857, -4.1159, -4.2404],
          ...,
          [-4.9072, -5.6090, -4.4477,  ..., -4.1471, -5.1054, -4.6439],
          [-5.2460, -5.6856, -4.9846,  ..., -5.4047, -6.0951, -5.2036],
          [-6.4418, -5.6409, -5.5475,  ..., -4.7478, -5.0343, -5.0288]]]])

Result obtained by using MSDeformAttnFunction.apply:
tensor([[[[ 1.8438,  2.2418,  2.6247,  ...,  2.7730,  2.7421,  2.1017],
          [ 2.5010,  3.0440,  3.3010,  ...,  2.9818,  2.9627,  2.3849],
          [ 2.9538,  3.5151,  3.5274,  ...,  3.2940,  2.8751,  2.1706],
          ...,
          [ 2.8567,  3.7798,  3.2173,  ...,  3.1232,  3.6217,  3.1084],
          [ 2.9719,  3.7946,  3.5238,  ...,  3.7322,  4.0911,  3.1923],
          [ 2.9801,  3.2443,  3.2952,  ...,  2.9713,  3.0874,  2.6471]],

         [[-3.9825, -3.8448, -4.4270,  ..., -4.3318, -4.5808, -3.9619],
          [-3.8493, -4.2982, -4.9436,  ..., -4.3074, -4.3032, -4.0174],
          [-4.4081, -4.7829, -4.8064,  ..., -4.6886, -4.1262, -3.5713],
          ...,
          [-4.4809, -5.3203, -4.5975,  ..., -4.8315, -5.4775, -4.2247],
          [-4.7791, -5.3360, -4.9657,  ..., -5.6872, -6.0733, -4.5714],
          [-4.5640, -4.6103, -4.4728,  ..., -4.4075, -4.3242, -4.2080]],

         [[-3.0350, -3.5848, -3.8832,  ..., -4.6107, -4.3282, -3.6638],
          [-3.7525, -4.4475, -4.5803,  ..., -4.4276, -4.8206, -3.9177],
          [-4.0838, -5.0615, -5.0520,  ..., -4.8226, -4.3999, -3.8922],
          ...,
          [-3.9630, -5.2760, -4.6183,  ..., -4.8337, -5.6740, -5.3469],
          [-4.3242, -5.5429, -5.3960,  ..., -5.7220, -6.5034, -5.4351],
          [-5.0641, -5.2365, -5.4068,  ..., -4.9085, -5.2471, -4.6935]],

         [[-4.2063, -4.4301, -4.8070,  ..., -4.4376, -5.2689, -5.5274],
          [-4.0575, -4.6916, -5.0092,  ..., -4.0446, -4.5659, -4.8326],
          [-5.1466, -5.4383, -5.1529,  ..., -4.5832, -4.1145, -4.2369],
          ...,
          [-4.9071, -5.6081, -4.4490,  ..., -4.1483, -5.1057, -4.6425],
          [-5.2457, -5.6848, -4.9844,  ..., -5.4051, -6.0948, -5.2022],
          [-6.4400, -5.6390, -5.5436,  ..., -4.7483, -5.0334, -5.0283]]]])

Can you give me some suggestions to make up for this error?

masterpv · 2024-04-15T04:08:49Z

does anybody have error free python code or worked on it for this mask2former demo?
@AK391 @lianzheng-research @bowenc0221 @bh-cai @blnfb , I need this very bad. I started it recently it would be very helpful if you guys could share it to my mail venkateshpunugoti001@gmail.com

bowenc0221 closed this as completed Feb 22, 2022

1poorman mentioned this issue Dec 19, 2022

TypeError: 'MultiPolygon' object is not iterable #140

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Colab demo doesn't work #39

Colab demo doesn't work #39

blnfb commented Jan 27, 2022

bowenc0221 commented Jan 27, 2022

blnfb commented Jan 27, 2022

AK391 commented Feb 8, 2022

bowenc0221 commented Feb 8, 2022

AK391 commented Feb 8, 2022

bowenc0221 commented Feb 9, 2022

AK391 commented Feb 9, 2022 •

edited

AK391 commented Feb 9, 2022 •

edited

bowenc0221 commented Feb 9, 2022

AK391 commented Feb 9, 2022 •

edited

bowenc0221 commented Feb 21, 2022

AK391 commented Feb 22, 2022

bh-cai commented Apr 4, 2023

lianzheng-research commented Dec 4, 2023

masterpv commented Apr 15, 2024

Colab demo doesn't work #39

Colab demo doesn't work #39

Comments

blnfb commented Jan 27, 2022

bowenc0221 commented Jan 27, 2022

blnfb commented Jan 27, 2022

AK391 commented Feb 8, 2022

bowenc0221 commented Feb 8, 2022

AK391 commented Feb 8, 2022

bowenc0221 commented Feb 9, 2022

AK391 commented Feb 9, 2022 • edited

AK391 commented Feb 9, 2022 • edited

bowenc0221 commented Feb 9, 2022

AK391 commented Feb 9, 2022 • edited

bowenc0221 commented Feb 21, 2022

AK391 commented Feb 22, 2022

bh-cai commented Apr 4, 2023

lianzheng-research commented Dec 4, 2023

masterpv commented Apr 15, 2024

AK391 commented Feb 9, 2022 •

edited

AK391 commented Feb 9, 2022 •

edited

AK391 commented Feb 9, 2022 •

edited