Channel/Spatial/Element wise Attention Modules

[innat]

In the timm package, it provides some soft attention modules to building network blocks and I think it's a good fit here, for example:

• Squeeze-and-Excitation (SE): https://arxiv.org/abs/1709.01507 Cited by 10411
• CBAM: https://arxiv.org/abs/1807.06521 Cited by 3530
• Residual Attention https://arxiv.org/abs/1704.06904 Cited by 2177
• Dual Attention https://arxiv.org/abs/1809.02983 Cited by 1937
• Selective Kernel (SK) https://arxiv.org/abs/1903.06586 Cited by 667
• GCNet https://arxiv.org/abs/1904.11492 Cited by 598
• SAN https://ieeexplore.ieee.org/document/8954252 Cited by 566
• ECA-Net https://arxiv.org/abs/1910.03151 Cited by 471
• BAM https://arxiv.org/abs/1807.06514 Cited by 365
• A2-Nets https://arxiv.org/abs/1810.11579 Cited by 278
• Bottleneck Transformer https://arxiv.org/abs/2101.11605 Cited by 147
• Lambda Layer: https://arxiv.org/abs/2102.08602 Cited by 67
• SGENet https://arxiv.org/abs/1905.09646 Cited by 51
• HaloNet https://arxiv.org/abs/2103.12731 Cited by 51
• CoAtNet https://arxiv.org/abs/2106.04803 Cited by 35 ( < 50 current)

and many others.

[bhack]

I have not extensively checked about these in the model gardens components, but just picking the First One in the list:

https://github.com/tensorflow/models/blob/master/official/vision/beta/projects/yolo/modeling/layers/nn_blocks.py#L1170

[LukeWood]

Currently we are prioritizing components that are required to achieve state of the art results on specific tasks. I.e., imagenet1k classification, COCO object detection, etc. Any chance I could get some guidance as to where these components excel?

Thanks

[innat]

@LukeWood
The above list isn't meant to be the replacement for the current priority list. The point is whether these cv components are fit in keras-cv or not.

KerasCV is a repository of modular building blocks (layers, ... .

It can be considered as a to-do list, an interested contributor can get references from here in the coming days. Let me know if I miss something. Also, I think a few components should be already on the current priority list, for example, Squeeze-and-Excitation or CBAM.

[bhack]

II think that in this specific field you need to always find a balance between popupalirity, state of the art, sedimentation and the available human and computing resources you have at a specific point on the time dimension in your community.
So you need to define a policy with an equilibrium over all these tensions expecially if you want try to growth your code contributor community and not just your user base.

E.g. we are still investing resoruces for resnext-rs and waiting for the citation/popularity threshold for the next STOA:
https://arxiv.org/abs/2201.03545

Often an high popupalirity component/model is needed to be maintained just cause it is a reccurent baseline for new accademic work.

Another addition dimension/tension Is the computing resources required by a model and its own components.

[old-school-kid]

@innat @LukeWood
I have a two part question about this?

1. Are we adding modules like Spatial Attention Module and Channel Attention Module or should we be adding the whole model (example CBAM as whole) as an application with parameters to change configuration of the model?
2. As @bhack pointed out we cannot solely depend on state of the art and we have to take into account the popularity and its "promosing"prospects. Soif you could throw some light on that area too.

Could you post these separately so people could be assigned if interested and maybe some of the model could be prioritized.

[old-school-kid]

Ah, also can SWIN transformer be added to the list?

[innat]

@old-school-kid
about (1). I would prefer these as modules. For example, if I want to experiment with my custom model and channel-wise attention module I should do that with the API call.
about (2). I agree with those points. But it's not clear to me how to define promising things here. And what role tfa.layers can serve here.

I didn't make a separate post regarding the above modules because I was not sure (and still) whether it's welcomed or we may need some discussion before approaching. The discussion thread wasn't created when I posted it. Maybe that's the right place. But I think, if it's welcomed, the interested contributor may pick up their interest from the above list and send PR.

For Swin-Transformer, it's been asked, #15545