Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New attention layers #44

Open
57 tasks
soran-ghaderi opened this issue Sep 14, 2022 · 0 comments
Open
57 tasks

New attention layers #44

soran-ghaderi opened this issue Sep 14, 2022 · 0 comments
Labels
documentation Improvements or additions to documentation enhancement New feature or request good first issue Good for newcomers issue list A list of issues closely related jax Related to JAX numpy Related to Numpy pytorch Related to Pytorch Stale tensorflow Related to Tensorflow tests Related to tests

Comments

@soran-ghaderi
Copy link
Member

soran-ghaderi commented Sep 14, 2022

A list of new attention layers to be added

You can pick any of the following frameworks and implement your layers:
Tensorflow
Pytorch
JAX
Numpy

To contribute, please:

  1. Create a new issue, copy an available (not closed and not opened by someone else) subtask name and paste it in the title followed by the reference of the task. (ex. subtask_name #source_issue_number). Make sure the subtask is not already opened by someone else. Another way is to hover over the subtasks and click on the Open convert to issue
  2. Copy and paste your new issue link here in the comments.
  3. Fork the repository
  4. Add your changes
  5. Create a pull request and mention your issue link

Please note that we use Numpy style for the docstrings and provide examples as much as possible!
Also please try to provide unit tests using Pytest

Whenever you need help with implementation or any other related issues you might be dealing with, please reach out to me via discussions or Discord server community @soran-ghaderi or @sigma1326

  • Strided Attention
  • Fixed Factorized Attention
  • Additive Attention
  • RAN
  • RAM
  • STN
  • Temporal Attention
  • Channel Attention
  • Axial Attention
  • Sliding Window Attention
  • Global And Sliding Window Attention
  • Dilated Sliding Window Attention
  • Dynamic Convolution
  • Content-Based Attention
  • Global-Local Attention
  • Attention Gate
  • Class Attention
  • Location-Based Attention
  • Channel-Wise Soft Attention
  • FAVOR+
  • Disentangled Attention Mechanism
  • Location Sensitive Attention
  • LSH Attenention
  • TAM
  • SRM
  • BAM
  • Set Transformer
  • Coordinate Attention
  • BigBird
  • Rendezvous
  • Adaptive Masking
  • DANet
  • Bi-Attention
  • RGA
  • SEAM
  • SPNet
  • DMA
  • GALA
  • Neighborhood Attention
  • Channel Squeeze And Spatial Excitation
  • GCT
  • Routing Attention
  • Cross-Covariance Attention
  • 3D SA
  • Sparse Sinkhorn Attention
  • Concurrent Spatial And Channel Squeeze And Excitation
  • Deformable ConvNets
  • SCA-CNN
  • Channel And Spatial Attention
  • Locally-Grouped Self-Attention
  • Class ActivationGuided Attention Mechanism
  • Factorized Dense Synthesized Attention
  • HyperHyperNetwork
  • ProCAN
  • scSE
  • MHMA
  • Branch Attention
@soran-ghaderi soran-ghaderi added documentation Improvements or additions to documentation enhancement New feature or request good first issue Good for newcomers issue list A list of issues closely related tests Related to tests tensorflow Related to Tensorflow pytorch Related to Pytorch jax Related to JAX numpy Related to Numpy labels Sep 14, 2022
@github-actions github-actions bot added the Stale label Apr 25, 2023
@soran-ghaderi soran-ghaderi reopened this May 30, 2023
@soran-ghaderi soran-ghaderi reopened this Jan 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request good first issue Good for newcomers issue list A list of issues closely related jax Related to JAX numpy Related to Numpy pytorch Related to Pytorch Stale tensorflow Related to Tensorflow tests Related to tests
Development

No branches or pull requests

1 participant