Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GELU layer for vision transformers #23219

Merged
merged 2 commits into from Feb 10, 2023
Merged

Conversation

fengyuentau
Copy link
Member

This PR adds the CPU and OCL kernels of GELU and GELU-Approximation layers.

Merge with opencv/opencv_extra#1044

References:

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

modules/dnn/src/layers/elementwise_layers.cpp Outdated Show resolved Hide resolved
Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@fengyuentau
Copy link
Member Author

@rogday Could you review if possible?

@vpisarev vpisarev removed the request for review from rogday February 10, 2023 08:45
Comment on lines 310 to 324
__kernel void GeluForward(const int n, __global T* in, __global T* out)
{
int index = get_global_id(0);
if(index < n)
out[index] = (T)0.5f * in[index] * ( (T)1.f + erf(in[index] * M_SQRT1_2) );
}

__kernel void GeluApproximationForward(const int n, __global T* in, __global T* out,
const KERNEL_ARG_DTYPE sqrt_2_pi,
const KERNEL_ARG_DTYPE coef_sqrt_2_pi)
{
int index = get_global_id(0);
if(index < n)
out[index] = (T)0.5f * in[index] * ( (T)1.f + tanh(in[index] * (sqrt_2_pi + coef_sqrt_2_pi * in[index] * in[index])) );
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use this OpenCL code:

__kernel void GeluForward(const int n, __global T* in, __global T* out)
{
    int index = get_global_id(0);
    if (index < n)
    {
        T x = in[index];
        out[index] = (T)0.5f * x * ( (T)1.f + erf(x * M_SQRT1_2) );
    }
}

__kernel void GeluApproximationForward(const int n, __global T* in, __global T* out)
{
    // see GeluApproximationConstants from .cpp
    const T sqrt_2_pi = 0.7978845834732056f;
    const T coef_sqrt_2_pi = 0.044714998453855515f * sqrt_2_pi;

    int index = get_global_id(0);
    if(index < n)
    {
        T x = in[index];
        out[index] = (T)0.5f * x * ( (T)1.f + tanh(x * (sqrt_2_pi + coef_sqrt_2_pi * x * x)) );
    }
}

and drop setKernelParams() method.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay

@alalek alalek merged commit c2b7c1f into opencv:4.x Feb 10, 2023
a-sajjad72 pushed a commit to a-sajjad72/opencv that referenced this pull request Mar 30, 2023
Add GELU layer for vision transformers

* add gelu and gelu approximation

* drop setKernelParams
@asmorkalov asmorkalov mentioned this pull request May 31, 2023
geversonsto pushed a commit to stodev-com-br/opencv that referenced this pull request Jun 3, 2023
Add GELU layer for vision transformers

* add gelu and gelu approximation

* drop setKernelParams
@fengyuentau fengyuentau deleted the add_gelu branch February 21, 2024 08:19
@fengyuentau fengyuentau mentioned this pull request Feb 21, 2024
48 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: dnn (onnx) ONNX suport issues in DNN module category: dnn feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants