Added Swish and Mish activations #15808

thebhatman · 2019-10-30T07:32:37Z

I have added the Swish and Mish activation functions. This resolves #15693

###This Pull request changes
[Feature addition] New activation functions Swish and Mish.

force_builders=Custom
buildworker:Custom=linux-4
build_image:Custom=ubuntu-cuda:18.04

build_image:Custom Mac=openvino-2019r3.0
test_modules:Custom Mac=dnn,python2,python3,java

modules/dnn/src/layers/elementwise_layers.cpp

modules/dnn/src/cuda4dnn/kernels/activations.hpp

dkurt · 2019-10-30T14:14:05Z

Hi! Thanks for contribution.

Please check that all the backends are valid (OpenCL, CUDA, Intel's Inference Engine). Otherwise keep only default C++ implementation.

modules/dnn/src/cuda/activations.cu

asmorkalov · 2019-11-22T07:14:18Z

cc @VadimLevin

dkurt · 2019-11-22T16:06:49Z

@thebhatman, Please add the tests for both layers.

thebhatman · 2019-11-22T16:12:30Z

Yeah I will add the tests. Where exactly are the tests for activations? Should I define sample neural networks with these activations ? Or is there a test file with all activation functions such as sigmoid and tanh?

dkurt · 2019-11-22T17:06:23Z

@thebhatman, So due the activations are simple, you can test them without test data. So you can just create a test which applies all the backends and compares with reference data (which is computed inside the test). Take a look at test_layers.cpp and test_halide_layers.cpp. The only parameter you need to vary is target and backend.

thebhatman · 2019-11-22T18:44:33Z

I have added the tests in test_halide_layers.cpp and they are passing. It seems like all tests in test_layers.cpp use Caffe Models to test layers by reading data from a file from opencv_extra. Is there something else to be done?

dkurt · 2019-11-23T07:58:25Z

modules/dnn/test/test_halide_layers.cpp

@@ -583,7 +583,7 @@ TEST_P(NoParamActivation, Accuracy)
    testInPlaceActivation(lp, backendId, targetId);
 }
 INSTANTIATE_TEST_CASE_P(Layer_Test_Halide, NoParamActivation, Combine(
-/*type*/ Values("TanH", "Sigmoid", "AbsVal", "BNLL"),
+/*type*/ Values("TanH", "Sigmoid", "AbsVal", "BNLL", "Swish", "Mish"),


Good choice! 😄

The CUDA tests are passing for the Swish activation.

The test for DNN_TARGET_CUDA is passing for the Mish activation. The test for DNN_TARGET_CUDA_FP16 is failing.

[ RUN ] Layer_Test_Halide/NoParamActivation.Accuracy/21, where GetParam() = ("Mish", CUDA/CUDA_FP16) .../opencv/modules/dnn/test/test_common.impl.hpp:68: Failure Expected: (normL1) <= (l1), actual: 0.0288897 vs 0.004 .../opencv/modules/dnn/test/test_common.impl.hpp:71: Failure Expected: (normInf) <= (lInf), actual: 0.0909604 vs 0.02

This means that the errors in the outputs are more than the accepted threshold.

You will have to increase the error tolerance for Mish activation when target is DNN_TARGET_CUDA_FP16. I am not sure what would be the correct way to do this. In the worst case, the test can be skipped for the FP16 target.

The test calls void testInPlaceActivation(LayerParams& lp, Backend backendId, Target targetId) which in turn calls void test(Mat& input, Net& net, Backend backendId, Target targetId, bool skipCheck = false). I think currently there is no way to pass custom thresholds to test. A solution could be to pass thresholds to testInPlaceActivation and test as arguments.

The default Thresholds are being called from here https://github.com/thebhatman/opencv/blob/f6221044661dedbf4ad729c8f24c2b72ff37aca0/modules/dnn/test/test_common.hpp#L113. I don't see a way to pass the thresholds as arguments without changing the prototype of testInPlaceActivation .

I think you should add new parameters (l1 and lInf) to test and testInPlaceActivation. Their default values would be 0.0.

Inside the test function:

if l1 and lInf arguments are zero, set them to the values given by getDefaultThresholds

if l1 and lInf arguments are not zero, use them

@dkurt is this ok?

@YashasSamaga, yes, that would be fine. Thanks!

thebhatman · 2019-11-24T06:00:49Z

Is this PR ready to be merged now?

YashasSamaga · 2019-11-24T07:02:10Z

To enable the CUDA backend, you have to tick the following CMake options:

WITH_CUDA
WITH_CUDNN
OPENCV_DNN_CUDA

Are you able to build the CUDA backend on your PC?

The CUDA build is failing on CI. Please have a look at CI build. There are errors in the compile log (open the log and search for "error:").

The CI does not run tests on CUDA devices. Hence, you (or someone) has to run them on their PC to verify that the tests are passing. Let me know when you are done with the PR, I'll build and test the PR once.

thebhatman · 2019-11-24T07:54:32Z

The error is because of the log function using device::log;. The log is acting on __half type. It is only defined for long double and float.

thebhatman · 2019-11-24T11:06:49Z

Thanks @YashasSamaga . The CUDA builds are now passing. They were failing due to type mismatch errors. I resolved them by using log1pexp instead of using log and exp separately.

dkurt · 2019-11-24T14:53:31Z

@thebhatman, there is no still OpenCL kernels implementation. Please add it or remove unused applyOCL.

dkurt

👍
@alalek, Can we merge it now to master? I'll backport it to 3.4 with https://github.com/dkurt/opencv/tree/thebhatman/Mish_swish

please do not merge yet

modules/dnn/src/layers/elementwise_layers.cpp

dkurt

👍

@alalek, can we merge this PR to master and another one to 3.4 branch (#16025)?

cansik · 2020-04-27T13:39:57Z

Has this already been released? In 4.3.0, I still get the error that it is unsupported:

OpenCV(4.3.0) /Users/travis/build/bytedeco/javacpp-presets/opencv/cppbuild/macosx-x86_64/opencv-4.3.0/modules/dnn/src/darknet/darknet_io.cpp:821: error: (-212:Parsing error) Unsupported activation: mish in function 'ReadDarknetFromCfgStream'

dkurt · 2020-04-27T14:24:28Z

@cansik, If I'm not mistaken, this fix for TensorFlow importer. Please track #17148

BlueNotesRobot · 2020-06-11T10:34:05Z

From what I found, mish is available in 3.4.10 but not in 4.3.0.
Would be keen to know when we can expect it in 4.3.0

YashasSamaga · 2020-06-11T12:07:53Z

@BlueNotesRobot It is already there on master.

BlueNotesRobot · 2020-06-12T06:46:32Z

Thanks @YashasSamaga I just compiled the newest master and can confirm it worked.
Master is currently at version '4.4.0-pre'.

* Added Swish and Mish activations * Fixed whitespace errors * Kernel implementation done * Added function for launching kernel * Changed type of 1.0 * Attempt to add test for Swish and Mish * Resolving type mismatch for log * exp from device * Use log1pexp instead of adding 1 * Added openCL kernels

thebhatman added 2 commits October 30, 2019 12:55

Added Swish and Mish activations

ae2d19b

Fixed whitespace errors

1448907

dkurt reviewed Oct 30, 2019

View reviewed changes

modules/dnn/src/layers/elementwise_layers.cpp Show resolved Hide resolved

dkurt reviewed Oct 30, 2019

View reviewed changes

modules/dnn/src/cuda4dnn/kernels/activations.hpp Show resolved Hide resolved

thebhatman added 2 commits October 30, 2019 20:07

Kernel implementation done

85e848a

Added function for launching kernel

8de6b9d

YashasSamaga reviewed Nov 15, 2019

View reviewed changes

modules/dnn/src/cuda/activations.cu Outdated Show resolved Hide resolved

Changed type of 1.0

54b5906

Attempt to add test for Swish and Mish

afd5392

thebhatman force-pushed the Mish_swish branch from 56b8404 to afd5392 Compare November 22, 2019 18:28

dkurt reviewed Nov 23, 2019

View reviewed changes

thebhatman added 3 commits November 24, 2019 13:32

Resolving type mismatch for log

95cf558

exp from device

de3d3a0

Use log1pexp instead of adding 1

fbf797f

Added openCL kernels

f622104

YashasSamaga mentioned this pull request Nov 28, 2019

fix and enable tests for DNN_TARGET_CUDA_FP16 #16010

Merged

dkurt previously approved these changes Nov 29, 2019

View reviewed changes

dkurt self-assigned this Nov 29, 2019

dkurt reviewed Nov 29, 2019

View reviewed changes

modules/dnn/src/layers/elementwise_layers.cpp Outdated Show resolved Hide resolved

thebhatman added 2 commits November 30, 2019 01:35

Removed InfEngine

fd308e7

InfEngine backend not implemented

21e45ca

dkurt reviewed Nov 30, 2019

View reviewed changes

modules/dnn/src/layers/elementwise_layers.cpp Outdated Show resolved Hide resolved

Removed IE backendID

ba6fcd3

dkurt mentioned this pull request Dec 1, 2019

Port Swish and Mish layers #16025

Merged

dkurt approved these changes Dec 1, 2019

View reviewed changes

alalek merged commit 78c5e41 into opencv:master Dec 1, 2019

AlexeyAB mentioned this pull request Dec 4, 2019

EfficientNetb0-Yolo speed slow AlexeyAB/darknet#4447

Open

alalek mentioned this pull request Sep 23, 2021

How do I add a new activation function to Opencv? #20737

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Swish and Mish activations #15808

Added Swish and Mish activations #15808

thebhatman commented Oct 30, 2019 •

edited by alalek

Loading

dkurt commented Oct 30, 2019

asmorkalov commented Nov 22, 2019

dkurt commented Nov 22, 2019

thebhatman commented Nov 22, 2019

dkurt commented Nov 22, 2019

thebhatman commented Nov 22, 2019

dkurt Nov 23, 2019

YashasSamaga Nov 24, 2019 •

edited

Loading

thebhatman Nov 24, 2019

YashasSamaga Nov 25, 2019 •

edited

Loading

dkurt Nov 25, 2019

thebhatman commented Nov 24, 2019

YashasSamaga commented Nov 24, 2019 •

edited

Loading

thebhatman commented Nov 24, 2019 •

edited

Loading

thebhatman commented Nov 24, 2019

dkurt commented Nov 24, 2019

dkurt left a comment

dkurt left a comment

cansik commented Apr 27, 2020

dkurt commented Apr 27, 2020

BlueNotesRobot commented Jun 11, 2020

YashasSamaga commented Jun 11, 2020

BlueNotesRobot commented Jun 12, 2020

Added Swish and Mish activations #15808

Added Swish and Mish activations #15808

Conversation

thebhatman commented Oct 30, 2019 • edited by alalek Loading

dkurt commented Oct 30, 2019

asmorkalov commented Nov 22, 2019

dkurt commented Nov 22, 2019

thebhatman commented Nov 22, 2019

dkurt commented Nov 22, 2019

thebhatman commented Nov 22, 2019

dkurt Nov 23, 2019

Choose a reason for hiding this comment

YashasSamaga Nov 24, 2019 • edited Loading

Choose a reason for hiding this comment

thebhatman Nov 24, 2019

Choose a reason for hiding this comment

YashasSamaga Nov 25, 2019 • edited Loading

Choose a reason for hiding this comment

dkurt Nov 25, 2019

Choose a reason for hiding this comment

thebhatman commented Nov 24, 2019

YashasSamaga commented Nov 24, 2019 • edited Loading

thebhatman commented Nov 24, 2019 • edited Loading

thebhatman commented Nov 24, 2019

dkurt commented Nov 24, 2019

dkurt left a comment

Choose a reason for hiding this comment

dkurt left a comment

Choose a reason for hiding this comment

cansik commented Apr 27, 2020

dkurt commented Apr 27, 2020

BlueNotesRobot commented Jun 11, 2020

YashasSamaga commented Jun 11, 2020

BlueNotesRobot commented Jun 12, 2020

thebhatman commented Oct 30, 2019 •

edited by alalek

Loading

YashasSamaga Nov 24, 2019 •

edited

Loading

YashasSamaga Nov 25, 2019 •

edited

Loading

YashasSamaga commented Nov 24, 2019 •

edited

Loading

thebhatman commented Nov 24, 2019 •

edited

Loading