Custom Activation Function: AttributeError: 'Activation' object has no attribute '__name__'

[wilderrodrigues]

Hi,

I'm working on the development of a custom activation function. It has already been tested with a number of neural networks architectures (e.g. shallow, intermedia, deep (4-layers) CNN (8-layers), ResNet, etc). It's working fine and actually beating ReLU in all the architectures used.

However, when trying to test it on a CNN and using the ModelCheckpoint callback, I got an error during serialisation of the Activation object:

AttributeError: 'Activation' object has no attribute '__name__'

I'm not disclosing my custom activation function yet because it has not been published in any paper, but if you want to reproduce the erro, do the following (this is just a simple ReLU example, not the one I'm working with):

from keras import backend as K
from keras.utils.generic_utils import get_custom_objects

def relus(Z):
    return K.maximum(0, Z)

get_custom_objects().update({'ReLU_s': Activation(relus)})

After that, apply the 'ReLU_s' as activation to your CNN layers, create a call back and train the model:

modelcheckpoint = ModelCheckpoint(filepath=output_dir+'/weights.{epoch:02d}.hdf5')

model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_data=(X_valid, y_valid), callbacks=[modelcheckpoint])

We need a name attribute in the Activation class.

Thanks in advance.

[wilderrodrigues]

I got it working by extending the Activation class:

class ReLUs(Activation):
    
    def __init__(self, activation, **kwargs):
        super(ReLUs, self).__init__(activation, **kwargs)
        self.__name__ = 'ReLU_s'

Up to you to accept as an issue and implement it, so nobody would have to extend it, or we can just close the issue.

Cheers,
Wilder

[brianleegit]

@wilderrodrigues Hi, I have the same issue with you. Can you show how you implemented the custom activation in class? thank you.

[wilderrodrigues]

Hi @brianleegit,

Yes, sure. See the code below:

from keras import backend as K
from keras.utils.generic_utils import get_custom_objects

class ReLUs(Activation):
    
    def __init__(self, activation, **kwargs):
        super(ReLUs, self).__init__(activation, **kwargs)
        self.__name__ = 'ReLU_s'

def relus(Z):
    # Your activation function specialties here
    A = K.maximum(m, Z)
    return A

get_custom_objects().update({'ReLU_s': ReLUs(relus)})

I used this inside a Jupyter Notebook cell. If you are going to do a plain Python implementation, I would suggest to have a 'config' function called from within the constructor. In that function, you set up the 'get_custom_objects'.

I hope this helps.

Cheers,
Wilder

[RaviRaaja]

____ interesting ! @wilderrodrigues thank you it is working!

[tobigithub]

just to add another example for the swish, this can be called with:

model(optimizer='Nadam', kernal_init='glorot_uniform', activation='swish' ....

from keras import backend as K
from keras.utils.generic_utils import get_custom_objects

# ------------------------------------------------------------
# needs to be defined as activation class otherwise error
# AttributeError: 'Activation' object has no attribute '__name__'    
class Swish(Activation):
    
    def __init__(self, activation, **kwargs):
        super(Swish, self).__init__(activation, **kwargs)
        self.__name__ = 'swish'

def swish(x):
    return (K.sigmoid(x) * x)

get_custom_objects().update({'swish': Swish(swish)})
# ------------------------------------------------------------

[rajneo]

Is this compatible while using tensor flow as well ??

[gokceneraslan]

Since functions in Python have __name__ attribute by default, get_custom_objects().update({'relus': relus}) would work, where relus is defined as:

def relus(Z):
    return K.maximum(0, Z)

[mariomeissner]

This also happens when you try to save the model with a custom activation.
It should definitely be implemented, as this is not expected behavior.
Probably just a name parameter for activation objects?

[mariomeissner]

In fact, I'm not even implementing my own activation, I am using:
keras.activations.LeakyReLU(alpha=0.3)

[linchundan88]

This also happens when you try to save the model with a custom activation.
I agree with you.

[wangyexiang]

In fact, I'm not even implementing my own activation, I am using:
keras.activations.LeakyReLU(alpha=0.3)

Same issue. How to fix it? I just use it like this:

leaky_relu = keras.activations.LeakyReLU(alpha=0.3)
model.add(Dense(512, activation = leaky_relu ))

@tobigithub @mariomeissner