LeakyReLU uses up too much memory.

[chasep255]

Right now I am implementing leaky relus like this tf.maximum(0.1 * x, x). This works fine except when it comes to memory usage. Networks which will fit on my GPU when using tf.nn.relu or tf.nn.elu fail when I am using my leaky relu implementation. I think this is because it needs to store both the intermediate 0.1 * x and x values of the activations to compute the gradients which essentially does the memory usage. However, I do not think this would be an issue if there were a dedicated tf.nn.leaky_relu. Can someone consider adding this to a future tensorflow release.

Try this:

def lrelu(x, leak=0.2, name="lrelu"):
     with tf.variable_scope(name):
         f1 = 0.5 * (1 + leak)
         f2 = 0.5 * (1 - leak)
         return f1 * x + f2 * abs(x)

[poxvoculi]

In the future we plan to generate faster, more memory efficient code by automatically fusing operations in some subgraphs, rather than adding lots more specialized Ops.