Deconvolution2D stack with different nb_filter

[titu1994]

I'm attempting to implement Image Restoration Using Convolutional Auto-encoders with Symmetric Skip
Connections in Keras using the new Deconvolution2D layer.

I have a working version of this architecture using the UpSampling2D and Conv2D method to implement Deconvolutions, but due to max pooling and subsequent upsampling, I couldn't create a very deep model.

With the new Deconvolution2D, I can implement this, but it crashes giving an Input dimension mismatch when using Deconv stacks with different number of output filters. An example script is given below. (I know that the authors use skip connections every 2 corresponding layers and that I use skip connection after every layer, but I have tested it with every two layers and the same error occurs.)

Gist : Conv Auto Encoder with Symmetric Skip Connections

System config:
Windows 10
Theano 0.9.devr2
Keras (master branch)
CuDNN 5.1 RC

Crash report:
Using Theano backend.
Using gpu device 0: GeForce GTX 980M (CNMeM is enabled with initial size: 80.0% of memory, cuDNN 5103)
Epoch 1/10
Traceback (most recent call last):
File "D:\Users\Yue\Anaconda3\lib\site-packages\theano\compile\function_module.py", line 866, in call
self.fn() if output_subset is None else
ValueError: GpuElemwise. Input dimension mis-match. Input 3 (indices start at 0) has shape[0] == 64, but the output's size on that axis is 128.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:/Users/Yue/PycharmProjects/ImageSuperResolution/conv_auto_encoder.py", line 52, in
model.fit(dataX, dataY, batch_size=batch_size, nb_epoch=10)
File "D:\Users\Yue\Anaconda3\lib\site-packages\keras\engine\training.py", line 1107, in fit
callback_metrics=callback_metrics)
File "D:\Users\Yue\Anaconda3\lib\site-packages\keras\engine\training.py", line 825, in _fit_loop
outs = f(ins_batch)
File "D:\Users\Yue\Anaconda3\lib\site-packages\keras\backend\theano_backend.py", line 643, in call
return self.function(*inputs)
File "D:\Users\Yue\Anaconda3\lib\site-packages\theano\compile\function_module.py", line 879, in call
storage_map=getattr(self.fn, 'storage_map', None))
File "D:\Users\Yue\Anaconda3\lib\site-packages\theano\gof\link.py", line 325, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "D:\Users\Yue\Anaconda3\lib\site-packages\six.py", line 685, in reraise
raise value.with_traceback(tb)

File "D:\Users\Yue\Anaconda3\lib\site-packages\theano\compile\function_module.py", line 866, in call
self.fn() if output_subset is None else
ValueError: GpuElemwise. Input dimension mis-match. Input 3 (indices start at 0) has shape[0] == 64, but the output's size on that axis is 128.
Apply node that caused the error: GpuElemwise{Composite{((i0 * i1) + (i2 * i3))}}[(0, 1)](GpuDimShuffle{x,x,x,x}.0, <CudaNdarrayType%28float32, 4D%29>, GpuDimShuffle{x,x,x,x}.0, GpuDnnConvGradW{algo='none', inplace=True}.0)
Toposort index: 364
Inputs types: [CudaNdarrayType(float32, (True, True, True, True)), CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, (True, True, True, True)), CudaNdarrayType(float32, 4D)]
Inputs shapes: [(1, 1, 1, 1), (128, 64, 3, 3), (1, 1, 1, 1), (64, 64, 3, 3)]
Inputs strides: [(0, 0, 0, 0), (576, 9, 3, 1), (0, 0, 0, 0), (576, 9, 3, 1)]
Inputs values: [b'CudaNdarray([[[[ 0.89999998]]]])', 'not shown', b'CudaNdarray([[[[ 0.10000002]]]])', 'not shown']
Outputs clients: [['output', GpuElemwise{Composite{(i0 - ((i1 * i2) / (i3 + sqrt(clip(i4, i5, i6)))))}}[(0, 0)](convolution2d_3_W, GpuDimShuffle{x,x,x,x}.0, GpuElemwise{Composite{%28%28i0 * i1%29 + %28i2 * i3%29%29}}[%280, 1%29].0, CudaNdarrayConstant{[[[[ 9.99999994e-09]]]]}, GpuElemwise{Composite{%28%28i0 * i1%29 + %28i2 * sqr%28i3%29%29%29}}[%280, 1%29].0, CudaNdarrayConstant{[[[[ 0.]]]]}, CudaNdarrayConstant{[[[[ inf]]]]})]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

To remove this error

• If n1 = n2 = any number, then crash does not occur. This isn't very useful since accuracy depends on number of filters as well. The previous version used 64, 128 and 256 filters, achieving an accuracy of 37.39 PSNR (Peak Signal to Noise Ratio). This one doesn't go above 36.41 PSNR with n1 = n2 = 64.

[gravity1989]

Why is this closed? did you find the solution?

[titu1994]

The problem is not fixed.

However, the initial error that I showed is due to my error in determining the output shape of the Deconvolution layers.

I was planning on updating this one with the new error when my laptop finished training another architecture.

[titu1994]

New Script:

from keras.models import Model
from keras.layers import Input
from keras.layers.convolutional import Convolution2D
from keras.layers.convolutional import Deconvolution2D
from keras.layers import merge
import numpy as np

'''
Attempt at implementing a smaller version of "Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections"
(http://arxiv.org/abs/1606.08921)
'''
batch_size = 128
channels = 3
height = 33
width = 33

n1 = 64
n2 = 128 # Compiles and fits properly if n1 == n2

init = Input(shape=(channels, height, width))

level1_1 = Convolution2D(n1, 3, 3, activation='relu', border_mode='same')(init)
level2_1 = Convolution2D(n1, 3, 3, activation='relu', border_mode='same')(level1_1)

level3_1 = Convolution2D(n2, 3, 3, activation='relu', border_mode='same')(level2_1)
level4_1 = Convolution2D(n2, 3, 3, activation='relu', border_mode='same')(level3_1)

level4_2 = Deconvolution2D(n2, 3, 3, activation='relu', output_shape=(None, n2, height, width), border_mode='same')(level4_1)
level3_2 = Deconvolution2D(n2, 3, 3, activation='relu', output_shape=(None, n2, height, width), border_mode='same')(level4_2)
level3 = merge([level3_1, level3_2], mode='sum')

level2_2 = Deconvolution2D(n1, 3, 3, activation='relu', output_shape=(None, n1, height, width), border_mode='same')(level3)
level1_2 = Deconvolution2D(n1, 3, 3, activation='relu', output_shape=(None, n1, height, width), border_mode='same')(level2_2)
level1 = merge([level1_1, level1_2], mode='sum')

decoded = Convolution2D(channels, 5, 5, activation='linear', border_mode='same')(level1)

model = Model(init, decoded)
model.compile(optimizer='adam', loss='mse')

np.random.seed(1)
dataX = np.random.random((200, channels, height, width))
dataY = np.random.random((200, channels, height, width))

model.fit(dataX, dataY, batch_size=batch_size, nb_epoch=10) # Crashes here if n1 != n2

The new problem that occurs :

Error :

Using Theano backend.
Using gpu device 0: GeForce GTX 980M (CNMeM is enabled with initial size: 80.0% of memory, cuDNN 5103)
Epoch 1/10
Traceback (most recent call last):
File "D:\Users\Yue\Anaconda3\lib\site-packages\theano\compile\function_module.py", line 866, in call
self.fn() if output_subset is None else\
ValueError: GpuDnnConv images and kernel must have the same stack size

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:/Users/Yue/PycharmProjects/ImageSuperResolution/conv_auto_encoder.py", line 44, in

File "D:\Users\Yue\Anaconda3\lib\site-packages\keras\engine\training.py", line 1107, in fit
callback_metrics=callback_metrics)
File "D:\Users\Yue\Anaconda3\lib\site-packages\keras\engine\training.py", line 825, in fit_loop
outs = f(ins_batch)
File "D:\Users\Yue\Anaconda3\lib\site-packages\keras\backend\theano_backend.py", line 643, in __call_
return self.function(*inputs)
File "D:\Users\Yue\Anaconda3\lib\site-packages\theano\compile\function_module.py", line 879, in call
storage_map=getattr(self.fn, 'storage_map', None))
File "D:\Users\Yue\Anaconda3\lib\site-packages\theano\gof\link.py", line 325, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "D:\Users\Yue\Anaconda3\lib\site-packages\six.py", line 685, in reraise
raise value.with_traceback(tb)
File "D:\Users\Yue\Anaconda3\lib\site-packages\theano\compile\function_module.py", line 866, in call
self.fn() if output_subset is None else\
ValueError: GpuDnnConv images and kernel must have the same stack size

Apply node that caused the error: GpuDnnConvGradW{algo='none', inplace=True}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty.0, GpuDnnConvDesc{border_mode='half', subsample=(1, 1), conv_mode='conv', precision='float32'}.0, Constant{1.0}, Constant{0.0})
Toposort index: 321
Inputs types: [CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D), , Scalar(float32), Scalar(float32)]
Inputs shapes: [(128, 128, 33, 33), (128, 128, 33, 33), (128, 64, 3, 3), 'No shapes', (), ()]
Inputs strides: [(139392, 1089, 33, 1), (139392, 1089, 33, 1), (576, 9, 3, 1), 'No strides', (), ()]
Inputs values: ['not shown', 'not shown', 'not shown', , 1.0, 0.0]
Inputs name: ('image', 'grad', 'output', 'descriptor', 'alpha', 'beta')

Outputs clients: [[GpuDimShuffle{1,0,2,3}(GpuDnnConvGradW{algo='none', inplace=True}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

[titu1994]

The problem is not fixed.

However, the initial error that I showed is due to my error in determining
the output shape of the Deconvolution layers.

I was planning on updating this one with the new error when my laptop
finished training another architecture.

On Aug 3, 2016 19:25, "Rahul V" notifications@github.com wrote:

Why is this closed? did you find the solution?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#3371 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC6EmmoIcgwpiKQPaMV9sAzW6NZntH9Kks5qcJnSgaJpZM4JZtzM
.

[perone]

Did you found the issue ? I'm facing the same error.

[titu1994]

Didn't find any solution. For now I've switched to Deconvolution using
upsampling instead. That still works because it doesn't use the
Deconvolution layer.

I am working on Windows, so Tensorflow isn't available to me, however I
don't think it will change the error.

On Aug 15, 2016 00:31, "Christian S. Perone" notifications@github.com
wrote:

Did you found the issue ? I'm facing the same error.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#3371 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC6EmppsJyLQ9p9doK92SIRYQW7-OwLIks5qf2X2gaJpZM4JZtzM
.

[perone]

@titu1994 Thanks Somshubra, I'll try to switch to Tensorflow to see what happens.

[Neltherion]

Any update on this?
I've been having the same problem...

[titu1994]

@Neltherion There was a recent PR which bypasses the shape inference for Deconvolution in Theano/Tensorflow. That fixes the height and width problem.

There is no fix for different number of filters in a stack of deconvolutions yet. As a temporary fix, I simply made all deconv layers have the same number of filters. In a autoencoder, that means the earlier convolution layers also need to have the same number of filters as the proceeding deconv layers.

If the network does not learn properly it's best to double the number of filters everywhere (this is what I did, instead of 3-64-128 --- 128-64-3, I simply make them all 128)

Edit:
Also, I have not been able to make merge([conv, deconv], mode='sum') work, even if I provide correct output shape for the deconv with the latest commit. There will always be one mismatch (either in nb_rows or nb_cols dimension).

For example if I try to merge a conv layer of shape (128, 32, 32) with a deconv layer of output shape (128, 32, 32), an Input Mismatch exception in Theano will be raised - stating that Input 1 (index begins at 0) has output size = 31, even though output shape on that axis is 32. (I don't remember the exact words, but it's close to the above).

[Neltherion]

@titu1994 Thanks for the quick response.
I really wish this to get fixed soon as Deconvolutions are really needed for when your output is 2-Dimensional and the way they are right now (having to stack the same number of DeConv layers according to the previous layers) always leads me to believe my model is somehow learning inefficiently.

by the way is there any way other than using a final 1 filter Convolution layer to get a 2D output (like an image) after the DeConv layers? Originally I wanted to reduce the number of DeConv layers from 128 to 64 then 32 and finally to 1 but since we can't do that, are there any alternatives?

[titu1994]

@Neltherion Deconv layers stacks need to be handled properly by Keras and by extension Theano and Tensorflow. You are correct in assuming that stacks of multiple deconv layers with same number of filters actually learns less than UpSampling + Convolution, at least in some circumstances (with skip connections).

To avoid this, you can create a type of auto encoder using conv subsampling (with same number of filters) to reduce the image size say 2 or 4 times, then use deconv (with same number of filters) and subsample it to increase the size say by 2 or 4 times. However, as I mentioned, since merge([conv, deconv]) dont work yet, this model cannot use large skip connections to learn efficiently.

I have a repo with upsampling + conv + skip connections, and it learns how to upscale images very rapidly (getting 37.45 Peak Signal to Noise Ratio in less than 100 epochs). However, until merge ([conv, deconv]) is fixed, the deconv version lags behind at just above 33.6 PSNR.

As to your last point about getting a 2D output after deconv layers, I don't think there is any other way. In all the papers I have seen, after using deconv for various purposes, they stick a final conv layer to get the 3 filters of RGB. If you use grayscale images, then using 1 filter conv is required.

[Neltherion]

@titu1994 Thanks again for the detailed solution. This has been bothering me for a while and now thanks to you I know it's Theano/Keras's fault!

[hadikazemi]

I have the same issue. Hopefully, it gets fixed soon. Switched to Conv + Upsampling

[generallc]

any one resolved this problem?