Add multiscale support

[jacobgil]

Hi,
Fun project:)
Just a small suggestion -
I think that an issue is that the current model assumes that the objects (Tanks) are large enough.
If you"l want to use this for other types of weapons, like AK-47s, you will need the ability to support different scales, since ak-47s might occupy only a small part of the image.
One relatively easy way of doing that (without training an actual object localization network) is converting the last fully connected layers to convolutional layers (each filter size should be set the size of the output of previous layer).
i.e instead of using Flatten() and Dense(), use Convolution2D with appropriate filter sizes, when not in training.
Similarly to the heatmap option here: https://github.com/heuritech/convnets-keras.
This applies the classifier in a sliding window fashion, and you can then feed the classifier larger images and process the output.
This can be done without retraining, using the existing model weights.

[thiippal]

Hey Jacob, thanks for your input!

My training data contains tanks in various sizes, i.e. shot from medium to long-distance as well, but you're absolutely right: opting for convolutional instead of fully-connected layers would probably make the model more robust to changes in target size.

I'll give this a try really soon!

[thiippal]

Hi @jacobgil,

I've been exploring this issue, but I have trouble wrapping my head around the idea of converting the dense layer to a convolutional one using previously trained weights, despite spending quite a lot of time looking at the repo you referred to: how can I reconstruct (or reshape) the weights once they have been flattened?

[thiippal]

I've been experimenting with converting the final layers of the model from dense to convolutional, adding another network architecture to nets.py under MiniVGGNetFC.

The initial work can be found in fcn_test.py.

For reshaping the pre-trained weights, I've adopted the solution proposed for producing heatmaps in the repo referenced above:

for layer in model.layers:
    if layer.name.startswith("conv"):  
    orig = model.get_layer(layer.name)  
    layer.set_weights(orig.get_weights())  
if layer.name.startswith("dense"):  
    orig = model.get_layer(layer.name)
    W, b = orig.get_weights()
    n_filter, previous_filter, ax1, ax2 = layer.get_weights()[0].shape
    new_W = W.reshape((previous_filter, ax1, ax2, n_filter))
    new_W = new_W.transpose((3, 0, 1, 2))
    new_W = new_W[:, :, ::-1, ::-1]
    layer.set_weights([new_W, b])

When attempting to load the weights using the Keras `model.load_weights()' function, the weights for the first dense layer (dense_1) are loaded successfully, but the second set of weights for the layer dense_2 returns the following error:

ValueError: Cannot feed value of shape (512, 3) for Tensor u'Placeholder_10:0', which has shape '(1, 1, 512, 3)'

Does this mean the pre-trained weights for the second layer have to be reshaped and set manually? If so, how to do this?