regarding using a dataset of different domain

[surfreta]

Hi,

Thanks for sharing the heatmap code.

If the studied data set is totally of different domain with the one that is used to train the VGG model, can I still use pipeline of your code? Do I have to re-train the model? If that's the case, what's the correct procedure?

Thanks a lot for your suggestions.

Reta

[gabrieldemarmiesse]

Hi,

It depends on how different your data is. Models like alexnet, VGG, ResNet, InceptionV3 or XCeption were all trained on the imagenet dataset (you can see the pictures at this adress: http://www.image-net.org/ ).

If your dataset is radically different from Imagenet, then, the best option would be to finetune the VGG. Only re-training the last layers so that you still keep a lot of what is learned from the imagenet pictures.

It's hard to tell you if you have to retrain a model without seeing your data or at least have a description of it. I think VGG will work fine in a lot of cases (if you have natural images taken with a camera in daylight, let's say). For example, you'd have to retrain the last layers of the VGG if your images were very blurry or all taken at night, I suppose. It depends on the dataset and the objective. If you can tell me a bit more about it, I can surely help you more with what to do next.

Gabriel

[surfreta]

Hi,

I am processing some images, which can look like these found from here .

In the context of this kind of image, do you think I still can use weights of pre-existing trained VGG model. Besides, would you like to share some insight on how to just re-train the last layer. Thanks.

[gabrieldemarmiesse]

Hi,

Since your images don't look at all like those, sure, the way to go is to retrain the VGG. Do you want to train it in a fully convolutional manner or rather in a classification manner? It depends again on the data that you have. For each input image in your training set, do you have only a label avaible or a complete heatmap for each image? (if you have a complete heatmap it would be the best option because your vgg would train way faster.)

[surfreta]

Hi, thanks for your reply. The mask image only two type of values, 0 or 1. In other words, I have a image matrix, where each pixel is 0 or 1. Is this the heatmap you were referring to? In the training process, I feed these mask images into the network as well.

[gabrieldemarmiesse]

Yes that was the heatmap I was refering to. My choice of words were poor since it actually is a "mask".

I think that in this case, you don't really need my code to make it work. The keras library should suffice. I'll suppose here that you want to cut the VGG before the flatten layer since your images don't look a lot like ImageNet. By calling the VGG and specifying include_top = False, in Keras, you'll be able to train it without problem. Here is a sample code that you can try and plug into your project.

If anything is unclear, (or buggy, but I ran this script on my computer so it should be fine), don't hesitate to ask questions.

from keras.applications.vgg19 import VGG19
from keras.preprocessing import image
from keras.applications.vgg19 import preprocess_input
from keras.models import Model
from keras.layers import *
from keras.applications.imagenet_utils import preprocess_input
import numpy as np

base_model = VGG19(weights='imagenet', include_top=False)
model = Model(input=base_model.input, output=base_model.layers[11].output) # cut at layer 11.

# Here you specify that you want to freeze the first two blocks of the VGG19 (7 layers).
# Their weights won't be updated during training.
for i in range(7):
    model.layers[i].trainable = False

# You add another layer to force the output to have only one channel  
# Since you only have zeros and ones as output.
# Then you add a sigmoid, which forces the activation of your output to be between zero and one.

input_frame = Input(shape=(3, None, None)) #
x = model(input_frame)
x = Convolution2D(1, 3, 3, activation="sigmoid")(x)

final_model = Model(input=input_frame, output=x)

print(final_model.summary()) # To check if it's what you want.

# We generate some random data. I assume that your images are of size 256x256.
# The output will be then of size 64x64 since you go through the first two 
# max-pooling layers of the VGG19 (downsample by a factor of 4).

X = np.random.uniform(0,256, (2000, 3, 256, 256))
X = preprocess_input(X) # VGG preprocessing
Y = np.around(np.random.uniform(0,1, (2000, 1, 64, 64)))

final_model.compile("sgd", "binary_crossentropy")
final_model.fit(X, Y)```

[surfreta]

Really appreciated! Thank you!