Fine-tuning pre-trained VGG16 not possible since `add` method is not defined for `Model` class? #4040

Closed
jrosebr1 opened this Issue Oct 12, 2016 · 9 comments

Comments

Projects
None yet
10 participants
@jrosebr1

I am trying to fine-tune the pre-trained VGG16 network from keras.applications.VGG16. I'm doing the standard approach that @fchollet detailed in his blog post.

My code is as follows:

# load the VGG16 network, ensuring the head FC layer sets are
# left off
print("[INFO] loading VGG16...")
model = VGG16(weights="imagenet", include_top=False)

# loop over the layers in VGG (until the final CONV block) and
# freeze the layers -- we will only be fine-tuning the final CONV
# block along with our dense FC layers
for layer in model.layers[:15]:
    layer.trainable = False

# load the FCHeadNet and add it to the convolutional base
print("[INFO] loading head...")
head = FCHeadNet.build((512 * 7 * 7,), 17, dropout=True)
head.load_weights(args["head"])
model.add(head)

The FCHeadNet class simply defines a Sequential model. However, when I try to add head to the model I receive the following error message:

File "finetune.py", line 30, in 
    model.add(head)
AttributeError: 'Model' object has no attribute 'add'

Inspecting the vgg16.py source I see that VGG16 is defined as a Model versus a Sequential, thus there is no .add method. My question is therefore:

How do I fine-tune the pre-trained VGG16 class? Or is this simply not possible and I need to define VGG16 by hand and load the weights manually?

@jrosebr1 jrosebr1 changed the title from Fine-tuning pre-trained `applications.VGG16` not possible since `add` method is not defined for `Model` class? to Fine-tuning pre-trained VGG16 not possible since `add` method is not defined for `Model` class? Oct 12, 2016

@fchollet

This comment has been minimized.

Show comment
Hide comment
@fchollet

fchollet Oct 12, 2016

Collaborator

You should recover the output you want to build on top of, and use it to instantiate a new model.

If you want to use an intermediate layer, you can use model.get_layer(name). If you want to use the last layer, you can just use model.output.

initial_model = VGG16(weights="imagenet", include_top=False)
last = model.output

x = Flatten()(last)
x = Dense(1024, activation='relu')(x)
preds = Dense(200, activation='softmax')(x)

model = Model(initial_model.input, preds)

This is detailed in the docs, too: https://keras.io/applications/#fine-tune-inceptionv3-on-a-new-set-of-classes

Collaborator

fchollet commented Oct 12, 2016

You should recover the output you want to build on top of, and use it to instantiate a new model.

If you want to use an intermediate layer, you can use model.get_layer(name). If you want to use the last layer, you can just use model.output.

initial_model = VGG16(weights="imagenet", include_top=False)
last = model.output

x = Flatten()(last)
x = Dense(1024, activation='relu')(x)
preds = Dense(200, activation='softmax')(x)

model = Model(initial_model.input, preds)

This is detailed in the docs, too: https://keras.io/applications/#fine-tune-inceptionv3-on-a-new-set-of-classes

@fchollet fchollet closed this Oct 12, 2016

@parikshit95

This comment has been minimized.

Show comment
Hide comment
@parikshit95

parikshit95 Feb 25, 2017

I am trying to get accuracy from my predicted results from the VGG16 model. However, the decode_predictions() function only returns a tuple containing ID and Label and not accuracy. Is there any way for decode_predictions() to return accuracy as well?
model = VGG16(weights="imagenet", include_top=True)
sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9,nesterov=True)
model.compile(optimizer=sgd,loss='categorical_crossentropy',metrics=['accuracy'])
print("network loaded")

#Get Lable for the images in directory
preds = []
i=0
for image in data:
preds.append(model.predict(image))
i = i+1
print(i)
#print("[INFO] loss={:.4f}, accuracy: {:.4f}%.format(loss,accuracy*100)")
#Decode the predictions
decode = []
for preds in preds:
decode.append(decode_predictions(preds, top=3)[0])
print(decode)

I am trying to get accuracy from my predicted results from the VGG16 model. However, the decode_predictions() function only returns a tuple containing ID and Label and not accuracy. Is there any way for decode_predictions() to return accuracy as well?
model = VGG16(weights="imagenet", include_top=True)
sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9,nesterov=True)
model.compile(optimizer=sgd,loss='categorical_crossentropy',metrics=['accuracy'])
print("network loaded")

#Get Lable for the images in directory
preds = []
i=0
for image in data:
preds.append(model.predict(image))
i = i+1
print(i)
#print("[INFO] loss={:.4f}, accuracy: {:.4f}%.format(loss,accuracy*100)")
#Decode the predictions
decode = []
for preds in preds:
decode.append(decode_predictions(preds, top=3)[0])
print(decode)

@ptisseur

This comment has been minimized.

Show comment
Hide comment
@ptisseur

ptisseur Apr 3, 2017

With Keras 2.02 this does not work
base_model = applications.VGG16(include_top=False, weights='imagenet')
last = base_model.output
x = Flatten()(last)
model = Model(input=base_model.input, output=x)

Trying with
base_model = applications.VGG16(include_top=False, weights='imagenet')
last = base_model.output
x=Flatten(input_shape=base_model.output_shape[1:])(x)
model = Model(input=base_model.input, output=x)
This does not work

With AveragePooling2D()(x) there is no more errors
#x=Flatten(input_shape=base_model.output_shape[1:])(x)
#x=Dense(64, activation='relu')(x)
#model.add(Dropout(0.5)
#x=Dense(8, activation='softmax')(x)

this is the model we will train

#model = Model(input=base_model.input, output=x)

Anyone can help me?

ptisseur commented Apr 3, 2017

With Keras 2.02 this does not work
base_model = applications.VGG16(include_top=False, weights='imagenet')
last = base_model.output
x = Flatten()(last)
model = Model(input=base_model.input, output=x)

Trying with
base_model = applications.VGG16(include_top=False, weights='imagenet')
last = base_model.output
x=Flatten(input_shape=base_model.output_shape[1:])(x)
model = Model(input=base_model.input, output=x)
This does not work

With AveragePooling2D()(x) there is no more errors
#x=Flatten(input_shape=base_model.output_shape[1:])(x)
#x=Dense(64, activation='relu')(x)
#model.add(Dropout(0.5)
#x=Dense(8, activation='softmax')(x)

this is the model we will train

#model = Model(input=base_model.input, output=x)

Anyone can help me?

@riccardosamperna

This comment has been minimized.

Show comment
Hide comment
@riccardosamperna

riccardosamperna Apr 21, 2017

Same problem here, it is not able to recover the size of the base_model.

Same problem here, it is not able to recover the size of the base_model.

@SinghGauravKumar

This comment has been minimized.

Show comment
Hide comment

@ptisseur @riccardosamperna Did you guys figure it out?

@tianchenwang

This comment has been minimized.

Show comment
Hide comment
@tianchenwang

tianchenwang Jul 19, 2017

@ptisseur My code works. Add a prediction layer before "model=Model(...)". If still not working, try to rename the prediction layer.

@ptisseur My code works. Add a prediction layer before "model=Model(...)". If still not working, try to rename the prediction layer.

@edoven

This comment has been minimized.

Show comment
Hide comment
@edoven

edoven Aug 2, 2017

I solved by creating "new_model = Sequential()". I then copied all the layers from "applications.VGG16(..)" into this new model.
This way I get a normal Sequential model and I can do the concatenation "new_model.add(top_model)".
Hope this helps.

from keras import applications, optimizers
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense, Input
import keras

print(keras.__version__)
# OUTPUT: '2.0.3'

keras.backend.set_image_dim_ordering('tf')


# SET ALL THE PARAMETERS
weights_path = 'models/vgg16.h5'
top_model_weights_path = 'bottleneck_fc_model.h5'
img_width, img_height = 150, 150
train_data_dir = 'data/dogscats1000/train'
validation_data_dir = 'data/dogscats1000/validation'
nb_train_samples = 2000
nb_validation_samples = 2000
epochs = 30
batch_size = 16



# LOAD VGG16
input_tensor = Input(shape=(150,150,3))
model = applications.VGG16(weights='imagenet', 
                           include_top=False,
                           input_tensor=input_tensor)


# CREATE A TOP MODEL
top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))
top_model.load_weights(top_model_weights_path)


# CREATE AN "REAL" MODEL FROM VGG16
# BY COPYING ALL THE LAYERS OF VGG16
new_model = Sequential()
for l in model.layers:
    new_model.add(l)


# CONCATENATE THE TWO MODELS
new_model.add(top_model)

# LOCK THE TOP CONV LAYERS
for layer in new_model.layers:
    layer.trainable = False

# COMPILE THE MODEL
new_model.compile(loss='binary_crossentropy',
              optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])



# CREATE THE IMAGE GENERATORS
train_datagen = ImageDataGenerator(rescale=1./255,
                                   shear_range=0.2,
                                   zoom_range=0.2,
                                   horizontal_flip=True)
validation_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
                        train_data_dir,
                        target_size=(img_height,img_width),
                        batch_size=batch_size,
                        class_mode='binary')

validation_generator = validation_datagen.flow_from_directory(
                            validation_data_dir,
                            target_size=(img_height,img_width),
                            batch_size=batch_size,
                            class_mode='binary')


#  FIT THE MODEL

new_model.fit_generator(
    train_generator,
    samples_per_epoch=nb_train_samples,
    epochs=epochs,
    validation_data=validation_generator,
    nb_val_samples=nb_validation_samples)

edoven commented Aug 2, 2017

I solved by creating "new_model = Sequential()". I then copied all the layers from "applications.VGG16(..)" into this new model.
This way I get a normal Sequential model and I can do the concatenation "new_model.add(top_model)".
Hope this helps.

from keras import applications, optimizers
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense, Input
import keras

print(keras.__version__)
# OUTPUT: '2.0.3'

keras.backend.set_image_dim_ordering('tf')


# SET ALL THE PARAMETERS
weights_path = 'models/vgg16.h5'
top_model_weights_path = 'bottleneck_fc_model.h5'
img_width, img_height = 150, 150
train_data_dir = 'data/dogscats1000/train'
validation_data_dir = 'data/dogscats1000/validation'
nb_train_samples = 2000
nb_validation_samples = 2000
epochs = 30
batch_size = 16



# LOAD VGG16
input_tensor = Input(shape=(150,150,3))
model = applications.VGG16(weights='imagenet', 
                           include_top=False,
                           input_tensor=input_tensor)


# CREATE A TOP MODEL
top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))
top_model.load_weights(top_model_weights_path)


# CREATE AN "REAL" MODEL FROM VGG16
# BY COPYING ALL THE LAYERS OF VGG16
new_model = Sequential()
for l in model.layers:
    new_model.add(l)


# CONCATENATE THE TWO MODELS
new_model.add(top_model)

# LOCK THE TOP CONV LAYERS
for layer in new_model.layers:
    layer.trainable = False

# COMPILE THE MODEL
new_model.compile(loss='binary_crossentropy',
              optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])



# CREATE THE IMAGE GENERATORS
train_datagen = ImageDataGenerator(rescale=1./255,
                                   shear_range=0.2,
                                   zoom_range=0.2,
                                   horizontal_flip=True)
validation_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
                        train_data_dir,
                        target_size=(img_height,img_width),
                        batch_size=batch_size,
                        class_mode='binary')

validation_generator = validation_datagen.flow_from_directory(
                            validation_data_dir,
                            target_size=(img_height,img_width),
                            batch_size=batch_size,
                            class_mode='binary')


#  FIT THE MODEL

new_model.fit_generator(
    train_generator,
    samples_per_epoch=nb_train_samples,
    epochs=epochs,
    validation_data=validation_generator,
    nb_val_samples=nb_validation_samples)
@Ksen17

This comment has been minimized.

Show comment
Hide comment
@Ksen17

Ksen17 Nov 20, 2017

@edoven
I tried to use Your code but accuracy falls down to 0.0195

Ksen17 commented Nov 20, 2017

@edoven
I tried to use Your code but accuracy falls down to 0.0195

@fedden

This comment has been minimized.

Show comment
Hide comment
@fedden

fedden Apr 11, 2018

EDIT: I realised actually looking at the VGG model, that the tutorial might be wrong? The tutorial suggests we block all layers up to 25, but that would freeze all the layers in VGG16!

If we print the summary of VGG16 then it looks like the first 15 layers the three convolutional blocks that we want to freeze and the last block that we want to unfreeze seems to begin from layer 15.


@edoven Thanks for the help on creating a new model out of the VGG model!

I believe your code a little bit incorrect with respect to the original tutorial, which instructs the first 25 (but I think it should be 15) layers should be frozen, but not all of them; the last layers comprise the last convolutional block and are to be finetuned with the top model.

Forgive (and correct) me if I'm wrong, but I think your code should be changed from this:

# LOCK THE TOP CONV LAYERS
for layer in new_model.layers:
    layer.trainable = False

to this:

# LOCK THE TOP CONV LAYERS
for layer in new_model.layers[:15]:
    layer.trainable = False

fedden commented Apr 11, 2018

EDIT: I realised actually looking at the VGG model, that the tutorial might be wrong? The tutorial suggests we block all layers up to 25, but that would freeze all the layers in VGG16!

If we print the summary of VGG16 then it looks like the first 15 layers the three convolutional blocks that we want to freeze and the last block that we want to unfreeze seems to begin from layer 15.


@edoven Thanks for the help on creating a new model out of the VGG model!

I believe your code a little bit incorrect with respect to the original tutorial, which instructs the first 25 (but I think it should be 15) layers should be frozen, but not all of them; the last layers comprise the last convolutional block and are to be finetuned with the top model.

Forgive (and correct) me if I'm wrong, but I think your code should be changed from this:

# LOCK THE TOP CONV LAYERS
for layer in new_model.layers:
    layer.trainable = False

to this:

# LOCK THE TOP CONV LAYERS
for layer in new_model.layers[:15]:
    layer.trainable = False
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment