error increases with Iteration increasing when using TrAdaBoostR2

[findyy99]

Thanks for this awsome libaray! I met this problem as I said in the title.
Here is the model information:

adam = keras.optimizers.legacy.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0, amsgrad=False)

def get_model():
    model = Sequential()
    model.add(keras.layers.Dense(64, activation='relu'))
    model.add(keras.layers.BatchNormalization()) 
    model.add(keras.layers.Dropout(0.2))  

    model.add(keras.layers.Dense(32, activation='relu'))
    model.add(keras.layers.BatchNormalization())  
    model.add(keras.layers.Dropout(0.1))  
    model.add(keras.layers.Dense(1))
    model.compile(optimizer=adam, loss='mean_squared_error')
    return model

from adapt.instance_based import TrAdaBoostR2
model2 = TrAdaBoostR2(get_model(), n_estimators=30, random_state=15, lr=0.25)
model2.fit(source_X, source_y, target_train_X, target_train_y, epochs=100, batch_size=16, verbose=0, validation_split=0.2);

And this is the result:

Iteration 0 - Error: 0.2534
Iteration 1 - Error: 0.2315
Iteration 2 - Error: 0.2340
Iteration 3 - Error: 0.3256
Iteration 4 - Error: 0.3048
Iteration 5 - Error: 0.3268
Iteration 6 - Error: 0.3902
Iteration 7 - Error: 0.4865
Iteration 8 - Error: 0.4385
Iteration 9 - Error: 0.4552
Iteration 10 - Error: 0.4693
Iteration 11 - Error: 0.5112
Iteration 12 - Error: 0.5955
Iteration 13 - Error: 0.5976
Iteration 14 - Error: 0.6013
Iteration 15 - Error: 0.6283
Iteration 16 - Error: 0.6649
Iteration 17 - Error: 0.6594
Iteration 18 - Error: 0.6908
Iteration 19 - Error: 0.7099
Iteration 20 - Error: 0.7315
Iteration 21 - Error: 0.7437
Iteration 22 - Error: 0.7821
Iteration 23 - Error: 0.7795
Iteration 24 - Error: 0.7922
Iteration 25 - Error: 0.7923
Iteration 26 - Error: 0.7965
Iteration 27 - Error: 0.8236
Iteration 28 - Error: 0.8147
Iteration 29 - Error: 0.8255

I have tried change the model structure, the lrearning rate and the value of estimators, it always shows the same result, the error raises as the iteration increasing.
This is the graph about weights.

• 
  n_estimators=30, lr=0.25
• 
  n_estimators=60, lr=1

[findyy99]

I found that when I add validation_split = 0.2 to the model.fit function, the error begins at a high value even with the example Toy-Regression. So, how to check if the model is overfitting if we use the class TrAdaBoostR2? Thanks!

[antoinedemathelin]

Hi @findyy99,
Thank you for your interest in the Adapt library!
I think the problem comes from the use of validation_split. I suspect Tensorflow to take the last data as validation (without shuffling), which, in the case of TradaBoostR2, are the target training data. I did some tests on the toy dataset and observe that using the validation_split parameter is the main issue here.

To solve your problem, you can either use validation_split=0. (no validation), as you don't use the validation data in your training (through early stopping callbacks for instance). Or you can pass some validation data with the argument validation_data = (x_val, y_val).

Last, the Error printed by TrAdaBoostR2 is the weighted average target error, it can increase at some point and then decrease. Be sure to use good optimization parameters: when trying your setup on the toy dataset, I observe that it is better to take lr=1 in TrAdaBoost and to use more epochs to fully converge at each iteration (epochs=300). But before, you need to remove the validation_split argument.

Best,

[findyy99]

@antoinedemathelin Thanks for your help!! I will try it as you advised.