Validation R2 drastically different in Keras 3 (eager mode) compared to Keras 2 (graph mode)

[DaniCosta92]

Hi Keras community,

I’m seeing a large discrepancy in validation metrics between Keras 2.13 with eager disabled and Keras 3 with eager enabled. I’m training a model with the Adam optimizer and a custom R2 metric:

def r2_keras(y_true, y_pred):
    ss_res = K.sum(K.square(y_true - y_pred))
    ss_tot = K.sum(K.square(y_true - K.mean(y_true)))
    return 1 - ss_res / (ss_tot + K.epsilon())

model.compile(loss='mean_squared_error', optimizer=adam, metrics=['mse', 'mae', r2_keras])

Using the same training and validation data (Xtrain, labels_train; Xtest, labels_test) and the call

history = model.fit(
    Xtrain,
    labels_train,
    validation_data=(Xtest, labels_test),
    epochs=n_epochs,
    batch_size=batch_size,
    verbose=0
)

In Keras 2.13 (graph mode) the validation R2 per epoch is around 0.6, similar to training, but in Keras 3 (eager mode) the validation R2 drops drastically to around -17,000 while training R2 remains the same.

Interestingly, if I set **validation_batch_size=len(Xtest)** in Keras 3, the validation R2 matches the old Keras 2 result.

This suggests that Keras 2 (graph mode) and Keras 3 (eager) handle validation metrics differently: Keras 2 may have effectively computed R2 over the entire validation set per epoch, while Keras 3 computes metrics per batch and averages them, which drastically affects non-linear metrics like R2.

I cannot find any web evidence that explains the difference I am seeing between Keras 2 (graph mode) and Keras 3 (eager mode). Any suggestions on why this behavior occurs would be very much appreciated.