Extend tf.keras.Model.evaluate with class_weight

[vicpara]

System information

• TensorFlow version (you are using):
  TF 2.0.0
• Are you willing to contribute it (Yes/No):
  Yes

Describe the feature and the current behaviour/state.
Currently tf.keras.models.Model.fit method allows the user to pass either 'sample_weight' and 'class_weight' parameters. These are used to compute at some point a standardised 'sample_weights' and used later on while calculating the loss.

This feature request is about extending the 'tf.keras.Model.evaluate' API so that is permits using class_weight directly. The evaluate function already permits for sample_weight.

Will this change the current api? How?
current API

evaluate(
    x=None,
    y=None,
    batch_size=None,
    verbose=1,
    sample_weight=None,
    steps=None,
    callbacks=None,
    max_queue_size=10,
    workers=1,
    use_multiprocessing=False
)

new API

evaluate(
    x=None,
    y=None,
    batch_size=None,
    verbose=1,
    sample_weight=None,
>>>    class_weight=None,
    steps=None,
    callbacks=None,
    max_queue_size=10,
    workers=1,
    use_multiprocessing=False
)

Who will benefit with this feature?

Those users of the API who would like to perform the evaluation of a model that was trained with bespoke class weights.

Any Other info.

[ayushmankumar7]

Since this issue is opened for a long time, let me work on this.

[mariemabid]

Is there any solution please to add class_weight to the evaluate() method?

[qlzh727]

Sorry for the long wait. I checked with Francois about the issue, and seems that it is expected to not having class_weights for model.evaluate.

Quote from Francois:

"The reason we don't support class weights in evaluate is that the class_weight argument represents sample weights that are computed from the labels, but the labels should not be an input to the model during evaluation. During training this is fine, but during evaluation this represents a data leak from the labels to your metrics. If you used class weighting in evaluate, your score would not be reproducible on real test data (when you don't have the labels).

So this is conceptually wrong."

I am closing this issue since it is working as intended.