Is it possible to evaluate the BaseModel with other keras metrics aside from binary and categorical accuracy?

[fbarusso]

I would like to evaluate a private model (to be specific the true/false negatives/positives metrics), but the evaluate function from the BaseModel seems to support only binary and categorical accuracy. I'm using a logistic regression model for binary classification.

[zicofish]

@fbarusso use predict?

[fbarusso]

@zicofish Thanks for your attention! I tried to use the benchmark/private_training.py by running:
tf-encrypted/examples/benchmark/training$ ./run-remote.sh logistic_regression LogisticArtificial

But on line 133, I used

result = model_2.predict(x=test_data_iter)

instead of

result = model_2.evaluate(x=test_data_iter, metrics=metrics, steps=test_dataset.iterations)

and it resulted in the following error:

Traceback (most recent call last):
  File "./private_training.py", line 133, in <module>
    result = model_2.predict(x=test_data_iter)
  File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/base_model.py", line 146, in predict
    y_pred = self.predict_function(input_x)
  File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/protocol/protocol.py", line 188, in wrap_function
    result = graph_function(args, kwargs)
  File "/home/febarusso/miniconda3/envs/tfe/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/tmp/__autograph_generated_filej69ox2qn.py", line 12, in tf__graph_function
    result = ag__.converted_call(ag__.ld(func), tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)
  File "/tmp/__autograph_generated_filezpzu8lwk.py", line 12, in tf__predict_step
    y_pred = ag__.converted_call(ag__.ld(self).call, (ag__.ld(input_x),), None, fscope)
  File "/tmp/__autograph_generated_filegmnf96cf.py", line 51, in tf__call
    ag__.for_stmt(ag__.ld(self).layers, None, loop_body, get_state_2, set_state_2, ('outputs', 'inputs'), {'iterate_names': 'layer'})
  File "/tmp/__autograph_generated_filegmnf96cf.py", line 48, in loop_body
    outputs = ag__.converted_call(ag__.ld(layer), (ag__.ld(inputs),), None, fscope)
  File "/tmp/__autograph_generated_file_c7f9xgv.py", line 28, in tf____call__
    outputs = ag__.converted_call(ag__.ld(self).call, ((ag__.ld(inputs),) + tuple(ag__.ld(args))), dict(**ag__.ld(kargs)), fscope)
  File "/tmp/__autograph_generated_file43d06n4z.py", line 11, in tf__call
    input_shape = ag__.converted_call(ag__.ld(inputs).shape.as_list, (), None, fscope)
AttributeError: in user code:

    File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/protocol/protocol.py", line 178, in graph_function  *
        result = func(*args, **kwargs)
    File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/base_model.py", line 129, in predict_step  *
        y_pred = self.call(input_x)
    File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/sequential.py", line 109, in call  *
        outputs = layer(inputs)
    File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/engine/base_layer.py", line 110, in __call__  *
        outputs = self.call(inputs, *args, **kargs)
    File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/layers/flatten.py", line 33, in call  *
        input_shape = inputs.shape.as_list()

    AttributeError: 'generator' object has no attribute 'shape'

[zicofish]

@zicofish Thanks for your attention! I tried to use the benchmark/private_training.py by running: tf-encrypted/examples/benchmark/training$ ./run-remote.sh logistic_regression LogisticArtificial

But on line 133, I used

result = model_2.predict(x=test_data_iter)

instead of

result = model_2.evaluate(x=test_data_iter, metrics=metrics, steps=test_dataset.iterations)

and it resulted in the following error:

Traceback (most recent call last):
  File "./private_training.py", line 133, in <module>
    result = model_2.predict(x=test_data_iter)
  File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/base_model.py", line 146, in predict
    y_pred = self.predict_function(input_x)
  File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/protocol/protocol.py", line 188, in wrap_function
    result = graph_function(args, kwargs)
  File "/home/febarusso/miniconda3/envs/tfe/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/tmp/__autograph_generated_filej69ox2qn.py", line 12, in tf__graph_function
    result = ag__.converted_call(ag__.ld(func), tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)
  File "/tmp/__autograph_generated_filezpzu8lwk.py", line 12, in tf__predict_step
    y_pred = ag__.converted_call(ag__.ld(self).call, (ag__.ld(input_x),), None, fscope)
  File "/tmp/__autograph_generated_filegmnf96cf.py", line 51, in tf__call
    ag__.for_stmt(ag__.ld(self).layers, None, loop_body, get_state_2, set_state_2, ('outputs', 'inputs'), {'iterate_names': 'layer'})
  File "/tmp/__autograph_generated_filegmnf96cf.py", line 48, in loop_body
    outputs = ag__.converted_call(ag__.ld(layer), (ag__.ld(inputs),), None, fscope)
  File "/tmp/__autograph_generated_file_c7f9xgv.py", line 28, in tf____call__
    outputs = ag__.converted_call(ag__.ld(self).call, ((ag__.ld(inputs),) + tuple(ag__.ld(args))), dict(**ag__.ld(kargs)), fscope)
  File "/tmp/__autograph_generated_file43d06n4z.py", line 11, in tf__call
    input_shape = ag__.converted_call(ag__.ld(inputs).shape.as_list, (), None, fscope)
AttributeError: in user code:

    File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/protocol/protocol.py", line 178, in graph_function  *
        result = func(*args, **kwargs)
    File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/base_model.py", line 129, in predict_step  *
        y_pred = self.call(input_x)
    File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/sequential.py", line 109, in call  *
        outputs = layer(inputs)
    File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/engine/base_layer.py", line 110, in __call__  *
        outputs = self.call(inputs, *args, **kargs)
    File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/layers/flatten.py", line 33, in call  *
        input_shape = inputs.shape.as_list()

    AttributeError: 'generator' object has no attribute 'shape'

@fbarusso you should not use the same test_data_iter for predict, because test_data_iter contains both x an y, which is good for evaluate, but not predict. You might need to figure out some way to extract only x. For example, in the predict function:

changing for input_x in data_iter: to for (input_x, input_y) in data_iter: will make it work. But there should be a better way to achieve what you want.

[zjn-code]

There is an example to use model.predict.

[fbarusso]

@zicofish @zjn-code it worked, thanks for the support.