Bug when predict with a model that have multiples outputs

[Racines]

When I create a model with 2 outputs:

var model = new Model(new BaseLayer[] { mainInput }, new BaseLayer[] { ph, vh });

and I want to use predict:

var outputs = model.Predict(input);
var tmp = outputs[0];

I have and exception when trying to access the first element in outputs:
Python.Runtime.PythonException : 'TypeError : list indices must be integers or slices, not tuple'

The outputs.str value is:

[array([[0.04858182, 0.02549346, 0.04383887, 0.07485574, 0.08390558,
        0.01660796, 0.02462194, 0.02408224, 0.08508438, 0.0349065 ,
        0.03103675, 0.10075846, 0.01927384, 0.02810584, 0.06927238,
        0.01279816, 0.02566949, 0.1094867 , 0.0269268 , 0.02743391,
        0.04374657, 0.04351271]], dtype=float32), array([[0.49687997]], dtype=float32)]

I suppose there is a bug because of the "array" keyword in that string?
Or maybe I so something wrong here?

PS: When I create a model with only one output, I have no problem to access the data with predict

[Racines]

After a a while I finally understood that Keras.Net miss a wrapper for multiple outputs prediction.
It's also miss a model.fit() wrapper that take multiple outputs for y parameter.
Also I had the same problem to define multiple loss fonctions in the model.compile() method like explained in this issue: #164

So I added the missing wrappers by my own, here is the code:

public static class KerasBaseModelExtensions
{
	/// <summary>
	/// source code from: https://githubmemory.com/repo/SciSharp/Keras.NET/issues/164
	/// </summary>
	/// <param name="model"></param>
	/// <param name="optimizer"></param>
	/// <param name="loss"></param>
	/// <param name="metrics"></param>
	/// <param name="loss_weights"></param>
	/// <param name="sample_weight_mode"></param>
	/// <param name="weighted_metrics"></param>
	/// <param name="target_tensors"></param>
	public static void Compile(
		this BaseModel model,
		StringOrInstance optimizer,
		string[] loss,
		string[] metrics = null,
		float[] loss_weights = null,
		string sample_weight_mode = null,
		string[] weighted_metrics = null,
		NDarray[] target_tensors = null)
	{
		var args = new Dictionary<string, object>();
		args["optimizer"] = optimizer;
		args["loss"] = loss;
		args["metrics"] = metrics;
		args["loss_weights"] = loss_weights;
		args["sample_weight_mode"] = sample_weight_mode;
		args["weighted_metrics"] = weighted_metrics;
		args["target_tensors"] = target_tensors;

		model.InvokeMethod("compile", args);
	}

	/// <summary>
	/// Trains the model for a given number of epochs (iterations on a dataset).
	/// </summary>
	/// <param name="x">Numpy array of training data (if the model has a single input), or list of Numpy arrays (if the model has multiple inputs). If input layers in the model are named, you can also pass a dictionary mapping input names to Numpy arrays. x can be None (default) if feeding from framework-native tensors (e.g. TensorFlow data tensors).</param>
	/// <param name="y">Numpy array of target (label) data (if the model has a single output), or list of Numpy arrays (if the model has multiple outputs). If output layers in the model are named, you can also pass a dictionary mapping output names to Numpy arrays. y can be None (default) if feeding from framework-native tensors (e.g. TensorFlow data tensors).</param>
	/// <param name="batch_size">Integer or None. Number of samples per gradient update. If unspecified, batch_sizewill default to 32.</param>
	/// <param name="epochs">Integer. Number of epochs to train the model. An epoch is an iteration over the entire x and y data provided. Note that in conjunction with initial_epoch, epochs is to be understood as "final epoch". The model is not trained for a number of iterations given by epochs, but merely until the epoch of index epochs is reached.</param>
	/// <param name="verbose">Integer. 0, 1, or 2. Verbosity mode. 0 = silent, 1 = progress bar, 2 = one line per epoch.</param>
	/// <param name="callbacks">List of keras.callbacks.Callback instances. List of callbacks to apply during training and validation (if ). See callbacks.</param>
	/// <param name="validation_split">Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch. The validation data is selected from the last samples in the x and y data provided, before shuffling.</param>
	/// <param name="validation_data">tuple (x_val, y_val) or tuple (x_val, y_val, val_sample_weights) on which to evaluate the loss and any model metrics at the end of each epoch. The model will not be trained on this data. validation_data will override validation_split.</param>
	/// <param name="shuffle">Boolean (whether to shuffle the training data before each epoch) or str (for 'batch'). 'batch' is a special option for dealing with the limitations of HDF5 data; it shuffles in batch-sized chunks. Has no effect when steps_per_epoch is not None.</param>
	/// <param name="class_weight">Optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only). This can be useful to tell the model to "pay more attention" to samples from an under-represented class.</param>
	/// <param name="sample_weight">Optional Numpy array of weights for the training samples, used for weighting the loss function (during training only). You can either pass a flat (1D) Numpy array with the same length as the input samples (1:1 mapping between weights and samples), or in the case of temporal data, you can pass a 2D array with shape (samples, sequence_length), to apply a different weight to every timestep of every sample. In this case you should make sure to specifysample_weight_mode="temporal" in compile().</param>
	/// <param name="initial_epoch">Integer. Epoch at which to start training (useful for resuming a previous training run).</param>
	/// <param name="steps_per_epoch">Integer or None. Total number of steps (batches of samples) before declaring one epoch finished and starting the next epoch. When training with input tensors such as TensorFlow data tensors, the default None is equal to the number of samples in your dataset divided by the batch size, or 1 if that cannot be determined.</param>
	/// <param name="validation_steps">Only relevant if steps_per_epoch is specified. Total number of steps (batches of samples) to validate before stopping.</param>
	/// <returns>A History object. Its History.history attribute is a record of training loss values and metrics values at successive epochs, as well as validation loss values and validation metrics values (if applicable).</returns>
	public static History FitWithMultipleOutputs
		(this BaseModel model, NDarray x, NDarray[] y, int? batch_size = null, int epochs = 1, int verbose = 1, 
					Callback[] callbacks = null,
					float validation_split = 0.0f, NDarray[] validation_data = null, 
					bool shuffle = true, Dictionary<int, float> class_weight = null,
					NDarray sample_weight = null, int initial_epoch = 0, int? steps_per_epoch = null, 
					int? validation_steps = null)
	{
		var args = new Dictionary<string, object>();
		args["x"] = x;
		args["y"] = y;
		args["batch_size"] = batch_size;
		args["epochs"] = epochs;
		args["verbose"] = verbose;
		args["callbacks"] = callbacks;
		args["validation_split"] = validation_split;
		if (validation_data != null)
		{
			if (validation_data.Length == 2)
				args["validation_data"] = new PyTuple(new PyObject[] { validation_data[0].PyObject, validation_data[1].PyObject });
			else if (validation_data.Length == 3)
				args["validation_data"] = new PyTuple(new PyObject[] { validation_data[0].PyObject, validation_data[1].PyObject, validation_data[2].PyObject });
		}

		args["shuffle"] = shuffle;
		if (class_weight != null)
			args["class_weight"] = ToDict(class_weight);
		args["sample_weight"] = sample_weight;
		args["initial_epoch"] = initial_epoch;
		args["steps_per_epoch"] = steps_per_epoch;
		args["validation_steps"] = validation_steps;

		PyObject py = model.InvokeMethod("fit", args);

		return new History(py);
	}

	static PyDict ToDict(Dictionary<int, float> input)
	{
		PyDict dict = new PyDict();

		foreach (var item in input)
		{
			dict[item.Key.ToPython()] = item.Value.ToPython();
		}

		return dict;
	}

	/// <summary>
	/// Generates output predictions for the input samples.
	/// Computation is done in batches.
	/// </summary>
	/// <param name="x">The input data, as a Numpy array (or list of Numpy arrays if the model has multiple inputs).</param>
	/// <param name="batch_size">Integer. If unspecified, it will default to 32.</param>
	/// <param name="verbose">Verbosity mode, 0 or 1.</param>
	/// <param name="steps">Total number of steps (batches of samples) before declaring the prediction round finished. Ignored with the default value of None.</param>
	/// <param name="callbacks">List of keras.callbacks.Callback instances. List of callbacks to apply during prediction. See callbacks.</param>
	/// <returns>Numpy array(s) of predictions.</returns>
	public static NDarray[] PredictMultipleOutputs(this BaseModel model, NDarray x, int? batch_size = null, 
		int verbose = 1, int? steps = null, Callback[] callbacks = null)
	{
		var args = new Dictionary<string, object>();
		args["x"] = x;
		args["batch_size"] = batch_size;
		args["verbose"] = verbose;
		args["steps"] = steps;
		args["callbacks"] = callbacks != null ? callbacks : null;

		var res = model.InvokeMethod("predict", args);
		var resTuple = PyTuple.AsTuple(res);

		var length = resTuple.Length();
		var finalRes = new NDarray[length];
		for (int i = 0; i < length; i++)
		{
			finalRes[i] = new NDarray(resTuple[i]);
		}
		return finalRes;
	}
}

[deepakkumar1984]

Thanks mate. Can you please raise a PR with the fix, I will merge the code?

…

On Sun, Sep 19, 2021 at 6:03 AM Racines ***@***.***> wrote: After a a while I finally understood that Keras.Net miss a wrapper for multiple outputs prediction. It's also miss a model.fit() wrapper that take multiple outputs for y parameter. Also I had the same problem to define multiple loss fonctions in the model.compile() method like explained in this issue: #164 <#164> So I added the missing wrappers by my own, here is the code: public static class KerasBaseModelExtensions { /// <summary> /// source code from: https://githubmemory.com/repo/SciSharp/Keras.NET/issues/164 /// </summary> /// <param name="model"></param> /// <param name="optimizer"></param> /// <param name="loss"></param> /// <param name="metrics"></param> /// <param name="loss_weights"></param> /// <param name="sample_weight_mode"></param> /// <param name="weighted_metrics"></param> /// <param name="target_tensors"></param> public static void Compile( this BaseModel model, StringOrInstance optimizer, string[] loss, string[] metrics = null, float[] loss_weights = null, string sample_weight_mode = null, string[] weighted_metrics = null, NDarray[] target_tensors = null) { var args = new Dictionary<string, object>(); args["optimizer"] = optimizer; args["loss"] = loss; args["metrics"] = metrics; args["loss_weights"] = loss_weights; args["sample_weight_mode"] = sample_weight_mode; args["weighted_metrics"] = weighted_metrics; args["target_tensors"] = target_tensors; model.InvokeMethod("compile", args); } /// <summary> /// Trains the model for a given number of epochs (iterations on a dataset). /// </summary> /// <param name="x">Numpy array of training data (if the model has a single input), or list of Numpy arrays (if the model has multiple inputs). If input layers in the model are named, you can also pass a dictionary mapping input names to Numpy arrays. x can be None (default) if feeding from framework-native tensors (e.g. TensorFlow data tensors).</param> /// <param name="y">Numpy array of target (label) data (if the model has a single output), or list of Numpy arrays (if the model has multiple outputs). If output layers in the model are named, you can also pass a dictionary mapping output names to Numpy arrays. y can be None (default) if feeding from framework-native tensors (e.g. TensorFlow data tensors).</param> /// <param name="batch_size">Integer or None. Number of samples per gradient update. If unspecified, batch_sizewill default to 32.</param> /// <param name="epochs">Integer. Number of epochs to train the model. An epoch is an iteration over the entire x and y data provided. Note that in conjunction with initial_epoch, epochs is to be understood as "final epoch". The model is not trained for a number of iterations given by epochs, but merely until the epoch of index epochs is reached.</param> /// <param name="verbose">Integer. 0, 1, or 2. Verbosity mode. 0 = silent, 1 = progress bar, 2 = one line per epoch.</param> /// <param name="callbacks">List of keras.callbacks.Callback instances. List of callbacks to apply during training and validation (if ). See callbacks.</param> /// <param name="validation_split">Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch. The validation data is selected from the last samples in the x and y data provided, before shuffling.</param> /// <param name="validation_data">tuple (x_val, y_val) or tuple (x_val, y_val, val_sample_weights) on which to evaluate the loss and any model metrics at the end of each epoch. The model will not be trained on this data. validation_data will override validation_split.</param> /// <param name="shuffle">Boolean (whether to shuffle the training data before each epoch) or str (for 'batch'). 'batch' is a special option for dealing with the limitations of HDF5 data; it shuffles in batch-sized chunks. Has no effect when steps_per_epoch is not None.</param> /// <param name="class_weight">Optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only). This can be useful to tell the model to "pay more attention" to samples from an under-represented class.</param> /// <param name="sample_weight">Optional Numpy array of weights for the training samples, used for weighting the loss function (during training only). You can either pass a flat (1D) Numpy array with the same length as the input samples (1:1 mapping between weights and samples), or in the case of temporal data, you can pass a 2D array with shape (samples, sequence_length), to apply a different weight to every timestep of every sample. In this case you should make sure to specifysample_weight_mode="temporal" in compile().</param> /// <param name="initial_epoch">Integer. Epoch at which to start training (useful for resuming a previous training run).</param> /// <param name="steps_per_epoch">Integer or None. Total number of steps (batches of samples) before declaring one epoch finished and starting the next epoch. When training with input tensors such as TensorFlow data tensors, the default None is equal to the number of samples in your dataset divided by the batch size, or 1 if that cannot be determined.</param> /// <param name="validation_steps">Only relevant if steps_per_epoch is specified. Total number of steps (batches of samples) to validate before stopping.</param> /// <returns>A History object. Its History.history attribute is a record of training loss values and metrics values at successive epochs, as well as validation loss values and validation metrics values (if applicable).</returns> public static History FitWithMultipleOutputs (this BaseModel model, NDarray x, NDarray[] y, int? batch_size = null, int epochs = 1, int verbose = 1, Callback[] callbacks = null, float validation_split = 0.0f, NDarray[] validation_data = null, bool shuffle = true, Dictionary<int, float> class_weight = null, NDarray sample_weight = null, int initial_epoch = 0, int? steps_per_epoch = null, int? validation_steps = null) { var args = new Dictionary<string, object>(); args["x"] = x; args["y"] = y; args["batch_size"] = batch_size; args["epochs"] = epochs; args["verbose"] = verbose; args["callbacks"] = callbacks; args["validation_split"] = validation_split; if (validation_data != null) { if (validation_data.Length == 2) args["validation_data"] = new PyTuple(new PyObject[] { validation_data[0].PyObject, validation_data[1].PyObject }); else if (validation_data.Length == 3) args["validation_data"] = new PyTuple(new PyObject[] { validation_data[0].PyObject, validation_data[1].PyObject, validation_data[2].PyObject }); } args["shuffle"] = shuffle; if (class_weight != null) args["class_weight"] = ToDict(class_weight); args["sample_weight"] = sample_weight; args["initial_epoch"] = initial_epoch; args["steps_per_epoch"] = steps_per_epoch; args["validation_steps"] = validation_steps; PyObject py = model.InvokeMethod("fit", args); return new History(py); } static PyDict ToDict(Dictionary<int, float> input) { PyDict dict = new PyDict(); foreach (var item in input) { dict[item.Key.ToPython()] = item.Value.ToPython(); } return dict; } /// <summary> /// Generates output predictions for the input samples. /// Computation is done in batches. /// </summary> /// <param name="x">The input data, as a Numpy array (or list of Numpy arrays if the model has multiple inputs).</param> /// <param name="batch_size">Integer. If unspecified, it will default to 32.</param> /// <param name="verbose">Verbosity mode, 0 or 1.</param> /// <param name="steps">Total number of steps (batches of samples) before declaring the prediction round finished. Ignored with the default value of None.</param> /// <param name="callbacks">List of keras.callbacks.Callback instances. List of callbacks to apply during prediction. See callbacks.</param> /// <returns>Numpy array(s) of predictions.</returns> public static NDarray[] PredictMultipleOutputs(this BaseModel model, NDarray x, int? batch_size = null, int verbose = 1, int? steps = null, Callback[] callbacks = null) { var args = new Dictionary<string, object>(); args["x"] = x; args["batch_size"] = batch_size; args["verbose"] = verbose; args["steps"] = steps; args["callbacks"] = callbacks != null ? callbacks : null; var res = model.InvokeMethod("predict", args); var resTuple = PyTuple.AsTuple(res); var length = resTuple.Length(); var finalRes = new NDarray[length]; for (int i = 0; i < length; i++) { finalRes[i] = new NDarray(resTuple[i]); } return finalRes; } } — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#209 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABQJAKN4CJ34WJ3B35U2VFDUCTZSLANCNFSM5EJHY3LQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Regards, Deepak

[Racines]

Here is the PR: #210

[github-actions]

Stale issue message