Scan and Evalutaion into csv

[FlorianBury]

Hi, thanks for your answers on #144.

About the first question, I am currently trying to add the result from the evaluation into the csv file. This is because I believe it will be a better way to select the best models when using Predict(). Right now I just open and write the .csv file in my scripts after the scan ... not always feasible ...

You mentioned new available tools in your answer, is there any doc about them and what version to use ?

Also, I have access to a computing network (handled with slurm) and I wanted to split the trainings between different nodes (CPU, of course slower than a GPU but since there is only one of the latter and hundreds or the former ... parallel wins over serial). SO basically I just split the experiment dict into smaller ones and distribute them over the different nodes. And in the end I concatenate the csv for analysis.

But this poses a problem when appending the csv file due to the file stageout. Is something related to that being investigated or in the making ?

[mikkokotila]

I will put together an example Notebook for the new features. Will try to do it today.

In the meantime, can you clarify the meaning of "stageout" here. Also, did I gather it correctly that what you are doing now for parallelism, is completely automated?

[mikkokotila]

For the evaluation result to go into csv, I will look into this as well. I think it makes sense to have that as an option as it's consistent with Scan().

[FlorianBury]

Great, thanks !

Basically I just re-open the csv file and append it with the evaluation error in another part of the script. I am not sure it will perform correctly when applied on different nodes, still working on that.
EDIT: I got it working now, maybe not optimal but enough for what I need

Yes it is automated but it is rather basic. I just recover the whole set of parameters that talos generates, split it into different subsets and revert them into dictionaries that are saved as pkl files. Then the script launches the scans of each of these sub-dicts on several nodes with slurm.

[mikkokotila]

Sorry it took a little longer. The below is available in v.0.4.6 (for now in dev and daily-dev) and dramatically streamlines the process of evaluation through cross-validation.

scan_object = ta.Scan(x=x_train , y=y_train, params=p, model=gen_model)
scan_object_with_eval = ta.Autom8(scan_object, x_val, y_val)

The above will entail completing an entire pipeline from the scan to evaluation and predictions. The prediction and evaluation results will be added to the scan_object and you can easily export to csv from there if you like. For example to get the evaluation scores with parameters scan_object.data.to_csv('results.csv').

For the most part, you might just want to be able to evaluate the models right there and then and not think about predictions yet, and that's when you can just simply:

scan_object = ta.Scan(x=x_train , y=y_train, params=p, model=gen_model)
scan_object.evaluate_models(x_val, y_val)

Again you could do export to csv using the same scan_object.data.to_csv('results.csv').

[FlorianBury]

Looks much simpler, thanks !