Definition of scoring method unclear

[LukasMosser]

Which score will be used to score an individual prediction?

accuracy(confusion_matrix)
accuracy_adjacent(confusion_matrix, adjacent_facies)

skm.metrics.f1_score(Y_true, Y_pred, sample="weighted")
accuracy*accuracy_adjacent?

[kwinkunks]

Notwithstanding the unresolved conversation in #2, my preference is to use the F1 score as the true measure of performance, since the confusion matrix — while really nice and very practical in a real-world problem — opens another can of worms. Probably several cans, eg.

• How should we relate facies?-
• Are some facies more alike than others?
• Does or should it change the objective function? etc)

Definitely open to others' opinions on this.

[LukasMosser]

I thought the:

• Are some facies more alike than others?
  question is dealt by accuracy_adjacent.

The easiest would be F1 score. Essentially a high F1 score would also mean that the well log is predicted correctly and "adjacency". Only going for high adjacency could mean that your well logs have all the "allowed adjacent" facies but a low F1 score i.e. wrong exact predictions.

Curious to see what others run into here.

[kwinkunks]

Connected to #14

I notice that the cross validation stuff has changed a little in scikit-learn 0.18. There are a couple of k-fold CV classes in there... has anyone played with them?

Docs >> http://scikit-learn.org/stable/tutorial/statistical_inference/model_selection.html#cross-validation-generators

cc @lperozzi

[lperozzi]

I tested stratiefiedKfold. I remember that it gives much higher results
(average f1 score of near 74%). However, as we would like to cross
validate the dataset for each well (i.e., take iteratively a well as a
blind), I'm not sure that any of these are suited. We could always give
a try (i.e., 80/20 train/test, without preserve the well distribution),
but from the geological point of view I'm not sure this is the best
approach.

Matt Hall wrote:

Connected to #14 #14

I notice that the cross validation stuff has changed a little in
scikit-learn 0.18. There are a couple of k-fold CV classes in there...
has anyone played with them?

Docs >>
http://scikit-learn.org/stable/tutorial/statistical_inference/model_selection.html#cross-validation-generators

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#4 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF6UYbGPphw7t-LQl3mVq3wnX1JAu-u9ks5q6idHgaJpZM4Khp-r.

[kwinkunks]

We now have STUART and CRAWFORD facies data so will score the contest based on F1 score against those known facies.

Please see this notebook for a demo of the k-fold workflow in scikit-learn.

I am closing this issue now, because I think those were the main two concerns and — I think! — they have both been addressed. Please let me know if you disagree!