function agf

[Koumougilles]

Hi @adriancorrendo,

It is Gilles Koumou. I thing that there is an error in the function agf designed to compute the Adjusted F-measure.

This is the function:

function (data = NULL, obs, pred, pos_level = 2, atom = FALSE,
tidy = FALSE, na.rm = TRUE)
{
matrix <- rlang::eval_tidy(data = data, rlang::quo(table({
{
pred
}
}, {
{
obs
}
})))
if (nrow(matrix) == 2) {
if (pos_level == 1) {
TP <- matrix[[1]]
TPFP <- matrix[[1]] + matrix[[3]]
TPFN <- matrix[[1]] + matrix[[2]]
TN <- matrix[[4]]
N <- matrix[[4]] + matrix[[3]]
}
if (pos_level == 2) {
TP <- matrix[[4]]
TPFP <- matrix[[4]] + matrix[[2]]
TPFN <- matrix[[4]] + matrix[[3]]
TN <- matrix[[1]]
N <- matrix[[1]] + matrix[[2]]
}
rec <- TP/(TPFN)
prec <- TP/(TPFP)
}
if (nrow(matrix) > 2) {
correct <- diag(matrix)
total_actual <- colSums(matrix)
total_pred <- rowSums(matrix)
TP <- diag(matrix)
TPFP <- rowSums(matrix)
TPFN <- colSums(matrix)
TN <- sum(matrix) - (TPFP + TPFN - TP)
FP <- TPFP - TP
if (atom == FALSE) {
prec <- mean(correct/total_pred)
rec <- mean(correct/total_actual)
warning("For multiclass cases, the agf should be estimated at a class level. Please, consider using atom = TRUE")
}
if (atom == TRUE) {
prec <- correct/total_pred
rec <- correct/total_actual
}
}
F2 <- 5 * ((rec * prec)/((4 * rec) + prec))
invF05 <- (5/4) * ((rec * prec)/(((0.5^2) * rec) + prec))
agf <- sqrt(F2 * invF05)
if (tidy == TRUE) {
return(as.data.frame(agf))
}
if (tidy == FALSE) {
return(list(agf = agf))
}
}
<bytecode: 0x000001283191fe68>
<environment: namespace:metrica>

The error is about invF05. In my opinion invF05 is related to specificity and negative predictive value.

Thanks!

[adriancorrendo]

Hello, @Koumougilles

How are you?

Thanks for your inquiry.

Yet I don't have a clear picture where the error is.
For the AGF equation I simply followed the proposed formula by Maratea, A., Petrosino, A., Manzo, M. (2014). Adjusted-F measure and kernel scaling for imbalanced data learning. Inf. Sci. 257: 331-341. doi:10.1016/j.ins.2013.04.016, as specified here https://adriancorrendo.github.io/metrica/reference/agf.html

In summary, do you mean that what Maratea el al (2014) proposed is misleading? If so, is there another paper demonstrating an undesired behaviour of the AGF ? Is there an alternative/improved equation that fixes what you think is incorrect so I can correct the formula or add a new index?

I would appreciate if you could provide an alternative formula from a paper. At some point, all indices present limitations, and that's how new and refined indices/metrics have been developed over time.

Thank you very much for your inquiry!
I hope we can keep improving the metrics and user experience.

Best regards,
Adrian

[Koumougilles]

Hi @adriancorrendo,

I think you miunderstand the definition in the paper of Maratea et al (2014). The authors write just after the formula of AGF:
"This index accounts for all elements of the original confusion matrix and
provides more weight to patterns correctly classified in the minority class
(the positive class)."

Your AGF uses recall and precision which is defined with only 3 elements of the confusion matrix:

1-TP (true positive)
2-FP (false positive)
3-FN (false negative)

In their paper, the authors precise that invF05 by applied the F score formula to the inverse of the confusion matrix obtained by switching the class labels of each sample (positive samples
become negative and vice versa). I think that you forget to compute the inverse confusion matrix.

Thank you!

[Koumougilles]

Hi @adriancorrendo,

I wrote to Maratea et al (2014). This is my message:

_Dear authors,

I hope that you are doing well.

I am Gilles Koumou. I am interested by your measure Adjusted-F. The measure is available in an R package named metrica. However, I think that the function of Adjusted-F in this package is not exact. In your paper you did not give an explicit formulation of your inverse F score InvF_0.5. I think that it depend on specificity and negative predictive value. If I am right could you please give me the exact formulation or confirm my following formulation:

InvF_0.5=(1+0.5^2)((npvspec)/(0.5^2npv+spec))

où npv=negative predictive value
spec=specificity

Thank you_

This is the answer of the corresponding author Antonio Maratea:

Dear Gilles,
in the paper I wrote "swapping labels" because at that time it seemed to me the clearest way to explain the measure.
If we swap positive and negative labels, then the precision becomes the negative predictive value and the recall becomes sensitivity.
The purpose of Adjusted-F is to account for all four terms of the confusion matrix and to give more weight to the minority class.
So you are absolutely right and the formula is correct. If I had to rewrite the paper, I would have used it.
Best regards
Antonio Maratea

So the correct formulation of InvF0.5 is
InvF0.5=(1+0.5^2)((npvspec)/(0.5^2npv+spec))

I transfer these emails to you.

Thank you!

[adriancorrendo]

Hi Gilles,

How are you?

I had some time available so I already updated the Agf formula on the "master" branch. I mentioned you in an acknowledgment in the NEWS.md document. Would you mind checking the new agf() out? You will need to install the package from GitHub instead of CRAN. There is another new metric added called P4.

Once receiving your confirmation I will close this issue and upload the new version to CRAN.

Thank you for your contribution!

Best
Adrian

[Koumougilles]

Hi Adrian Well received. I will check the function agf() next week and give you feedbacks. Best regards. Obtenir Outlook pour Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Adrian A. Correndo ***@***.***> Sent: Sunday, March 24, 2024 6:36:13 PM To: adriancorrendo/metrica ***@***.***> Cc: Nettey Boevi Gilles Koumou ***@***.***>; Mention ***@***.***> Subject: Re: [adriancorrendo/metrica] function agf (Issue #37) Hi Gilles, How are you? I had some time available so I already updated the Agf formula on the "master" branch. I mentioned you in an acknowledgment in the NEWS.md document. Would you mind checking the new agf() out? You will need to install the package from GitHub instead of CRAN. There is another new metric added called P4. Once receiving your confirmation I will close this issue and upload the new version to CRAN. Thank you for your contribution! Best Adrian — Reply to this email directly, view it on GitHub<#37 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BG7GL7UIV4TEUDMSCHWLX43YZ5IN3AVCNFSM6AAAAABEW5T4COVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJWHE3TAMJZHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[adriancorrendo]

Hi Adrian Well received. I will check the function agf() next week and give you feedbacks. Best regards. Obtenir Outlook pour Androidhttps://aka.ms/AAb9ysg
…
________________________________ From: Adrian A. Correndo @.> Sent: Sunday, March 24, 2024 6:36:13 PM To: adriancorrendo/metrica @.> Cc: Nettey Boevi Gilles Koumou @.>; Mention @.> Subject: Re: [adriancorrendo/metrica] function agf (Issue #37) Hi Gilles, How are you? I had some time available so I already updated the Agf formula on the "master" branch. I mentioned you in an acknowledgment in the NEWS.md document. Would you mind checking the new agf() out? You will need to install the package from GitHub instead of CRAN. There is another new metric added called P4. Once receiving your confirmation I will close this issue and upload the new version to CRAN. Thank you for your contribution! Best Adrian — Reply to this email directly, view it on GitHub<#37 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BG7GL7UIV4TEUDMSCHWLX43YZ5IN3AVCNFSM6AAAAABEW5T4COVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJWHE3TAMJZHA. You are receiving this because you were mentioned.Message ID: @.***>

Hi, Gilles

How are you?
Could you please let me know if you were able to test the updated version on the master branch?

Thanks,
Adrian

[Koumougilles]

Hi Adrian,

I check you function agf ():

#' @title Adjusted F-score
#' @name agf
#' @description It estimates the Adjusted F-score for a nominal/categorical
#' predicted-observed dataset.
#' @param data (Optional) argument to call an existing data frame containing the data.
#' @param obs Vector with observed values (character | factor).
#' @param pred Vector with predicted values (character | factor).
#' @param pos_level Integer, for binary cases, indicating the order (1|2) of the level
#' corresponding to the positive. Generally, the positive level is the second (2)
#' since following an alpha-numeric order, the most common pairs are
#' (Negative | Positive), (0 | 1), (FALSE | TRUE). Default : 2.
#' @param atom Logical operator (TRUE/FALSE) to decide if the estimate is made for
#' each class (atom = TRUE) or at a global level (atom = FALSE); Default : FALSE.
#' When dataset is "binomial" atom does not apply.
#' @param tidy Logical operator (TRUE/FALSE) to decide the type of return. TRUE
#' returns a data.frame, FALSE returns a list; Default : FALSE.
#' @param na.rm Logic argument to remove rows with missing values
#' (NA). Default is na.rm = TRUE.
#' @return an object of class numeric within a list (if tidy = FALSE) or within a
#' data frame (if tidy = TRUE).
#' @details The Adjusted F-score (or Adjusted F-measure) is an improvement over the
#' F1-score especially when the data classes are imbalanced. This metric more properly
#' accounts for the different misclassification costs across classes. It weights more
#' the sensitivity (recall) metric than precision and gives strength to the false negative
#' values. This index accounts for all elements of the original confusion matrix and
#' provides more weight to patterns correctly classified in the minority class (positive).
#'
#' It is bounded between 0 and 1.
#' The closer to 1 the better. Values towards zero indicate low performance.
#' For the formula and more details, see
#' online-documentation
#' @references
#' Maratea, A., Petrosino, A., Manzo, M. (2014).
#' Adjusted-F measure and kernel scaling for imbalanced data learning.
#' Inf. Sci. 257: 331-341. \doi{10.1016/j.ins.2013.04.016}
#' @examples
#' \donttest{
#' set.seed(123)
#' # Two-class
#' binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE),
#' predictions = sample(c("True","False"), 100, replace = TRUE))
#' # Multi-class
#' multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, replace = TRUE),
#' predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE) )
#'
#' # Get F-score estimate for two-class case
#' agf(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)
#'
#' # Get F-score estimate for each class for the multi-class case
#' agf(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)
#'
#' # Get F-score estimate for the multi-class case at a global level
#' agf(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)
#' }
#' @Rdname agf
#' @importFrom rlang eval_tidy quo enquo
#' @export
agf <- function(data=NULL, obs, pred,
pos_level = 2, atom = FALSE,
tidy = FALSE, na.rm = TRUE){

matrix <- rlang::eval_tidy(
data = data,
rlang::quo(table({{pred}}, {{obs}}) ) )

If binomial

if (nrow(matrix) == 2){
if (pos_level == 1){
TP <- matrix[[1]]
TPFP <- matrix[[1]] + matrix[[3]]
TPFN <- matrix[[1]] + matrix[[2]]
TNFN <- matrix[[4]] + matrix[[2]]
TN <- matrix[[4]]
N <- matrix[[4]] + matrix[[3]] }

if (pos_level == 2){ 
  TP <- matrix[[4]]
  TPFP <- matrix[[4]] + matrix[[2]]
  TPFN <- matrix[[4]] + matrix[[3]]
  TNFN <- matrix[[1]] + matrix[[3]]
  TN <- matrix[[1]]
  N <- matrix[[1]] + matrix[[2]] }

rec <- TP/ (TPFN) 
prec <- TP/ (TPFP)
spec <- TN/N
npv <- TN/ (TNFN)

}

If multinomial

if (nrow(matrix) >2) {

# Calculations
correct <- diag(matrix)
total_actual <- colSums(matrix)
total_pred <- rowSums(matrix)

TP   <- diag(matrix)
TPFP <- rowSums(matrix)
TPFN <- colSums(matrix)
TN   <- sum(matrix) - (TPFP + TPFN - TP)
FP   <- TPFP - TP
FN <- TPFN - TP

if (atom == FALSE) { 
  prec <- mean(correct / total_pred)
  rec <- mean(correct / total_actual)
  spec <- mean( TN / (TN + FP) )
  npv <- mean( TN / (TN + FN) )
  warning("For multiclass cases, the agf should be estimated at a class level. Please, consider using `atom = TRUE`")
}

if (atom == TRUE) { 
  prec <- correct / total_pred
  rec <- correct / total_actual
  spec <- TN / (TN + FP)
  npv <- TN / (TN + FN)
} 

}

Formula components

F2 <- 5 * ( (recprec) / ((4rec) + prec) )
#invF05 <- (5/4) * ( (rec*prec) / (( (0.5^2) * rec)+prec) )
invF05 <- (5/4) ( (npvspec) / (( (0.5^2) * npv) + spec))

agf <- sqrt(F2 * invF05)

if (tidy==TRUE){
return(as.data.frame(agf)) }

if (tidy==FALSE){
return(list("agf"=agf)) }
}

There is an error in your F2:

F2 must be F2 <- 5 * ( (recprec) / ((4prec) + rec) )

Thanks.