Question about documentation in probability_of_improvement

[zhefan]

Hi, I wonder if the documentation in probability_of_improvement function in metrics.py is wrong?
Specifically,

scores_x: A matrix of size (num_runs_x x num_tasks) where scores_x[m][n] represent the score on run n of task m for algorithm X.

rliable/rliable/metrics.py

Line 77 in cc5eff5

scores_x: A matrix of size (`num_runs_x` x `num_tasks`) where scores_x[m][n]

)

Should scores_x[n][m] be the score on run n of task m for algorithm X?

Thanks.

[agarwl]

You are right -- I think it should be that score_x[n][m] is score on run n on task m around looking at the code here: https://github.com/google-research/rliable/blob/master/rliable/metrics.py#L92. Will fix.

[zhefan]

Thanks for the clarification.