Significance of MSE vs Kendall's Tau / Spearman's Rank Correlation

[afzalxo]

Hi folks. I am working on designing a surrogate benchmark for some hardware specific performance metrics based on the principles suggested in your work.

I am currently using a small dataset of between 500 - 1000 model architectures from within an MNASNet-like search space with XGB to evaluate the performance of this surrogate with only this small dataset. The hyperparameters utilized for XGB are copied from your work.

I am getting high validation/test MSE results (~ 0.4 to 0.6) but with a high Kendall's Tau (~0.92) and Spearman's rank correlation (~0.98).

When I utilize the same number of models selected randomly from nb301_dataset (from random search directory) offered by you to train the surrogate, I get low MSE (~0.16) but with low KT (0.60) and Spearman's (0.78).

I'm wondering if this disparity could potentially be due to sub-optimal values of the hyperparameters. Do you have some insights on what could cause such a huge difference in the performance of the predictor? Furthermore, for evaluating performance of a surrogate, do you think Kendall's Tau or Spearman's rank correlation is a better metric compared to MSE, or vice versa.

[arberzela]

Thanks for trying out our code. The discrepancy might be because we used an ensemble of XGB models in NB301. But I agree that better training hyperparameters might increase the performance even more.