Optimizing CA model toward different metrics

[mfayoub]

Hi everyone,

I've a question regarding running Coordinate Ascent. I noticed from the colab file https://colab.research.google.com/drive/1IjF7yTin1XaNO_6mBNxAoQYTmF0nckk1 that the model is trained using NDCG. Is it possible to train the model using a user-defined metric? Or how can we make the model optimize for different metrics, e.g. MAP or ERR?

Thanks!

[jjfiv]

I added this cell to the bottom of the notebook, but the TrainRequest object has a measure field as a string.

train_request = TrainRequest.coordinate_ascent()
train_request.measure = 'map'
params = train_request.params
params.init_random = True
params.normalize = True
params.seed = 1234567
coordinate_ascent = dataset.train_model(train_request)

# simple python-accessible models (serialization is up to you)
coordinate_ascent.to_dict()

The supported measures are: AP, RR, and NDCG for now. I know ERR is pretty popular in LTR literature, but I've never really used it, myself, so that's why it's been on the backburner.

fastrank/src/evaluators.rs

Lines 148 to 153 in b05cf8a

	evaluator: match name.as_str() {
	"ap" | "map" => Arc::new(AveragePrecision::new(dataset, judgments.clone())),
	"rr" | "mrr" => Arc::new(ReciprocalRank),
	"ndcg" => Arc::new(NDCG::new(depth, dataset, judgments.clone())),
	_ => Err(format!("Invalid training measure: \"{}\"", orig_name))?,
	},

I don't have plans for user-defined functions as metrics, because most of the 'speed' of coordinate ascent comes from how fast you can predict new model scores, sort, and compute your metric, but it shouldn't be hard to add some additional measures.

[mfayoub]

Ok, thanks for your answer.