libSVM Parser Shuffles Documents

[eggie5]

Using the libsvm parser I noticed my documents were getting shuffled. I found this code:

ranking/tensorflow_ranking/python/data.py

Line 411 in d7008f2

np.random.shuffle(doc_list)

Why would one shuffle the documents? Doesn't the order (position) of your relevance labels imply meaning? Wouldn't this throw off DCG?

DCG for reference:

dcg = lambda r: np.sum(r/np.log2(np.arange(2, r.size+2)))

[ramakumar1729]

Learning to rank algorithms ideally do not depend on the input order of the list of documents. DCG and other measures are over the ranking induced by the scores for the documents, and the position/rank of relevance labels within this.

Let me know if you have any further questions.

[eggie5]

Is there any particular reason tf_ranking_libsvm.py (the non-notebook example) doesn't shuffle?

https://github.com/tensorflow/ranking/blob/master/tensorflow_ranking/examples/tf_ranking_libsvm.py#L116

[xuanhuiwang]

Since we only keep the list_size number of documents, shuffling makes us select a random set of list_size documents for every call. Otherwise, we only keep the top list_size documents.

Personally, I prefer to non-shuffling version.