Create results class #54
Conversation
src/rank_llm/result.py
Outdated
| values = [] | ||
| for info in result.ranking_exec_summary: | ||
| values.append(info.__dict__) | ||
| exec_summary[key] = values |
There was a problem hiding this comment.
would this work in case we have duplicate queries in our dataset? Feels like things might be a bit off in this case
There was a problem hiding this comment.
No it would not since the query is the key, but this is identical to the current behavior in input token file where query is the key:
{
"How much impact do masks have on preventing the spread of the COVID-19?": [
35947,
810
],
...
}
I can change this to an array of dictionaries instead, with "query" and "ranking_exec_summary" as keys, WDYT?
There was a problem hiding this comment.
yep that array idea works better! thanks
| @@ -169,28 +170,21 @@ def retrieve_and_store( | |||
| Path(f"retrieve_results/{self._retrieval_method.name}").mkdir( | |||
There was a problem hiding this comment.
Can this go in the writer somewhere too?
There was a problem hiding this comment.
The writer is used after the reranking too. That's why I let the retriever and reranker to identify the target file.
| f"{hit['qid']} Q0 {hit['docid']} {hit['rank']} {hit['score']} rank\n" | ||
| ) | ||
| writer.write_in_trec_eval_format( | ||
| f"retrieve_results/{self._retrieval_method.name}/trec_results_{self._dataset}.txt" |
There was a problem hiding this comment.
Can this naming be automated in writer too?
There was a problem hiding this comment.
I prefer to keep it this way since: 1- the writer is shared between retriever and reranker. 2- all this information about the dataset, ertrieval method, etc should be passed to the writer to be able generate the names properly.
This cl:
1- creates a
Resultclass and integrates it to retriever. In all retrieval modes the retriever now returns a list[Result] rather than list[dict].2- creates a helper 'ResultsWriter` class to take care of writing results to files.
Changing the reranker to read from this new result type will be submitted in a follow up cl.
TESTED=ran all demos of different retrieval modes.