feat: implement funtions to retrieve metrics and examples #687
Conversation
github-actions
bot
added
size/s
area/client
area/core
area/testing
guenthermi
requested review from
gmastrapas,
bwanglzu,
LMMilliken and
matousek-martin
and removed request for
gmastrapas
finetuner/run.py
Outdated
| def example_results(self) -> Dict: | ||
| """Get the results of example queries from the evaluation data of the | ||
| :class:`Run`. | ||
|
|
||
| :return: dictionary with results before and after fine-tuning. | ||
| """ | ||
| self._check_run_status_finished() | ||
| return self._client.get_run_examples( | ||
| experiment_name=self._experiment_name, run_name=self._name | ||
| ) |
There was a problem hiding this comment.
if we can make this function more user friendly, would be even better. For example, 2 colums compare, if it is image/mesh, try to display
There was a problem hiding this comment.
Good idea! However, I think sometimes it is good to get them as dictionaries, if you want to further process them. For the case that you want to display them directly to the console, I implemented two additional functions display_metrics and display_examples. It looks like this:
finetuner/console.py
Outdated
| for i, match in enumerate(results[query]): | ||
| if i >= k: | ||
| break |
There was a problem hiding this comment.
| for i, match in enumerate(results[query]): | |
| if i >= k: | |
| break | |
| for i, match in enumerate(results[query])[:k]: |
| You can retrieve them with the {func}`~Run.example_results()` function. | ||
| Alternatively, you can use the {func}`~Run.display_metrics()` function to display a table of the Top-K results before and after fine-tuning to the console. |
There was a problem hiding this comment.
Maybe explain what K is in this context and how to set it
There was a problem hiding this comment.
ok I mention what Top-K means, how to set k is explained in the developer documentation which you get when clicking on the link
| ### Show evaluation metrics | ||
|
|
||
| During the fine-tuning process, the evaluation metrics are displayed in the logs, which you can retrieve via the {func}`~Run.logs()` function. | ||
| After running the fine-tuning job has finished, the evaluation metrics can be retrieved from the cloud by calling the {func}`~Run.metrics()` function. |
There was a problem hiding this comment.
| After running the fine-tuning job has finished, the evaluation metrics can be retrieved from the cloud by calling the {func}`~Run.metrics()` function. | |
| After the fine-tuning job has finished, the evaluation metrics can be retrieved from the cloud by calling the {func}`~Run.metrics()` function. | |
| If you want to compare the top-k retrieval results before and after fine-tuning, you can set the `gather_examples` parameter of the evaluation callback to `True`. | ||
| In this case, the evaluation callback will store the top-k results for each query document before and after fine-tuning. | ||
| You can retrieve them with the {func}`~Run.example_results()` function. | ||
| Alternatively, you can use the {func}`~Run.display_metrics()` function to display a table of the Top-K results before and after fine-tuning to the console. |
There was a problem hiding this comment.
| Alternatively, you can use the {func}`~Run.display_metrics()` function to display a table of the Top-K results before and after fine-tuning to the console. | |
| Alternatively, you can use the {func}`~Run.display_examples()` function to display a table of the Top-K results before and after fine-tuning to the console. | |
finetuner/run.py
Outdated
| @@ -99,6 +99,45 @@ def stream_logs(self, interval: int = 5) -> Iterator[str]: | |||
| experiment_name=self._experiment_name, run_name=self._name | |||
| ) | |||
|
|
|||
| def metrics(self) -> Dict: | |||
There was a problem hiding this comment.
return type hint? Dict[str, float]?
There was a problem hiding this comment.
no its Dicct[str, Dict[str, float]] but yes I can add it
finetuner/run.py
Outdated
| for stage in metrics: | ||
| print_metrics(stage, metrics[stage]) | ||
|
|
||
| def example_results(self) -> Dict: |
There was a problem hiding this comment.
| def example_results(self) -> Dict: | |
| def example_results(self) -> Dict[str, str]: | |
| ```? |
|
📝 Docs are deployed on https://ft-feat-support-metrics-and-examples--jina-docs.netlify.app 🎉 |
Implement functions to retrieve metrics and examples