Made embeddings and LLMs dependent on metric in evaluate function #628
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since we already specify whether or not a metric requires an LLM/embeddings model via inheritence from
MetricWithLLM
andMetricWithEmbeddings
, there isn't really a need to force the use of default LLMs/embeddings inevaluate
if, for example, no metrics that need embeddings are specified inmetrics
.I believe that initiating an LLM/embedding model at the metric level will help clarify how to use
evaluate
, and will make things simpler in the future when more metrics are added as it decouples needing to initialize LLMs/embedding models for metrics that potentially don't need it. They are even optional arguments to the function itself.Copilot Description
This pull request mainly refactors the
evaluate
function in thesrc/ragas/evaluation.py
file. The changes aim to optimize the import and usage ofllm_factory
andembedding_factory
, and clarify the function comments.Here are the main changes:
src/ragas/evaluation.py
: Two new imports were added to the top of the file:embedding_factory
andllm_factory
fromragas.embeddings.base
andragas.llms
respectively. This change helps to avoid repetitive imports within theevaluate
function.Changes within the
evaluate
function:llm
andembeddings
parameters were updated to specify that the default language model and embeddings are used for metrics which require an LLM or embeddings. This provides more clarity on the function's behavior.llm
andembeddings
was simplified. Thellm_factory
andembedding_factory
are now only called whenllm
andembeddings
areNone
and the corresponding metric requires them. This change removes the need for importingllm_factory
andembedding_factory
inside the function.