Ragas Evaluate Function broken with ChatVertexAI Langchain class.

[shawnmittal]

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
When passing a ChatVertexAI based llm object to the evaluate function, the function attempts to run .set_run_config on the object. However, the ChatVertexAI object has no such attribute.

Ragas version: 0.1.7
Python version: 3.11.8

Code to Reproduce

from datasets import load_dataset

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")

from ragas.metrics import (
    context_precision,
    answer_relevancy,  # AnswerRelevancy
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness

# list of metrics we're going to use
metrics = [
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
    harmfulness,
]

from langchain_google_vertexai import ChatVertexAI, VertexAIEmbeddings

llm = ChatVertexAI(model_name="gemini-1.5-pro-preview-0409", safety_settings=safety_settings)
embeddings = VertexAIEmbeddings(model_name="textembedding-gecko@003")

for m in metrics:
    # change LLM for metric
    m.__setattr__("llm", llm)

    # check if this metric needs embeddings
    if hasattr(m, "embeddings"):
        # if so change with VertexAI Embeddings
        m.__setattr__("embeddings", embeddings)

from ragas import evaluate

result = evaluate(
    amnesty_qa["eval"].select(range(1)),  # using 1 as example due to quota constrains
    metrics=metrics,
)

result

Error trace

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[9], line 38
     34         m.__setattr__("embeddings", embeddings)
     36 from ragas import evaluate
---> 38 result = evaluate(
     39     amnesty_qa["eval"].select(range(1)),  # using 1 as example due to quota constrains
     40     metrics=metrics,
     41 )
     43 result

File ~/Documents/code/ragas-evaluation/venv/lib/python3.11/site-packages/ragas/evaluation.py:179, in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, run_config, raise_exceptions, column_map)
    176             answer_correctness_is_set = i
    178 # initialize all the models in the metrics
--> 179 [m.init(run_config) for m in metrics]
    181 executor = Executor(
    182     desc="Evaluating",
    183     keep_progress_bar=True,
    184     raise_exceptions=raise_exceptions,
    185     run_config=run_config,
    186 )
    187 # new evaluation chain

File ~/Documents/code/ragas-evaluation/venv/lib/python3.11/site-packages/ragas/evaluation.py:179, in <listcomp>(.0)
    176             answer_correctness_is_set = i
...
    139         f"Metric '{self.name}' has no valid LLM provided (self.llm is None). Please initantiate a the metric with an LLM to run."  # noqa
    140     )
--> 141 self.llm.set_run_config(run_config)

AttributeError: 'ChatVertexAI' object has no attribute 'set_run_config'

Expected behavior
The evaluate function to run.

Additional context
The documentation from which the code is pulled uses a deprecated version of langchain's ChatVertexAI object. This could be the cause of the problem, I've not investigated that yet. This is captured in issue #515.

[Kaby-Lake]

Hi @shawnmittal , try add a LangchainLLMWrapper can help, for me this works:

from ragas.llms import LangchainLLMWrapper

llm = ChatVertexAI(model_name="gemini-pro")
ragas_vertexai_llm = LangchainLLMWrapper(llm)

evaluate_result = evaluate(
        dataset=evaluate_dataset,
        metrics=metrics,
        llm=ragas_vertexai_llm,
)

[shawnmittal]

@Kaby-Lake thanks! That solution works. I'll update the docs and submit a PR addressing this issue.

[jjmachan]

thanks for making the PR @shawnmittal ❤️
much appreciated 🙂

[shawnmittal]

@jjmachan premature issue closure since PR isn't approved yet, but realize this is a dupe of #515 so all is well. I've submitted PR #915 to address this one and the dupe issue.