Harrison/arize #2180
Merged
Harrison/arize #2180
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
**Arize CallBack Handler for LangChain <> Arize Integration**
This pull request adds a new Python class, ArizeCallbackHandler, which
serves as a callback handler that logs data into an ML Observability
platform named Arize whenever a Langchain Language Model (LLM) runs
within Langchain. Specifically, it logs prompt and response pairs along
with their embeddings for monitoring and further finetuning.
The ArizeCallbackHandler class is implemented as a subclass of
BaseCallbackHandler and contains the following methods for handling
different events triggered during the LLM execution:
on_llm_start: records the prompts when an LLM starts
on_llm_end: logs data to Arize when an LLM ends, including prompt,
response embeddings, id and timestamp
The constructor of ArizeCallbackHandler takes four optional arguments
(Needed for logging into Arize):
- model_id: the ID of the model in Arize (default: None)
- model_version: the version of the model in Arize (default: None)
- SPACE_KEY: the space key for the Arize client (default: None)
- API_KEY: the API key for the Arize client (default: None)
When ArizeCallbackHandler is instantiated, it initializes empty lists to
store the prompt/response pairs and other necessary data. It also
creates an embedding generator for generating embeddings from prompts
and responses. For embeddings consistency and ease of use, Arize's
embedding generator is used. Finally, it creates an Arize client and
checks if the SPACE_KEY and API_KEY are set with the correct value.
**Motivation and Context**
The purpose of this pull request is to provide a way to log data into
Arize whenever an LLM runs within Langchain. This will allow us to
monitor the performance of the LLM and make further fine-tuning to the
model if necessary. As Arize develops LLM-supporting capabilities, some
minor changes might be necessary in the future.
**How Has This Been Tested?**
I have tested this code by using weights and biases test cases with
LLMs, LLM chains, and LLM agents. I checked if the prompt/response pairs
and their embeddings were logged into Arize correctly. I have also
tested this code with different model IDs and model versions in Arize to
ensure that the data was logged to the correct model.
An example test is provided below:
```
from langchain.callbacks import StdOutCallbackHandler
from langchain.callbacks.base import CallbackManager
from langchain.llms import OpenAI
arize_callback = ArizeCallbackHandler(
model_id="llm-langchain-demo",
model_version="1.0",
SPACE_KEY="88aa7af",
API_KEY="7cab3a79306ef1cd9e7"
)
manager = CallbackManager([StdOutCallbackHandler(), arize_callback])
llm = OpenAI(temperature=0, callback_manager=manager, verbose=True)
```
```llm_result = llm.generate(["Tell me a joke","Tell me a poem"])```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.