core: Azure OpenAI support for ChatGPT and GPT-4 chat models #195

zioproto · 2023-05-04T12:06:25Z

To use with an Azure OpenAI Service resource with either the gpt-35-turbo (preview), or the gpt-4 (preview)1 models deployed

Fixes #211

core/cat/factory/llm.py

zioproto · 2023-05-12T19:16:23Z

@pieroit this PR LGTM and is ready to merge. It would be nice to have someone else testing. Thanks

cristianorevil · 2023-05-13T16:54:14Z

I have tested the PR with multiples prompts (simples and complex) and all works fine! @pieroit for me you can merge.

zioproto · 2023-05-15T12:10:28Z

core/cat/factory/llm.py

+    openai_api_base: str
+    api_type: str = "azure"
+    # https://learn.microsoft.com/en-gb/azure/cognitive-services/openai/reference#chat-completions
+    api_version: str = "2023-03-15-preview"


@pieroit I found a bug. This is supposed to be:

openai_api_version: str = "2023-03-15-preview"

https://github.com/hwchase17/langchain/blob/master/langchain/chat_models/azure_openai.py#LL46C10-L46C10

I think I did not catch it earlier because I had an env variable set.

However when I fix the name then I have the problem that the requests have api-version=2022-12-01 and they result in HTTP 404

cheshire_cat_core | [2023-05-15 11:58:45,916] DEBUG util.py 60 -=> message='Request to OpenAI API' method=post path=https://xxxxxxx.openai.azure.com//openai/deployments/gpt-35-turbo/chat/completions?api-version=2022-12-01

Do you have any idea why ?

@cristianorevil did you have environment variables set for api version when you tested ?

nope. My .env always is:

Decide host and port for your Cat. Default will be localhost:1865

CORE_HOST=localhost
CORE_PORT=1865

Decide host and port for your Cat Admin. Default will be localhost:3000

ADMIN_HOST=localhost
ADMIN_PORT=3000

It is not the environment variables. The cat is mixing configurations between the completion model and the chat model. In particular it seems to be using the api_version from the completion model all the times.
For this reason if you never configured in the database the completion model, the chat model will not work.
I need @pieroit to help me to debug this because it seems that the function that loads the llm is involved.

@pieroit I need your help here:

for some reason it works only if in the DB there is both api_version and openai_api_version`. Here is a dump of a working db:

PRAGMA foreign_keys=OFF; BEGIN TRANSACTION; CREATE TABLE setting ( setting_id CHAR(32) NOT NULL, value JSON, "createdAt" TIMESTAMP DEFAULT (CURRENT_TIMESTAMP) NOT NULL, "updatedAt" TIMESTAMP DEFAULT (CURRENT_TIMESTAMP) NOT NULL, name VARCHAR NOT NULL, category VARCHAR NOT NULL, enabled BOOLEAN NOT NULL, PRIMARY KEY (setting_id) ); INSERT INTO setting VALUES('13c2c4bd72864e74bd13f9ee124ce3ef','{"model_name": "gpt-35-turbo", "api_type": "azure", "openai_api_version": "2023-03-15-preview", "api_version": "2023-03-15-preview", "openai_api_key": "<redacted>", "openai_api_base": "https://xxxxxxxxxxxxxx.openai.azure.com/", "deployment_name": "gpt-35-turbo"}','2023-05-15 14:14:27','2023-05-15 14:44:12','LLMAzureChatOpenAIConfig','llm_factory',1); INSERT INTO setting VALUES('7a1dd487913b49d58c50b02fc71a0e53','{}','2023-05-15 14:16:14','2023-05-15 14:16:14','LLMDefaultConfig','llm_factory',1); INSERT INTO setting VALUES('529e8710dc894ad78daecd874e7132a3','{"api_type": "azure", "api_version": "2023-03-15-preview", "deployment_name": "gpt-35-turbo", "model_name": "gpt-35-turbo", "openai_api_base": "https://xxxxxxxxxxxxxx.openai.azure.com/", "openai_api_key": "<redacted>"}','2023-05-15 14:16:55','2023-05-15 14:44:08','LLMAzureOpenAIConfig','llm_factory',1); INSERT INTO setting VALUES('f1d72450772e4674bcaa4ebc453554d3','{"name": "LLMAzureChatOpenAIConfig"}','2023-05-15 14:44:12','2023-05-15 14:44:12','llm_selected','llm',1); COMMIT;

I confirm that importing this DB works:

PRAGMA foreign_keys=OFF; BEGIN TRANSACTION; CREATE TABLE setting ( setting_id CHAR(32) NOT NULL, value JSON, "createdAt" TIMESTAMP DEFAULT (CURRENT_TIMESTAMP) NOT NULL, "updatedAt" TIMESTAMP DEFAULT (CURRENT_TIMESTAMP) NOT NULL, name VARCHAR NOT NULL, category VARCHAR NOT NULL, enabled BOOLEAN NOT NULL, PRIMARY KEY (setting_id) ); INSERT INTO setting VALUES('13c2c4bd72864e74bd13f9ee124ce3ef','{"model_name": "gpt-35-turbo", "api_type": "azure", "openai_api_version": "2023-03-15-preview", "api_version": "2023-03-15-preview", "openai_api_key": "<redacted>", "openai_api_base": "https://xxxxxxxx.openai.azure.com/", "deployment_name": "gpt-35-turbo"}','2023-05-15 14:14:27','2023-05-15 14:44:12','LLMAzureChatOpenAIConfig','llm_factory',1); INSERT INTO setting VALUES('f1d72450772e4674bcaa4ebc453554d3','{"name": "LLMAzureChatOpenAIConfig"}','2023-05-15 14:44:12','2023-05-15 14:44:12','llm_selected','llm',1); COMMIT;

I create the file like this:

sqlite3 core/metadata-v3.db < db.sql

For some reasons when the db both values are present:

"openai_api_version": "2023-03-15-preview", "api_version": "2023-03-15-preview"

everything works fine.

@cristianorevil could you check in your sqlite db if you also have both "openai_api_version": "2023-03-15-preview", "api_version": "2023-03-15-preview" ?

Yes i have both

pieroit · 2023-05-15T13:09:49Z

@zioproto should I wait to merge?

zioproto · 2023-05-15T13:19:21Z

@pieroit wait.

pieroit · 2023-05-15T13:34:08Z

@zioproto for Openai there are two different classes in langhcain, langchain.llms.OpenAI and langchain.llms.OpenAIChat (respectively for completion and chat). They are used respectively by two cat factory classes,
cat.factory.lm.LLMOpenAIConfig and LLMOpenAIChatConfig

It should be the same with Azure I guess

zioproto · 2023-05-15T13:52:43Z

@zioproto for Openai there are two different classes in langhcain, langchain.llms.OpenAI and langchain.llms.OpenAIChat (respectively for completion and chat). They are used respectively by two cat factory classes, cat.factory.lm.LLMOpenAIConfig and LLMOpenAIChatConfig

It should be the same with Azure I guess

It is not the same, there is langchain.chat_models.AzureChatOpenAI

zioproto · 2023-05-15T15:29:43Z

Maybe we should wait for this LangChain PR before moving forward ?
langchain-ai/langchain#3775

To use with an Azure OpenAI Service resource with either the gpt-35-turbo (preview), or the gpt-4 (preview)1 models deployed * https://learn.microsoft.com/en-us/azure/cognitive-services/openai/chatgpt-quickstart?pivots=programming-language-python&tabs=command-line * https://learn.microsoft.com/en-gb/azure/cognitive-services/openai/reference#chat-completions

zioproto · 2023-05-15T21:43:07Z

I need some help debugging this:

This should be what happens in the cat:
https://gist.github.com/zioproto/0f4b198a79a1ac6e44579a06768ed2bc
This script works!

In the db of the cat we save a config like:

{
  "model_name": "gpt-35-turbo",
  "openai_api_type": "azure",
  "openai_api_version": "2023-03-15-preview",
  "openai_api_key": "<REDACTED>",
  "openai_api_base": "https://xxxxxxxx.openai.azure.com/",
  "deployment_name": "gpt-35-turbo"
}

If I inspect the LLMChain object just before the call hyde_text = self.hypothetis_chain.run(text) everything looks in order:

LLMChain(memory=None, callbacks=None, callback_manager=None, verbose=True, prompt=PromptTemplate(input_variables=['input'], output_parser=None, partial_variables={}, template='You will be given a sentence.\nIf the sentence is a question, convert it to a plausible answer. If the sentence does not contain an question, just repeat the sentence as is without adding anything to it.\n\nExamples:\n- what furniture there is in my room? --> In my room there is a bed, a guardrobe and a desk with my computer\n- where did you go today --> today I was at school\n- I like ice cream --> I like ice cream\n- how old is Jack --> Jack is 20 years old\n\nSentence:\n- {input} --> ', template_format='f-string', validate_template=True), llm=AzureChatOpenAI(verbose=False, callbacks=None, callback_manager=None, client=<class 'openai.api_resources.chat_completion.ChatCompletion'>, model_name='gpt-35-turbo', temperature=0.7, model_kwargs={}, openai_api_key='<REDACTED>', openai_api_base='https://xxxxxxx.openai.azure.com/', openai_organization='', request_timeout=None, max_retries=6, streaming=False, n=1, max_tokens=None, deployment_name='gpt-35-turbo', openai_api_type='azure', openai_api_version='2023-03-15-preview'), output_key='text')

But the call fails because the API version is not used, I have no idea where it reads 2022-12-01

cheshire_cat_core           | [2023-05-15 21:21:59.336] DEBUG  openai.util.py 60 -=> message='Request to OpenAI API' method=post path=https://xxxxxxxxx.openai.azure.com//openai/deployments/gpt-35-turbo/chat/completions?api-version=2022-12-01

I discovered by accident that if in the DB I adde the extra value

  "api_version": "2023-03-15-preview",

then this api_version is propagated to the mode_kwargs, that now are empty: model_kwargs={}

model_kwargs{'api_version': '2023-03-15-preview'}

then everything works and the value is used to build the URL.

This makes no sense to me looking at:
https://github.com/hwchase17/langchain/blob/1e467d9fc41e30f80ae8c7763b640b1fa7857561/langchain/chat_models/azure_openai.py#L16-L48

pieroit · 2023-05-15T21:49:40Z

What is the difference between openai_api_version and api-version ?
Are they supposed to be the same thing?

zioproto · 2023-05-15T22:22:29Z

I fixed the problem with commit f74d7b3

it is the same problem described here:
langchain-ai/langchain#4234

zioproto · 2023-05-16T06:41:40Z

@cristianorevil could you test this latest version please ? But you need to delete your sqlite3 database before testing, thank you

zioproto · 2023-05-16T08:16:28Z

This is why it was hard to debug:

import os
import openai
import langchain
from pydantic import PyObject
from langchain.llms import AzureOpenAI
from langchain.chat_models import AzureChatOpenAI
from langchain.embeddings import OpenAIEmbeddings

embedderconfig = {
    "openai_api_key": "<secret>",
    "model": "ada",
    "openai_api_base": "https://dummy.openai.azure.com/",
    "api_type": "azure",
    "api_version": "embedderVERSION",
    "deployment": "ada",
}


class EmbedderAzureOpenAIConfig(OpenAIEmbeddings):
    openai_api_key: str
    model: str
    openai_api_base: str
    api_type: str
    api_version: str
    deployment: str

    _pyclass: PyObject = langchain.embeddings.OpenAIEmbeddings


chatconfig = {
    "model_name": "gpt-35-turbo",
    "openai_api_type": "azure",
    "openai_api_version": "chatVERSION",
    "openai_api_key": "<secret>",
    "openai_api_base": "https://dummy.openai.azure.com/",
    "deployment_name": "gpt-35-turbo",
}


class LLMAzureChatOpenAIConfig(AzureChatOpenAI):
    openai_api_key: str
    model_name: str = "gpt-35-turbo"  # or gpt-4, use only chat models !
    openai_api_base: str
    openai_api_type: str = "azure"
    openai_api_version: str = "2023-03-15-preview"

    deployment_name: str

    _pyclass: PyObject = langchain.chat_models.AzureChatOpenAI


llmconfig = {
    "api_type": "azure",
    "openai_api_key": "<secret>",
    "api_version": "LLMVersion",
    "openai_api_base": "https://dummy.openai.azure.com/",
}


class LLMAzureOpenAIConfig(AzureOpenAI):
    openai_api_key: str
    openai_api_base: str
    api_type: str = "azure"
    # https://learn.microsoft.com/en-us/azure/cognitive-services/openai/reference#completions
    # Current supported versions 2022-12-01 or 2023-03-15-preview
    api_version: str
    deployment_name: str = "text-davinci-003"
    model_name: str = "text-davinci-003"  # Use only completion models !

    _pyclass: PyObject = langchain.llms.AzureOpenAI


llm = LLMAzureOpenAIConfig(**llmconfig)
print(openai.api_version)
chat = LLMAzureChatOpenAIConfig(**chatconfig)
print(openai.api_version)
embedder = EmbedderAzureOpenAIConfig(**embedderconfig)
print(openai.api_version)

The output of this code is:

None
chatVERSION
2022-12-01

This means that LLMAzureChatOpenAIConfig and EmbedderAzureOpenAIConfig overwrite the global openai.api_version object.

I will open a bug to LangChain

cristianorevil · 2023-05-16T11:58:26Z

@zioproto i have tested (deleting first sqlite db) and NOT work. I have configured only 1 model (Azure Chat) and saved. After that an error occurs in log:

"Did not find openai_api_version, please add an environment variable OPENAI_API_VERSION which contains it, or pass openai_api_version as a named parameter."

zioproto · 2023-05-16T12:22:36Z

This is how I test:

git clone https://github.com/pieroit/cheshire-cat
cd cheshire-cat
git fetch origin pull/195/head:pr195
git checkout pr195
git clean -fdx
cp .env.example .env
docker-compose build --no-cache
docker-compose up

to configure the cat, either from the admin UI, or to be reproducible:

#!/bin/bash
ENDPOINT=https://XXXXXXXX.openai.azure.com/
KEY=SECRETKEY

# When using docker-compose
CORE_URL="http://127.0.0.1"

curl -X 'PUT' \
  ''"$CORE_URL"':1865/settings/llm/LLMAzureChatOpenAIConfig' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "openai_api_type": "azure",
  "openai_api_version": "2023-03-15-preview",
  "deployment_name": "gpt-35-turbo",
  "model_name": "gpt-35-turbo",
  "openai_api_base": "'"$ENDPOINT"'",
  "openai_api_key": "'"$KEY"'"
}'

Can you confirm these steps dont work for you ? Are you doing anything different from this ?

cristianorevil · 2023-05-16T12:27:01Z

i don't have cloned repo in new folder, i have changed the branch to PR, checkout, deleted database and after docker-compose build --no-cache. The cat was configured by ui admin. I will try in complete new folder

zioproto · 2023-05-16T12:47:43Z

Yes please try a new folder. The Docker containers mount the source code form your local folder, where you have probably a lot of cached pyc files. Building the new containers adds little value. You can see them with:

find . -name "*.pyc"

cristianorevil · 2023-05-16T13:46:30Z

@zioproto ok in a new folder all works fine!

zioproto · 2023-05-17T07:13:21Z

@pieroit LGTM, please merge

zioproto commented May 4, 2023

View reviewed changes

core/cat/factory/llm.py Show resolved Hide resolved

zioproto force-pushed the feature/azure-chat-gpt4-config branch from a68dbd6 to 08a5233 Compare May 12, 2023 19:13

zioproto changed the title ~~WiP: Azure, create completions for chat messages with the ChatGPT and GPT-4 models.~~ core: Azure OpenAI support for ChatGPT and GPT-4 chat models May 12, 2023

zioproto commented May 15, 2023

View reviewed changes

zioproto force-pushed the feature/azure-chat-gpt4-config branch from 08a5233 to 2eb7829 Compare May 15, 2023 13:29

zioproto added 2 commits May 15, 2023 23:29

Catch exception if chain run fails

6f88f33

zioproto force-pushed the feature/azure-chat-gpt4-config branch from 2eb7829 to 6f88f33 Compare May 15, 2023 21:30

Use openai_api_version 2023-03-15-preview everywhere

f74d7b3

zioproto mentioned this pull request May 16, 2023

LangChain classes share openai global values langchain-ai/langchain#4775

Closed

14 tasks

pieroit merged commit 90fb00a into cheshire-cat-ai:main May 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core: Azure OpenAI support for ChatGPT and GPT-4 chat models #195

core: Azure OpenAI support for ChatGPT and GPT-4 chat models #195

zioproto commented May 4, 2023 •

edited

Loading

zioproto commented May 12, 2023

cristianorevil commented May 13, 2023

zioproto May 15, 2023

zioproto May 15, 2023

cristianorevil May 15, 2023

zioproto May 15, 2023

zioproto May 15, 2023

zioproto May 15, 2023

zioproto May 15, 2023

cristianorevil May 15, 2023

pieroit commented May 15, 2023

zioproto commented May 15, 2023

pieroit commented May 15, 2023

zioproto commented May 15, 2023

zioproto commented May 15, 2023

zioproto commented May 15, 2023

pieroit commented May 15, 2023 •

edited

Loading

zioproto commented May 15, 2023

zioproto commented May 16, 2023

zioproto commented May 16, 2023

cristianorevil commented May 16, 2023

zioproto commented May 16, 2023

cristianorevil commented May 16, 2023

zioproto commented May 16, 2023

cristianorevil commented May 16, 2023

zioproto commented May 17, 2023

core: Azure OpenAI support for ChatGPT and GPT-4 chat models #195

core: Azure OpenAI support for ChatGPT and GPT-4 chat models #195

Conversation

zioproto commented May 4, 2023 • edited Loading

zioproto commented May 12, 2023

cristianorevil commented May 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Decide host and port for your Cat. Default will be localhost:1865

Decide host and port for your Cat Admin. Default will be localhost:3000

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pieroit commented May 15, 2023

zioproto commented May 15, 2023

pieroit commented May 15, 2023

zioproto commented May 15, 2023

zioproto commented May 15, 2023

zioproto commented May 15, 2023

pieroit commented May 15, 2023 • edited Loading

zioproto commented May 15, 2023

zioproto commented May 16, 2023

zioproto commented May 16, 2023

cristianorevil commented May 16, 2023

zioproto commented May 16, 2023

cristianorevil commented May 16, 2023

zioproto commented May 16, 2023

cristianorevil commented May 16, 2023

zioproto commented May 17, 2023

zioproto commented May 4, 2023 •

edited

Loading

pieroit commented May 15, 2023 •

edited

Loading