OpenAI async support #15
Comments
|
Hey @bcordo , Thanks for raising this issue. I have added support for from openai import OpenAI
import asyncio
from grafana_openai_monitoring import chat_v2
client = OpenAI(
api_key="sk-***",
)
# Apply the custom decorator to the OpenAI API function
client.chat.completions.create = chat_v2.monitor(
client.chat.completions.create,
metrics_url="https://prometheus.grafana.net/api/prom", # Example: "https://prometheus.grafana.net/api/prom"
logs_url="https://logs.grafana.net/loki/api/v1/push", # Example: "https://logs.example.com/loki/api/v1/push/"
metrics_username=123456, # Example: "123456"
logs_username=987654, # Example: "987654"
access_token="glc_ey....",
use_async=True, # Set to True if the function is async
)
async def main() -> None:
chat_completion = await client.chat.completions.create(
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
model="gpt-3.5-turbo",
)
print(chat_completion)
asyncio.run(main())Please feel free to reopen the issue if you still encounter issues |
|
Wow @ishanjainn that was super fast! I appreciate it, you're awesome. Thanks. |
|
Hi @ishanjainn. Thanks for your fast response. I ran your sample code: import os
from openai import OpenAI
import asyncio
from grafana_openai_monitoring import chat_v2
from dotenv import load_dotenv, find_dotenv
load_dotenv()
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
client = OpenAI(
api_key=OPENAI_API_KEY,
)
load_dotenv()
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
DEEPL_AUTH_KEY = os.getenv("DEEPL_AUTH_KEY")
GRAFANA_API_KEY = os.getenv("GRAFANA_API_KEY")
GRAFANA_METRICS_URL = os.getenv("GRAFANA_METRICS_URL")
GRAFANA_LOGS_URL = os.getenv("GRAFANA_LOGS_URL")
GRAFANA_METRICS_USERNAME = os.getenv("GRAFANA_METRICS_USERNAME")
GRAFANA_LOGS_USERNAME = os.getenv("GRAFANA_LOGS_USERNAME")
client = OpenAI(api_key=OPENAI_API_KEY)
# Apply the custom decorator to the OpenAI API client functions to measure usage
client.chat.completions.create = chat_v2.monitor(
client.chat.completions.create,
metrics_url=GRAFANA_METRICS_URL,
logs_url=GRAFANA_LOGS_URL,
metrics_username=GRAFANA_METRICS_USERNAME,
logs_username=GRAFANA_LOGS_USERNAME,
access_token=GRAFANA_API_KEY,
)
async def main() -> None:
chat_completion = await client.chat.completions.create(
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
model="gpt-3.5-turbo",
)
print(chat_completion)
asyncio.run(main())But I get the error: I thought there may have been a typo so I reran using client = AsyncOpenAI(
api_key=OPENAI_API_KEY,
)And I get the error: And just to make sure I was using the updated version, which seems right: Maybe I am just missing something here. |
|
This seems to missing |
|
@ishanjainn Thanks again for the quick response. Yes, that worked thanks, the problem was I added import os
import asyncio
from openai import OpenAI
from grafana_openai_monitoring import chat_v2
from dotenv import load_dotenv
load_dotenv()
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
load_dotenv()
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
DEEPL_AUTH_KEY = os.getenv("DEEPL_AUTH_KEY")
GRAFANA_API_KEY = os.getenv("GRAFANA_API_KEY")
GRAFANA_METRICS_URL = os.getenv("GRAFANA_METRICS_URL")
GRAFANA_LOGS_URL = os.getenv("GRAFANA_LOGS_URL")
GRAFANA_METRICS_USERNAME = os.getenv("GRAFANA_METRICS_USERNAME")
GRAFANA_LOGS_USERNAME = os.getenv("GRAFANA_LOGS_USERNAME")
client = OpenAI(api_key=OPENAI_API_KEY)
# Apply the custom decorator to the OpenAI API client functions to measure usage
client.chat.completions.create = chat_v2.monitor(
client.chat.completions.create,
metrics_url=GRAFANA_METRICS_URL,
logs_url=GRAFANA_LOGS_URL,
metrics_username=GRAFANA_METRICS_USERNAME,
logs_username=GRAFANA_LOGS_USERNAME,
access_token=GRAFANA_API_KEY,
use_async=True,
)
async def main() -> None:
chat_completion = await client.chat.completions.create(
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
model="gpt-3.5-turbo",
stream=True,
)
for chunk in chat_completion:
current_content = chunk.choices[0].delta.content
print(current_content)
asyncio.run(main())I guess the problem as reported here How_to_stream_completions.ipynb
Any good solutions to this? Thanks again. |
|
Hey @bcordo , For the streaming responses, More than token usage, The library doesnt yet supports streaming as it needs a bit different processing, We have it as an open issue right now #13. For token calculation when streaming, yeah |
|
Also reopened this issue while we get streaming implemented here. Thanks! |
@ishanjainn
What happened?
I'm following this guide: https://grafana.com/blog/2023/11/02/monitor-your-openai-usage-with-grafana-cloud/ and it works great with normal synchronous OpenAI client "from openai import OpenAI; client = OpenAI(apikey=OPENAIAPIKEY)" but it doesn't work with the asynchronous client "from openai import AsyncOpenAI; aclient = AsyncOpenAI(apikey=OPENAIAPIKEY)"
I get the error:
"Exception has occurred: AttributeError
'coroutine' object has no attribute 'usage'
File ..., in asynccall_chatgpt
response = await aclient.chat.completions.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1, in
AttributeError: 'coroutine' object has no attribute 'usage'"
It has to do with the fact that internally it's not using await and so it's not expecting a coroutine.
What was expected to happen?
I was expecting it to also work, or to have access to an async version of the monitoring code.
Steps to reproduce the problem:
from openai import AsyncOpenAI
aclient = AsyncOpenAI(apikey=OPENAIAPIKEY) aclient.chat.completions.create = chatv2.monitor(
aclient.chat.completions.create,
metricsurl="[redacted]",
metricsusername=[redacted],
logsusername=[redacted],
access_token="[redacted]"
response = await aclient.chat.completions.create(
model=model,
temperature=temperature,
maxtokens=maxtokens,
n=maxresponses, topp=topp, frequencypenalty=frequencypenalty, presencepenalty=presence_penalty,
messages=messages,
stream=stream,
)
Exception has occurred: AttributeError
'coroutine' object has no attribute 'usage'
File ... in asynccall_chatgpt
response = await aclient.chat.completions.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1, in
AttributeError: 'coroutine' object has no attribute 'usage'
Version numbers (grafana, prometheus, graphite, plugins, operating system, etc.):
Mac, grafana cloud, OpenAI plugin
The text was updated successfully, but these errors were encountered: