Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
cd075fd
[fix] bump prerelease version in pyproject.toml
actions-user Oct 10, 2024
272794e
[fix] bump prerelease version in pyproject.toml
actions-user Oct 10, 2024
e8f8444
[fix] bump prerelease version in pyproject.toml
actions-user Oct 10, 2024
ec89d7d
Fix docker image (#138)
diogoazevedo15 Oct 10, 2024
111c54f
[fix] bump prerelease version in pyproject.toml
actions-user Oct 10, 2024
a555c40
Fix docker image (#139)
diogoazevedo15 Oct 10, 2024
2f0f482
[fix] bump prerelease version in pyproject.toml
actions-user Oct 10, 2024
f06f595
Add llmstudio docs (#136)
diogoazevedo15 Oct 15, 2024
57ea379
[fix] bump prerelease version in pyproject.toml
actions-user Oct 15, 2024
d95c337
docs: update ollama.mdx
eltociear Oct 17, 2024
70d007e
[feat] beadrock added to LLM Core
brunoalho99 Oct 25, 2024
077dfd7
[bugfix] datetime
brunoalho99 Oct 26, 2024
4507de4
[feat] tool_calling
brunoalho99 Oct 26, 2024
b651789
[feat] processing langchain tools
brunoalho99 Oct 27, 2024
2a1d6ab
Merge branch 'main' into feat/bedrock
brunoalho99 Nov 5, 2024
7e6043c
feat: add ci for modules
diogoncalves Nov 4, 2024
6cc7000
chore: bump versions
diogoncalves Nov 6, 2024
4bc5e1c
feat: adapt develop actions
diogoncalves Nov 6, 2024
285e868
fix: github actions
diogoncalves Nov 6, 2024
324c08d
fix: github actions dev
diogoncalves Nov 6, 2024
3ed9d80
[fix] bump prerelease version in pyproject.toml
actions-user Nov 6, 2024
9006b78
[feat] bedrock agent with langchain and langgraph
brunoalho99 Nov 6, 2024
3d65f0a
[chore] remove print
brunoalho99 Nov 6, 2024
fcfe33e
[feat] bedrock mapper
brunoalho99 Nov 8, 2024
48c5113
[chore] clean unused variable
brunoalho99 Nov 8, 2024
49a402d
[feat] bedrock arguments added
brunoalho99 Nov 8, 2024
20e7abe
[feat] bedrock anthropic
brunoalho99 Nov 11, 2024
fd0bb6a
[bugfix] import corrected
brunoalho99 Nov 11, 2024
7bfacd0
[feat] module added
brunoalho99 Nov 11, 2024
7ad9c7e
[bugfix] async methods corrected
brunoalho99 Nov 12, 2024
51790d7
Create chore.yml
MiNeves00 Nov 12, 2024
b324888
Merge branch 'main' into patch-1
MiNeves00 Nov 12, 2024
072981e
CHORE: Create chore.yml (#172)
MiNeves00 Nov 12, 2024
02d2c6f
[feat] async methods corrected
brunoalho99 Nov 14, 2024
aa5f25a
[fastapi] version bump
brunoalho99 Nov 14, 2024
b7c42bb
[bugfix] new release
brunoalho99 Nov 14, 2024
2e537b6
[feat] code refactor
brunoalho99 Nov 14, 2024
1e73f1d
[feat] function calling
brunoalho99 Nov 16, 2024
6e809dd
[chore] remove async duplicated code from openai
brunoalho99 Nov 17, 2024
2a027b0
[chore] remove duplicated code from azure
brunoalho99 Nov 17, 2024
add329f
docs: update ollama.mdx (#159)
MiNeves00 Nov 18, 2024
fdf84e4
[chore] removed unnecessary code
brunoalho99 Nov 18, 2024
bf1b277
[bugfix] processing more than one tool
brunoalho99 Nov 18, 2024
d5c985d
[feat] test added to langchain integration
brunoalho99 Nov 18, 2024
4b037c7
Merge branch 'develop' into feat/bedrock
brunoalho99 Nov 18, 2024
eda7b4e
Merge branch 'feat/bedrock' into feature/vertexai-refactor
brunoalho99 Nov 18, 2024
2448e27
feat: update ci for libs
diogoncalves Nov 18, 2024
5bb4ad0
feat: update dev workflow
diogoncalves Nov 18, 2024
e4807b9
feat: support for logs on bigquery
claramoreirag Nov 19, 2024
103383f
[bugfix] get cost using langchain wrapper
brunoalho99 Nov 19, 2024
b1d3a3e
[bugfix] azure init with kwargs
brunoalho99 Nov 19, 2024
9a0d831
Merge branch 'feat/bedrock' into feature/vertexai-refactor
brunoalho99 Nov 19, 2024
6f8f132
fix: workflow
diogoncalves Nov 19, 2024
519f42e
feat: clean db models definition
claramoreirag Nov 19, 2024
cb22897
docs: add bq example
claramoreirag Nov 19, 2024
1216453
chore: bump versions
diogoncalves Nov 19, 2024
27f113a
chore: fix typo
diogoncalves Nov 19, 2024
ddde4c4
chore: name workflow
diogoncalves Nov 19, 2024
42217fd
Merge branch 'main' into develop
diogoncalves Nov 19, 2024
94a19ae
[chore] solve pr comments
brunoalho99 Nov 19, 2024
b689ceb
[chore] normalize return
brunoalho99 Nov 19, 2024
e577f8b
Providers Refactor (#175)
brunoalho99 Nov 19, 2024
a3df3fb
Merge branch 'feat/bedrock' into develop
diogoncalves Nov 19, 2024
1494bd7
Merge branch 'feat/bedrock' into develop
diogoncalves Nov 19, 2024
c700a00
Merge branch 'main' into develop
diogoncalves Nov 19, 2024
9d3c754
fix: workflow
diogoncalves Nov 19, 2024
d4a4ed1
fix: workflow
diogoncalves Nov 19, 2024
25b5847
Merge branch 'main' into feat/bigquery-tracking
claramoreirag Nov 20, 2024
27e0108
style: linting
claramoreirag Nov 20, 2024
7f19ec5
fix: dev workflow
diogoncalves Nov 20, 2024
13d1ed9
fix: workflow dev
diogoncalves Nov 20, 2024
678c3aa
fix: log_id in int64 for bigquery
claramoreirag Nov 20, 2024
859d911
style: linting
claramoreirag Nov 20, 2024
94cfc4b
fix: workflow dev
diogoncalves Nov 20, 2024
5bb0f6b
fix: dev workflow
diogoncalves Nov 20, 2024
19ce260
[fix] bump prerelease version in pyproject.toml
actions-user Nov 20, 2024
c27bb93
fix: dev workflow
diogoncalves Nov 20, 2024
36ab3b0
[chore] updated python dotenv
brunoalho99 Nov 20, 2024
c257e04
[chore] update dotenv in proxy
brunoalho99 Nov 20, 2024
93da5bc
[fix] bump prerelease version in pyproject.toml
actions-user Nov 21, 2024
3c6be06
feat: main workflow to bump automatically alpha versions
diogoncalves Nov 21, 2024
21074f1
chore: bump versions
diogoncalves Nov 21, 2024
991caed
chore: update lock files
diogoncalves Nov 21, 2024
2f79738
Merge branch 'main' into develop
diogoncalves Nov 21, 2024
e145950
Update pyproject.toml
diogoncalves Nov 21, 2024
e3e7a63
Merge branch 'main' into feat/bigquery-tracking
diogoncalves Nov 21, 2024
a3ac82a
chore: update poetry lock
diogoncalves Nov 21, 2024
b36df02
[fix] bump prerelease version in pyproject.toml
actions-user Nov 21, 2024
fda3eab
Merge branch 'feat/bigquery-tracking' into develop
diogoncalves Nov 21, 2024
4a7b144
[fix] bump prerelease version in pyproject.toml
actions-user Nov 21, 2024
5d2dc90
docs: update tracker readme
claramoreirag Nov 22, 2024
69946ae
style: linting
claramoreirag Nov 22, 2024
8382042
Feat/bigquery tracking (#178)
claramoreirag Nov 27, 2024
b275a77
chore: bump tracker version
claramoreirag Nov 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
294 changes: 294 additions & 0 deletions examples/04_bigquery_integration.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,294 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# BigQuery Integration\n",
"\n",
"This notebook demonstrates how to set up and use the LLMstudio tracking functionalities integrated to BigQuery. \n",
"\n",
"You'll learn:\n",
"1. Authenticate BigQuery\n",
"2. Start a local Tracker server\n",
"3. See the saved logs\n",
"\n",
"First things first:\n",
"* run `pip install llmstudio[tracker]`\n",
"* update your .env file with `GOOGLE_API_KEY` or `OPENAI_API_KEY` \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### BigQuery setup\n",
"\n",
"To use BigQuery, follow these steps:\n",
"\n",
"1. Select or create a Cloud Platform project.\n",
"2. [Create a BigQuery Dataset](https://cloud.google.com/bigquery/docs/datasets)\n",
"2. [Enable the BigQuery Storage API](https://console.cloud.google.com/apis/library/bigquery.googleapis.com).\n",
"3. [Setup Authentication:](https://googleapis.dev/python/google-api-core/latest/auth.html)\n",
" - If you’re running in a Google Virtual Machine Environment (Compute Engine, App Engine, Cloud Run, Cloud Functions), authentication should “just work”.\n",
" - If you’re developing locally, the easiest way to authenticate is using the Google Cloud SDK:\n",
" ```$ gcloud auth application-default login```\n",
" - If you’re running your application elsewhere, you should download a service account JSON keyfile and point to it using an environment variable: \n",
" ```$ export GOOGLE_APPLICATION_CREDENTIALS=\"/path/to/keyfile.json\"```\n",
"\n",
" \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## LLMstudio Tracker setup\n",
"\n",
"For LLMstudio to store your logs in BigQuery you need to set the 'LLMSTUDIO_TRACKING_URI' environment variable with the corresponding uri, which will be in this format: `bigquery://<YOUR-GCP-PROJECT-ID>/<YOUR-BQ-DATASET-ID>`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"os.environ[\"LLMSTUDIO_TRACKING_URI\"] = \"bigquery://<YOUR-GCP-PROJECT-ID>/<YOUR-BQ-DATASET-ID>\""
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from llmstudio.providers import LLM\n",
"from pprint import pprint"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running LLMstudio Tracking on http://0.0.0.0:8002 \n"
]
}
],
"source": [
"from llmstudio.server import start_servers\n",
"start_servers(proxy=False, tracker=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from llmstudio_tracker.tracker import TrackingConfig\n",
"# default port is 50002. set the environment varible to specify which host and port; LLMSTUDIO_TRACKING_HOST, LLMSTUDIO_TRACKING_PORT\n",
"tracker_config = TrackingConfig(host=\"0.0.0.0\", port=\"50002\")\n",
"# You can set OPENAI_API_KEY and ANTHROPIC_API_KEY on .env file\n",
"openai = LLM(\"openai\", tracking_config = tracker_config)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Analyse logs"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"from llmstudio_tracker.tracker import Tracker\n",
"\n",
"tracker = Tracker(tracking_config=tracker_config)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'chat_input': \"Write a paragraph explaining why you're not a cat\",\n",
" 'chat_output': \"I’m not a cat because I lack the physical form and instincts that characterize feline creatures. Unlike a cat, I don’t have a furry coat, retractable claws, or the ability to pounce playfully on a sunbeam. I don’t experience the world through senses like smell, sight, or sound, nor do I possess the whimsical personality traits that make cats so captivating, such as their curiosity and independence. Instead, I am a collection of algorithms and data, designed to process information and generate responses, which allows me to assist you in ways that a cat simply can't—like answering questions, providing explanations, or engaging in conversation.\",\n",
" 'session_id': '20241024-110303-e8b361d9-d5f6-4b73-80f1-6d77a4be3793',\n",
" 'context': [{'role': 'user',\n",
" 'content': \"Write a paragraph explaining why you're not a cat\"}],\n",
" 'provider': 'openai',\n",
" 'model': 'gpt-4o-mini',\n",
" 'deployment': 'gpt-4o-mini-2024-07-18',\n",
" 'parameters': {},\n",
" 'metrics': {'input_tokens': 10,\n",
" 'output_tokens': 132,\n",
" 'total_tokens': 142,\n",
" 'cost_usd': 8.07e-05,\n",
" 'latency_s': 1.7590069770812988,\n",
" 'time_to_first_token_s': 0.450714111328125,\n",
" 'inter_token_latency_s': 0.010057984865628755,\n",
" 'tokens_per_second': 74.47383762932361},\n",
" 'log_id': 56,\n",
" 'created_at': '2024-10-24T11:03:26'}"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"logs = tracker.get_logs()\n",
"logs.json()[-1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Add a session id to tracking logs\n",
"\n",
"* this is especially benefitial if running an app, chatbot agent, etc in production and you need to correlate user feedback, costs etc with user sessions, agent runs, etc"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# default port is 50002. set the environment varible to specify which host and port; LLMSTUDIO_TRACKING_HOST, LLMSTUDIO_TRACKING_PORT\n",
"# You can set OPENAI_API_KEY on .env file\n",
"openai = LLM(\"openai\", tracking_config = tracker_config, session_id=\"openai-session-1\")\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"Tracking AI agents' usage metrics and costs, alongside correlating them with user feedback, is crucial\n",
"\n",
" for optimizing performance and user satisfaction. Usage metrics provide insights into how often and in what ways AI agents\n",
"\n",
" are being utilized, helping to identify patterns, peak usage times, and potential bottlenecks that could\n",
"\n",
" affect service quality. Monitoring costs is equally important to ensure that resources are allocated efficiently, preventing financial waste\n",
"\n",
" while maximizing return on investment. By correlating these metrics with user feedback, developers and stakeholders can gain\n",
"\n",
" a holistic understanding of how the AI agent is performing in real-world settings. This integrated approach enables the\n",
"\n",
" identification of areas for improvement, the fine-tuning of algorithms, and the enhancement of user experience,\n",
"\n",
" ultimately leading to more effective, scalable, and user-friendly AI solutions. Additionally, it allows for the\n",
"\n",
" alignment of AI functionalities with user demands and business objectives, fostering continuous innovation and value creation.\n",
"\n",
"## Metrics:\n",
"{'cost_usd': 0.0027600000000000003,\n",
" 'input_tokens': 21,\n",
" 'inter_token_latency_s': 0.015782629505971844,\n",
" 'latency_s': 3.4153928756713867,\n",
" 'output_tokens': 177,\n",
" 'time_to_first_token_s': 0.6054329872131348,\n",
" 'tokens_per_second': 52.40978315410135,\n",
" 'total_tokens': 198}\n"
]
}
],
"source": [
"response = openai.chat(\"Write a paragraph explaining why it's important to track AI agents usage metrics and costs and correlate with user feedback\", model=\"gpt-4o\", is_stream=True)\n",
"for i, chunk in enumerate(response):\n",
" if i%20==0:\n",
" print(\"\\n\")\n",
" if not chunk.metrics:\n",
" print(chunk.chat_output_stream, end=\"\", flush=True)\n",
" else:\n",
" print(\"\\n\\n## Metrics:\")\n",
" pprint(chunk.metrics)\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'chat_input': \"Write a paragraph explaining why it's important to track AI agents usage metrics and costs and correlate with user feedback\",\n",
" 'chat_output': \"Tracking AI agents' usage metrics and costs, alongside correlating them with user feedback, is crucial for optimizing performance and user satisfaction. Usage metrics provide insights into how often and in what ways AI agents are being utilized, helping to identify patterns, peak usage times, and potential bottlenecks that could affect service quality. Monitoring costs is equally important to ensure that resources are allocated efficiently, preventing financial waste while maximizing return on investment. By correlating these metrics with user feedback, developers and stakeholders can gain a holistic understanding of how the AI agent is performing in real-world settings. This integrated approach enables the identification of areas for improvement, the fine-tuning of algorithms, and the enhancement of user experience, ultimately leading to more effective, scalable, and user-friendly AI solutions. Additionally, it allows for the alignment of AI functionalities with user demands and business objectives, fostering continuous innovation and value creation.\",\n",
" 'session_id': 'openai-session-1',\n",
" 'context': [{'role': 'user',\n",
" 'content': \"Write a paragraph explaining why it's important to track AI agents usage metrics and costs and correlate with user feedback\"}],\n",
" 'provider': 'openai',\n",
" 'model': 'gpt-4o',\n",
" 'deployment': 'gpt-4o-2024-08-06',\n",
" 'parameters': {},\n",
" 'metrics': {'input_tokens': 21,\n",
" 'output_tokens': 177,\n",
" 'total_tokens': 198,\n",
" 'cost_usd': 0.0027600000000000003,\n",
" 'latency_s': 3.4153928756713867,\n",
" 'time_to_first_token_s': 0.6054329872131348,\n",
" 'inter_token_latency_s': 0.015782629505971844,\n",
" 'tokens_per_second': 52.40978315410135},\n",
" 'log_id': 57,\n",
" 'created_at': '2024-10-24T11:04:21'}"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"logs = tracker.get_session_logs(session_id=\"openai-session-1\")\n",
"logs.json()[-1]"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading