Skip to content

Commit 7bb35b0

Browse files
jannikmaierhoeferclemramarcklingen
authored
docs: add ollama integration and some misc edits (langfuse#774)
 --------- Co-authored-by: Clemo <121163007+clemra@users.noreply.github.com> Co-authored-by: Marc Klingen <git@marcklingen.com>
1 parent ed0f979 commit 7bb35b0

18 files changed

+1115
-204
lines changed

‎cookbook/_routes.json

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,5 +114,9 @@
114114
{
115115
"notebook": "integration_mirascope.ipynb",
116116
"docsPath": "docs/integrations/mirascope/example-python"
117+
},
118+
{
119+
"notebook": "integration_ollama.ipynb",
120+
"docsPath": "docs/integrations/ollama"
117121
}
118122
]

‎cookbook/integration_ollama.ipynb

Lines changed: 349 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,349 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"---\n",
8+
"title: Observe local LLMs with Ollama and Langfuse\n",
9+
"description: Learn how to run Open Source LLMs locally on your machine using Ollama and trace the outputs with Langfuse. \n",
10+
"category: Integrations\n",
11+
"---"
12+
]
13+
},
14+
{
15+
"cell_type": "markdown",
16+
"metadata": {},
17+
"source": [
18+
"# Trace your local Ollama model with Langfuse\n",
19+
"\n",
20+
"In this cookbook, we will show you how to trace local language models with Ollama and Langfuse.\n",
21+
"\n",
22+
"**Note: We'll use the Langfuse OpenAI SDK integration for Python in this example. This works the same for [JS/TS](https://langfuse.com/docs/integrations/openai/js/get-started) or via the Langfuse integrations with [LangChain](https://langfuse.com/docs/integrations/langchain/tracing) and [LlamaIndex](https://langfuse.com/docs/integrations/llama-index/get-started).**\n",
23+
"\n",
24+
"## What is Ollama?\n",
25+
"\n",
26+
"Ollama ([GitHub](https://github.com/ollama/ollama)) is an open-source platform that allows you to run large language models (LLMs) locally on your machine, supporting a variety of models including [Llama 3.1](https://ollama.com/library/llama3.1) and [Mistral 7B](https://ollama.com/library/mistral). It optimizes setup and configuration by bundling model weights, configuration, and data into a single package defined by a Modelfile.\n",
27+
"\n",
28+
"## What is Langfuse?\n",
29+
"\n",
30+
"Langfuse ([GitHub](https://github.com/langfuse/langfuse)) is an open-source LLM engineering platform that helps teams collaboratively debug, analyze, and iterate on their LLM applications via tracing, prompt management and evaluations.\n",
31+
"\n",
32+
"### Local Deployment of Langfuse\n",
33+
"\n",
34+
"Of course, you can also locally deploy Langfuse to run models and trace LLM outputs only on your own device. [Here](https://langfuse.com/docs/deployment/local) is a guide on how to run Langfuse on your local machine using Docker Compose. This method is ideal for testing Langfuse and troubleshooting integration issues.\n",
35+
"\n",
36+
"For this example, we will use the Langfuse cloud version."
37+
]
38+
},
39+
{
40+
"cell_type": "markdown",
41+
"metadata": {},
42+
"source": [
43+
"## Example 1: Llama 3.1 Model\n",
44+
"\n",
45+
"In this example, we will use the Llama 3.1 model to create a simple chat completions application using the OpenAI Python SDK and Langfuse tracing."
46+
]
47+
},
48+
{
49+
"cell_type": "markdown",
50+
"metadata": {},
51+
"source": [
52+
"### **Step 1:** Setup Ollama\n",
53+
"\n",
54+
"Start by [downloading Ollama](https://ollama.com/download) and pull the [Llama 3.1](https://ollama.com/library/llama3.1) model. See the [Ollama documentation](https://github.com/ollama/ollama/tree/main/docs) for further information.\n",
55+
"\n",
56+
"```bash\n",
57+
"ollama pull llama3.1\n",
58+
"```"
59+
]
60+
},
61+
{
62+
"cell_type": "markdown",
63+
"metadata": {},
64+
"source": [
65+
"To invoke Ollama’s OpenAI compatible API endpoint, use the same [OpenAI format](https://platform.openai.com/docs/quickstart?context=curl) and change the hostname to `http://localhost:11434`:\n",
66+
"\n",
67+
"```bash\n",
68+
"curl http://localhost:11434/v1/chat/completions \\\n",
69+
" -H \"Content-Type: application/json\" \\\n",
70+
" -d '{\n",
71+
" \"model\": \"llama3.1\",\n",
72+
" \"messages\": [\n",
73+
" {\n",
74+
" \"role\": \"system\",\n",
75+
" \"content\": \"You are a helpful assistant.\"\n",
76+
" },\n",
77+
" {\n",
78+
" \"role\": \"user\",\n",
79+
" \"content\": \"Hello!\"\n",
80+
" }\n",
81+
" ]\n",
82+
" }'\n",
83+
"```"
84+
]
85+
},
86+
{
87+
"cell_type": "markdown",
88+
"metadata": {},
89+
"source": [
90+
"### **Step 2:** Setup Langfuse\n",
91+
"\n",
92+
"Initialize the Langfuse client with your [API keys](https://langfuse.com/faq/all/where-are-langfuse-api-keys) from the project settings in the Langfuse UI and add them to your environment."
93+
]
94+
},
95+
{
96+
"cell_type": "code",
97+
"execution_count": 9,
98+
"metadata": {},
99+
"outputs": [],
100+
"source": [
101+
"import os\n",
102+
"\n",
103+
"# Get keys for your project from the project settings page\n",
104+
"# https://cloud.langfuse.com\n",
105+
"os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"\"\n",
106+
"os.environ[\"LANGFUSE_SECRET_KEY\"] = \"\"\n",
107+
"os.environ[\"LANGFUSE_HOST\"] = \"https://cloud.langfuse.com\" # 🇪🇺 EU region\n",
108+
"# os.environ[\"LANGFUSE_HOST\"] = \"https://us.cloud.langfuse.com\" # 🇺🇸 US region"
109+
]
110+
},
111+
{
112+
"cell_type": "code",
113+
"execution_count": null,
114+
"metadata": {},
115+
"outputs": [],
116+
"source": [
117+
"%pip install langfuse openai --upgrade"
118+
]
119+
},
120+
{
121+
"cell_type": "markdown",
122+
"metadata": {},
123+
"source": [
124+
"### **Step 3:** Use the OpenAI Python SDK to call the Llama3.1 Model"
125+
]
126+
},
127+
{
128+
"cell_type": "markdown",
129+
"metadata": {},
130+
"source": [
131+
"To use the Ollama model, we use the OpenAI Python SDK as Ollama has the same API (see above). To trace your LLM calls in Langfuse you can use the **drop-in replacement** ([docs](https://langfuse.com/docs/integrations/openai/python/get-started), this also works for JS/TS and via LangChain and LlamaIndex) to get full logging by changing only the import.\n",
132+
"\n",
133+
"```diff\n",
134+
"- import openai\n",
135+
"+ from langfuse.openai import openai\n",
136+
" \n",
137+
"Alternative imports:\n",
138+
"+ from langfuse.openai import OpenAI, AsyncOpenAI, AzureOpenAI, AsyncAzureOpenAI\n",
139+
"```"
140+
]
141+
},
142+
{
143+
"cell_type": "code",
144+
"execution_count": 5,
145+
"metadata": {},
146+
"outputs": [
147+
{
148+
"name": "stdout",
149+
"output_type": "stream",
150+
"text": [
151+
"A famous moment in history! When Neil Armstrong took his historic first steps on the moon, his first words were: \"That's one small step for man, one giant leap for mankind.\" (Note: The word was actually \"man\", not \"men\" - it's often been reported as \"one small step for men\", but Armstrong himself said he meant to say \"man\")\n"
152+
]
153+
}
154+
],
155+
"source": [
156+
"# Drop-in replacement to get full logging by changing only the import\n",
157+
"from langfuse.openai import OpenAI\n",
158+
"\n",
159+
"# Configure the OpenAI client to use http://localhost:11434/v1 as base url \n",
160+
"client = OpenAI(\n",
161+
" base_url = 'http://localhost:11434/v1',\n",
162+
" api_key='ollama', # required, but unused\n",
163+
")\n",
164+
"\n",
165+
"response = client.chat.completions.create(\n",
166+
" model=\"llama3.1\",\n",
167+
" messages=[\n",
168+
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
169+
" {\"role\": \"user\", \"content\": \"Who was the first person to step on the moon?\"},\n",
170+
" {\"role\": \"assistant\", \"content\": \"Neil Armstrong was the first person to step on the moon on July 20, 1969, during the Apollo 11 mission.\"},\n",
171+
" {\"role\": \"user\", \"content\": \"What were his first words when he stepped on the moon?\"}\n",
172+
" ]\n",
173+
")\n",
174+
"print(response.choices[0].message.content)"
175+
]
176+
},
177+
{
178+
"cell_type": "markdown",
179+
"metadata": {},
180+
"source": [
181+
"### **Step 4:** See Traces in Langfuse \n",
182+
"\n",
183+
"[Example Trace in the Langfuse UI](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/6ad58e47-3bff-4287-9a96-af85d2627ea4)\n",
184+
"\n",
185+
"![View example trace in the Langfuse UI](https://langfuse.com/images/cookbook/integration-ollama/integration-ollama-llama-trace.png)"
186+
]
187+
},
188+
{
189+
"cell_type": "markdown",
190+
"metadata": {},
191+
"source": [
192+
"## Example 2: Mistral 7B Model\n",
193+
"\n",
194+
"In this example, we will use the Mistral 7B model to create a simple chat completions application using the OpenAI Python SDK and Langfuse tracing."
195+
]
196+
},
197+
{
198+
"cell_type": "markdown",
199+
"metadata": {},
200+
"source": [
201+
"### **Step 1:** Setup Ollama\n",
202+
"\n",
203+
"Start by [downloading Ollama](https://ollama.com/download) and pulling the [Mistral 7B](https://ollama.com/library/mistral) model:\n",
204+
"\n",
205+
"```bash\n",
206+
"ollama pull mistral\n",
207+
"\n",
208+
"```"
209+
]
210+
},
211+
{
212+
"cell_type": "markdown",
213+
"metadata": {},
214+
"source": [
215+
"To invoke Ollama’s OpenAI compatible API endpoint, use the same [OpenAI format](https://platform.openai.com/docs/quickstart?context=curl) and change the hostname to http://localhost:11434:\n",
216+
"\n",
217+
"```bash\n",
218+
"curl http://localhost:11434/v1/chat/completions \\\n",
219+
" -H \"Content-Type: application/json\" \\\n",
220+
" -d '{\n",
221+
" \"model\": \"mistral\",\n",
222+
" \"messages\": [\n",
223+
" {\n",
224+
" \"role\": \"system\",\n",
225+
" \"content\": \"You are a helpful assistant.\"\n",
226+
" },\n",
227+
" {\n",
228+
" \"role\": \"user\",\n",
229+
" \"content\": \"Hello!\"\n",
230+
" }\n",
231+
" ]\n",
232+
" }'\n",
233+
"\n",
234+
"```"
235+
]
236+
},
237+
{
238+
"cell_type": "markdown",
239+
"metadata": {},
240+
"source": [
241+
"### **Step 2:** Setup Langfuse\n",
242+
"\n",
243+
"Initialize the Langfuse client with your [API keys](https://langfuse.com/faq/all/where-are-langfuse-api-keys) from the project settings in the Langfuse UI and add them to your environment."
244+
]
245+
},
246+
{
247+
"cell_type": "code",
248+
"execution_count": 6,
249+
"metadata": {},
250+
"outputs": [],
251+
"source": [
252+
"import os\n",
253+
"\n",
254+
"# Get keys for your project from the project settings page\n",
255+
"# https://cloud.langfuse.com\n",
256+
"os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"\"\n",
257+
"os.environ[\"LANGFUSE_SECRET_KEY\"] = \"\"\n",
258+
"os.environ[\"LANGFUSE_HOST\"] = \"https://cloud.langfuse.com\" # 🇪🇺 EU region\n",
259+
"# os.environ[\"LANGFUSE_HOST\"] = \"https://us.cloud.langfuse.com\" # 🇺🇸 US region\n",
260+
"\n",
261+
"# Your openai key\n",
262+
"os.environ[\"OPENAI_API_KEY\"] = \"\""
263+
]
264+
},
265+
{
266+
"cell_type": "markdown",
267+
"metadata": {},
268+
"source": [
269+
"### **Step 3:** Use the OpenAI Python SDK to call the Mistral Model"
270+
]
271+
},
272+
{
273+
"cell_type": "code",
274+
"execution_count": 7,
275+
"metadata": {},
276+
"outputs": [
277+
{
278+
"name": "stdout",
279+
"output_type": "stream",
280+
"text": [
281+
" The most recently confirmed element is oganesson (Og), with symbol Og and atomic number 118. It was officially recognized by IUPAC (International Union of Pure and Applied Chemistry) in 2016, following the synthesis of several atoms at laboratories in Russia and Germany. The latest unofficially-recognized element is ununsextium (Uus), with atomic number 138. However, its synthesis is still under investigation, and IUPAC has yet to officially confirm its existence.\n"
282+
]
283+
}
284+
],
285+
"source": [
286+
"# Drop-in replacement to get full logging by changing only the import\n",
287+
"from langfuse.openai import OpenAI\n",
288+
"\n",
289+
"# Configure the OpenAI client to use http://localhost:11434/v1 as base url \n",
290+
"client = OpenAI(\n",
291+
" base_url = 'http://localhost:11434/v1',\n",
292+
" api_key='ollama', # required, but unused\n",
293+
")\n",
294+
"\n",
295+
"response = client.chat.completions.create(\n",
296+
" model=\"mistral\",\n",
297+
" messages=[\n",
298+
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
299+
" {\"role\": \"user\", \"content\": \"How many elements are there in the periodic table?\"},\n",
300+
" {\"role\": \"assistant\", \"content\": \"There are 118 elements in the periodic table.\"},\n",
301+
" {\"role\": \"user\", \"content\": \"Which element was discovered most recently?\"}\n",
302+
" ]\n",
303+
")\n",
304+
"print(response.choices[0].message.content)"
305+
]
306+
},
307+
{
308+
"cell_type": "markdown",
309+
"metadata": {},
310+
"source": [
311+
"### **Step 4:** See Traces in Langfuse \n",
312+
"\n",
313+
"[Example Trace in the Langfuse UI](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/85693874-9ddb-4fd4-a386-0031933cb784)\n",
314+
"\n",
315+
"![View example trace in the Langfuse UI](https://langfuse.com/images/cookbook/integration-ollama/integration-ollama-mistral-trace.png)"
316+
]
317+
},
318+
{
319+
"cell_type": "markdown",
320+
"metadata": {},
321+
"source": [
322+
"## Feedback\n",
323+
"\n",
324+
"If you have any feedback or requests, please create a GitHub [Issue](https://langfuse.com/issue) or share your idea with the community on [Discord](https://discord.langfuse.com/)."
325+
]
326+
}
327+
],
328+
"metadata": {
329+
"kernelspec": {
330+
"display_name": "Python 3",
331+
"language": "python",
332+
"name": "python3"
333+
},
334+
"language_info": {
335+
"codemirror_mode": {
336+
"name": "ipython",
337+
"version": 3
338+
},
339+
"file_extension": ".py",
340+
"mimetype": "text/x-python",
341+
"name": "python",
342+
"nbconvert_exporter": "python",
343+
"pygments_lexer": "ipython3",
344+
"version": "3.9.18"
345+
}
346+
},
347+
"nbformat": 4,
348+
"nbformat_minor": 2
349+
}

0 commit comments

Comments
 (0)