Skip to content

feat(ollama): support LAN GPU server #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
developersdigest merged 3 commits into from
Apr 1, 2024
Merged

feat(ollama): support LAN GPU server #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
388e20e
add langchain/openai dependency
QIN2DIM Mar 29, 2024
78b9880
Merge branch 'developersdigest:main' into main
QIN2DIM Apr 1, 2024
5e281ac
feat(ollama): support LAN GPU server
QIN2DIM Apr 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,9 @@ GROQ_API_KEY=APIKEYGOESHERE
# https://brave.com/search/api/
BRAVE_SEARCH_API_KEY=APIKEYGOESHERE
# https://serper.dev/
SERPER_API=APIKEYGOESHERE
SERPER_API=APIKEYGOESHERE

# Set LAN GPU server, examples:
# PC | http://localhost:11434/v1
# LAN GPU server | http://192.168.1.100:11434/v1
OLLAMA_BASE_URL=http://localhost:11434/v1
3 changes: 2 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,5 @@ node_modules
.cache
.turbo
.vercel
.vscode
.vscode
.idea
2 changes: 1 addition & 1 deletion app/action.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import { config } from './config';
let openai: OpenAI;
if (config.useOllamaInference) {
openai = new OpenAI({
baseURL: 'http://localhost:11434/v1',
baseURL: config.ollamaBaseUrl,
apiKey: 'ollama'
});
} else {
Expand Down
7 changes: 5 additions & 2 deletions app/config.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
// - IMPORTANT: Follow-up questions are not yet implrmented with Ollama models, only OpenAI compatible models that use {type: "json_object"}

export const config = {
useOllamaInference: false,
useOllamaEmbeddings: false,
useOllamaInference: false,
useOllamaEmbeddings: false,
inferenceModel: 'mixtral-8x7b-32768', // Groq: 'mixtral-8x7b-32768', 'gemma-7b-it' // OpenAI: 'gpt-3.5-turbo', 'gpt-4' // Ollama 'mistral', 'llama2' etc
inferenceAPIKey: process.env.GROQ_API_KEY, // Groq: process.env.GROQ_API_KEY // OpenAI: process.env.OPENAI_API_KEY // Ollama: 'ollama' is the default
embeddingsModel: 'text-embedding-3-small', // Ollama: 'llama2', 'nomic-embed-text' // OpenAI 'text-embedding-3-small', 'text-embedding-3-large'
Expand All @@ -15,4 +15,7 @@ export const config = {
numberOfSimilarityResults: 4, // Numbher of similarity results to return per page
numberOfPagesToScan: 10, // Recommended to decrease for Ollama
nonOllamaBaseURL: 'https://api.groq.com/openai/v1', //Groq: https://api.groq.com/openai/v1 // OpenAI: https://api.openai.com/v1

// Set LAN GPU server, example: http://192.168.1.100:11434/v1
ollamaBaseUrl: process.env.OLLAMA_BASE_URL || 'http://localhost:11434/v1',
};
1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
},
"dependencies": {
"@langchain/community": "^0.0.40",
"@langchain/openai": "^0.0.25",
"@radix-ui/react-icons": "^1.3.0",
"@radix-ui/react-label": "^2.0.2",
"@radix-ui/react-separator": "^1.0.3",
Expand Down