Skip to content

Commit d771356

Browse files
edenreichedenreich
authored
docs: Add more examples how to use this SDK (#15)
* docs(examples): Add example environment configuration and README files for SDK usage Signed-off-by: Eden Reich <eden.reich@gmail.com> * docs: Update README to include pre-requisites for using the Typescript SDK examples Signed-off-by: Eden Reich <eden.reich@gmail.com> * docs(examples): Update example links in README and add list example README Signed-off-by: Eden Reich <eden.reich@gmail.com> * docs(examples): Add examples for chat, list, and MCP using Inference Gateway SDK - Created a chat example with package.json, tsconfig.json, and index.ts to demonstrate chat completions and streaming responses. - Added a list example with package.json, tsconfig.json, and index.ts for listing models and MCP tools. - Included README.md files for both chat and list examples with instructions for getting started. - Added .gitignore files to exclude node_modules in both list and MCP examples. Signed-off-by: Eden Reich <eden.reich@gmail.com> * docs(exmaples): Add sample files and configuration for MCP example - Created a new directory with sample files for the MCP filesystem server, including `sample.txt`, `data.json`, and `config.yaml`. - Added a README.md file to describe the purpose and usage of the sample data. - Implemented a TypeScript configuration file (`tsconfig.json`) for the MCP example. - Created a package.json and package-lock.json for managing dependencies in the examples directory. - Updated the OpenAPI specification to include new endpoints and improved formatting for consistency. Signed-off-by: Eden Reich <eden.reich@gmail.com> * docs(examples): Update LLM model references to remove provider prefix Signed-off-by: Eden Reich <eden.reich@gmail.com> * docs(mcp): Add note to ensure no containers are running before starting the example Signed-off-by: Eden Reich <eden.reich@gmail.com> * feat: Support both local tools and remote MCP tools; Add onMCPTool callback - Created a new MCP Web Search Server with a package.json file. - Updated the main MCP package.json to include example scripts for remote and local tools. - Added a .gitignore file in the shared directory to exclude unnecessary files. - Removed outdated README.md, config.yaml, data.json, and sample.txt files from the shared directory. - Introduced a new sample_sales_data.csv file for demonstration purposes. - Enhanced the client.ts to support MCP tool callbacks and improved streaming response processing. Signed-off-by: Eden Reich <eden.reich@gmail.com> * docs(examples): Refactor MCP examples and client for improved error handling and token tracking - Updated docker-compose.yml to set environment to production and commented out optional Ollama service. - Enhanced example-mcp-tools.ts to use console.info for logging and added model availability checks. - Modified index.ts to implement token tracking for prompt and completion tokens, displaying usage metrics at the end. - Improved filesystem MCP server (index.js) to use console.info for logging and ensure consistent messaging. - Updated web-search MCP server (index-http.js and index.js) to use console.info for logging and improved request handling. - Enhanced InferenceGatewayClient to handle mid-stream errors and added type definitions for stream chunks that may contain errors. Signed-off-by: Eden Reich <eden.reich@gmail.com> --------- Signed-off-by: Eden Reich <eden.reich@gmail.com>
1 parent 177e9f3 commit d771356

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+9839
-495
lines changed

‎.gitattributes‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,5 @@
11
.devcontainer/** linguist-vendored=true
2+
src/examples/** linguist-vendored=true
3+
24
src/types/generated/** linguist-generated=true
5+

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
node_modules
22
dist
33
coverage
4+
**/.env

‎README.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,14 @@ An SDK written in TypeScript for the [Inference Gateway](https://github.com/eden
77
- [Usage](#usage)
88
- [Creating a Client](#creating-a-client)
99
- [Listing Models](#listing-models)
10+
- [Listing MCP Tools](#listing-mcp-tools)
1011
- [Creating Chat Completions](#creating-chat-completions)
1112
- [Streaming Chat Completions](#streaming-chat-completions)
1213
- [Tool Calls](#tool-calls)
1314
- [Proxying Requests](#proxying-requests)
1415
- [Health Check](#health-check)
1516
- [Creating a Client with Custom Options](#creating-a-client-with-custom-options)
17+
- [Examples](#examples)
1618
- [Contributing](#contributing)
1719
- [License](#license)
1820

@@ -51,7 +53,7 @@ try {
5153
console.log('All models:', models);
5254

5355
// List models from a specific provider
54-
const openaiModels = await client.listModels(Provider.OpenAI);
56+
const openaiModels = await client.listModels(Provider.openai);
5557
console.log('OpenAI models:', openaiModels);
5658
} catch (error) {
5759
console.error('Error:', error);
@@ -235,7 +237,7 @@ To proxy requests directly to a provider:
235237
import { InferenceGatewayClient, Provider } from '@inference-gateway/sdk';
236238

237239
const client = new InferenceGatewayClient({
238-
baseURL: 'http://localhost:8080/v1',
240+
baseURL: 'http://localhost:8080',
239241
});
240242

241243
try {
@@ -261,7 +263,7 @@ To check if the Inference Gateway is running:
261263
import { InferenceGatewayClient } from '@inference-gateway/sdk';
262264

263265
const client = new InferenceGatewayClient({
264-
baseURL: 'http://localhost:8080/v1',
266+
baseURL: 'http://localhost:8080',
265267
});
266268

267269
try {
@@ -292,6 +294,10 @@ const clientWithHeaders = client.withOptions({
292294
});
293295
```
294296

297+
### Examples
298+
299+
For more examples, check the [examples directory](./examples).
300+
295301
## Contributing
296302

297303
Please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file for information about how to get involved. We welcome issues, questions, and pull requests.

‎eslint.config.mjs‎

Lines changed: 77 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,26 @@ export default [
1010
plugins: {
1111
prettier: prettier,
1212
},
13+
languageOptions: {
14+
globals: {
15+
console: 'readonly',
16+
process: 'readonly',
17+
Buffer: 'readonly',
18+
__dirname: 'readonly',
19+
__filename: 'readonly',
20+
global: 'readonly',
21+
module: 'readonly',
22+
require: 'readonly',
23+
exports: 'readonly',
24+
},
25+
},
1326
rules: {
1427
'prettier/prettier': 'error',
1528
...eslint.configs.recommended.rules,
1629
},
1730
},
1831
{
19-
files: ['**/*.ts'],
32+
files: ['src/**/*.ts', 'tests/**/*.ts'],
2033
languageOptions: {
2134
parser: tsParser,
2235
parserOptions: {
@@ -25,12 +38,28 @@ export default [
2538
sourceType: 'module',
2639
},
2740
globals: {
28-
// Add Jest globals
41+
console: 'readonly',
42+
process: 'readonly',
43+
Buffer: 'readonly',
44+
__dirname: 'readonly',
45+
__filename: 'readonly',
46+
global: 'readonly',
47+
module: 'readonly',
48+
require: 'readonly',
49+
exports: 'readonly',
2950
describe: 'readonly',
3051
it: 'readonly',
3152
expect: 'readonly',
3253
beforeEach: 'readonly',
54+
afterEach: 'readonly',
55+
beforeAll: 'readonly',
56+
afterAll: 'readonly',
3357
jest: 'readonly',
58+
Headers: 'readonly',
59+
fetch: 'readonly',
60+
ReadableStream: 'readonly',
61+
Response: 'readonly',
62+
Request: 'readonly',
3463
},
3564
},
3665
plugins: {
@@ -45,10 +74,54 @@ export default [
4574
...tseslint.configs.recommended.rules,
4675
},
4776
},
77+
{
78+
files: ['examples/**/*.ts'],
79+
languageOptions: {
80+
parser: tsParser,
81+
parserOptions: {
82+
ecmaVersion: 'latest',
83+
sourceType: 'module',
84+
},
85+
globals: {
86+
console: 'readonly',
87+
process: 'readonly',
88+
Buffer: 'readonly',
89+
__dirname: 'readonly',
90+
__filename: 'readonly',
91+
global: 'readonly',
92+
module: 'readonly',
93+
require: 'readonly',
94+
exports: 'readonly',
95+
},
96+
},
97+
plugins: {
98+
'@typescript-eslint': tseslint,
99+
},
100+
rules: {
101+
'@typescript-eslint/no-unused-vars': [
102+
'error',
103+
{ argsIgnorePattern: '^_' },
104+
],
105+
},
106+
},
48107
{
49108
files: ['**/*.test.ts'],
50-
env: {
51-
'jest/globals': true,
109+
languageOptions: {
110+
globals: {
111+
describe: 'readonly',
112+
it: 'readonly',
113+
expect: 'readonly',
114+
beforeEach: 'readonly',
115+
afterEach: 'readonly',
116+
beforeAll: 'readonly',
117+
afterAll: 'readonly',
118+
jest: 'readonly',
119+
Headers: 'readonly',
120+
fetch: 'readonly',
121+
ReadableStream: 'readonly',
122+
Response: 'readonly',
123+
Request: 'readonly',
124+
},
52125
},
53126
},
54127
];

‎examples/.env.example‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
2+
# General settings
3+
ENVIRONMENT=development
4+
ENABLE_TELEMETRY=false
5+
ENABLE_AUTH=false
6+
# Model Context Protocol (MCP)
7+
MCP_ENABLE=false
8+
MCP_EXPOSE=false
9+
MCP_SERVERS=
10+
MCP_CLIENT_TIMEOUT=5s
11+
MCP_DIAL_TIMEOUT=3s
12+
MCP_TLS_HANDSHAKE_TIMEOUT=3s
13+
MCP_RESPONSE_HEADER_TIMEOUT=3s
14+
MCP_EXPECT_CONTINUE_TIMEOUT=1s
15+
MCP_REQUEST_TIMEOUT=5s
16+
# OpenID Connect
17+
OIDC_ISSUER_URL=http://keycloak:8080/realms/inference-gateway-realm
18+
OIDC_CLIENT_ID=inference-gateway-client
19+
OIDC_CLIENT_SECRET=
20+
# Server settings
21+
SERVER_HOST=0.0.0.0
22+
SERVER_PORT=8080
23+
SERVER_READ_TIMEOUT=30s
24+
SERVER_WRITE_TIMEOUT=30s
25+
SERVER_IDLE_TIMEOUT=120s
26+
SERVER_TLS_CERT_PATH=
27+
SERVER_TLS_KEY_PATH=
28+
# Client settings
29+
CLIENT_TIMEOUT=30s
30+
CLIENT_MAX_IDLE_CONNS=20
31+
CLIENT_MAX_IDLE_CONNS_PER_HOST=20
32+
CLIENT_IDLE_CONN_TIMEOUT=30s
33+
CLIENT_TLS_MIN_VERSION=TLS12
34+
# Providers
35+
ANTHROPIC_API_URL=https://api.anthropic.com/v1
36+
ANTHROPIC_API_KEY=
37+
CLOUDFLARE_API_URL=https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai
38+
CLOUDFLARE_API_KEY=
39+
COHERE_API_URL=https://api.cohere.ai
40+
COHERE_API_KEY=
41+
GROQ_API_URL=https://api.groq.com/openai/v1
42+
GROQ_API_KEY=
43+
OLLAMA_API_URL=http://ollama:8080/v1
44+
OLLAMA_API_KEY=
45+
OPENAI_API_URL=https://api.openai.com/v1
46+
OPENAI_API_KEY=
47+
DEEPSEEK_API_URL=https://api.deepseek.com
48+
DEEPSEEK_API_KEY=

‎examples/QUICKSTART.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# Quick Start Guide
2+
3+
This guide will help you run all TypeScript SDK examples quickly.
4+
5+
## Prerequisites
6+
7+
1. **Docker** - For running the Inference Gateway
8+
2. **Node.js** - For running the examples
9+
3. **API Key** - For at least one AI provider
10+
11+
## 1. Setup Environment
12+
13+
1. Copy the environment template:
14+
15+
```bash
16+
cp .env.example .env
17+
```
18+
19+
2. Add your API keys to `.env`:
20+
```bash
21+
# Choose one or more providers
22+
GROQ_API_KEY=your_groq_key_here
23+
OPENAI_API_KEY=your_openai_key_here
24+
ANTHROPIC_API_KEY=your_anthropic_key_here
25+
```
26+
27+
## 2. Start Inference Gateway
28+
29+
Choose one of these options:
30+
31+
### Option A: Basic Gateway (for List and Chat examples)
32+
33+
```bash
34+
docker run --rm -p 8080:8080 --env-file .env ghcr.io/inference-gateway/inference-gateway:latest
35+
```
36+
37+
### Option B: Gateway with MCP (for all examples)
38+
39+
```bash
40+
cd mcp
41+
npm run compose:up
42+
```
43+
44+
## 3. Test the Examples
45+
46+
### Quick Test - List Models
47+
48+
```bash
49+
cd list
50+
npm install
51+
npm start
52+
```
53+
54+
### Chat Example
55+
56+
```bash
57+
cd chat
58+
export PROVIDER=groq
59+
export LLM=meta-llama/llama-3.3-70b-versatile
60+
npm install
61+
npm start
62+
```
63+
64+
### MCP Example (requires Docker Compose setup)
65+
66+
```bash
67+
cd mcp
68+
export PROVIDER=groq
69+
export LLM=meta-llama/llama-3.3-70b-versatile
70+
npm install
71+
npm start
72+
```
73+
74+
## 4. Popular Provider/Model Combinations
75+
76+
### Groq (Fast inference)
77+
78+
```bash
79+
export PROVIDER=groq
80+
export LLM=meta-llama/llama-3.3-70b-versatile
81+
```
82+
83+
### OpenAI (High quality)
84+
85+
```bash
86+
export PROVIDER=openai
87+
export LLM=gpt-4o
88+
```
89+
90+
### Anthropic (Strong reasoning)
91+
92+
```bash
93+
export PROVIDER=anthropic
94+
export LLM=claude-3-5-sonnet-20241022
95+
```
96+
97+
## Troubleshooting
98+
99+
### Gateway not responding
100+
101+
- Check if Docker container is running: `docker ps`
102+
- Test health: `curl http://localhost:8080/health`
103+
- Check logs: `docker logs <container_id>`
104+
105+
### Authentication errors
106+
107+
- Verify API key is correct in `.env`
108+
- Ensure the key has sufficient permissions
109+
- Try a different provider
110+
111+
### Model not found
112+
113+
- Use the list example to see available models
114+
- Check if the model name is correct
115+
- Try a different model from the same provider
116+
117+
## Next Steps
118+
119+
1. Explore each example in detail
120+
2. Modify the examples for your use case
121+
3. Build your own applications using the patterns shown
122+
4. Check the main [README](../README.md) for more advanced usage

0 commit comments

Comments
 (0)