Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
.devcontainer/** linguist-vendored=true
src/examples/** linguist-vendored=true

src/types/generated/** linguist-generated=true

1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
node_modules
dist
coverage
**/.env
12 changes: 9 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,14 @@ An SDK written in TypeScript for the [Inference Gateway](https://github.com/eden
- [Usage](#usage)
- [Creating a Client](#creating-a-client)
- [Listing Models](#listing-models)
- [Listing MCP Tools](#listing-mcp-tools)
- [Creating Chat Completions](#creating-chat-completions)
- [Streaming Chat Completions](#streaming-chat-completions)
- [Tool Calls](#tool-calls)
- [Proxying Requests](#proxying-requests)
- [Health Check](#health-check)
- [Creating a Client with Custom Options](#creating-a-client-with-custom-options)
- [Examples](#examples)
- [Contributing](#contributing)
- [License](#license)

Expand Down Expand Up @@ -51,7 +53,7 @@ try {
console.log('All models:', models);

// List models from a specific provider
const openaiModels = await client.listModels(Provider.OpenAI);
const openaiModels = await client.listModels(Provider.openai);
console.log('OpenAI models:', openaiModels);
} catch (error) {
console.error('Error:', error);
Expand Down Expand Up @@ -235,7 +237,7 @@ To proxy requests directly to a provider:
import { InferenceGatewayClient, Provider } from '@inference-gateway/sdk';

const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080/v1',
baseURL: 'http://localhost:8080',
});

try {
Expand All @@ -261,7 +263,7 @@ To check if the Inference Gateway is running:
import { InferenceGatewayClient } from '@inference-gateway/sdk';

const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080/v1',
baseURL: 'http://localhost:8080',
});

try {
Expand Down Expand Up @@ -292,6 +294,10 @@ const clientWithHeaders = client.withOptions({
});
```

### Examples

For more examples, check the [examples directory](./examples).

## Contributing

Please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file for information about how to get involved. We welcome issues, questions, and pull requests.
Expand Down
81 changes: 77 additions & 4 deletions eslint.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,26 @@ export default [
plugins: {
prettier: prettier,
},
languageOptions: {
globals: {
console: 'readonly',
process: 'readonly',
Buffer: 'readonly',
__dirname: 'readonly',
__filename: 'readonly',
global: 'readonly',
module: 'readonly',
require: 'readonly',
exports: 'readonly',
},
},
rules: {
'prettier/prettier': 'error',
...eslint.configs.recommended.rules,
},
},
{
files: ['**/*.ts'],
files: ['src/**/*.ts', 'tests/**/*.ts'],
languageOptions: {
parser: tsParser,
parserOptions: {
Expand All @@ -25,12 +38,28 @@ export default [
sourceType: 'module',
},
globals: {
// Add Jest globals
console: 'readonly',
process: 'readonly',
Buffer: 'readonly',
__dirname: 'readonly',
__filename: 'readonly',
global: 'readonly',
module: 'readonly',
require: 'readonly',
exports: 'readonly',
describe: 'readonly',
it: 'readonly',
expect: 'readonly',
beforeEach: 'readonly',
afterEach: 'readonly',
beforeAll: 'readonly',
afterAll: 'readonly',
jest: 'readonly',
Headers: 'readonly',
fetch: 'readonly',
ReadableStream: 'readonly',
Response: 'readonly',
Request: 'readonly',
},
},
plugins: {
Expand All @@ -45,10 +74,54 @@ export default [
...tseslint.configs.recommended.rules,
},
},
{
files: ['examples/**/*.ts'],
languageOptions: {
parser: tsParser,
parserOptions: {
ecmaVersion: 'latest',
sourceType: 'module',
},
globals: {
console: 'readonly',
process: 'readonly',
Buffer: 'readonly',
__dirname: 'readonly',
__filename: 'readonly',
global: 'readonly',
module: 'readonly',
require: 'readonly',
exports: 'readonly',
},
},
plugins: {
'@typescript-eslint': tseslint,
},
rules: {
'@typescript-eslint/no-unused-vars': [
'error',
{ argsIgnorePattern: '^_' },
],
},
},
{
files: ['**/*.test.ts'],
env: {
'jest/globals': true,
languageOptions: {
globals: {
describe: 'readonly',
it: 'readonly',
expect: 'readonly',
beforeEach: 'readonly',
afterEach: 'readonly',
beforeAll: 'readonly',
afterAll: 'readonly',
jest: 'readonly',
Headers: 'readonly',
fetch: 'readonly',
ReadableStream: 'readonly',
Response: 'readonly',
Request: 'readonly',
},
},
},
];
48 changes: 48 additions & 0 deletions examples/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@

# General settings
ENVIRONMENT=development
ENABLE_TELEMETRY=false
ENABLE_AUTH=false
# Model Context Protocol (MCP)
MCP_ENABLE=false
MCP_EXPOSE=false
MCP_SERVERS=
MCP_CLIENT_TIMEOUT=5s
MCP_DIAL_TIMEOUT=3s
MCP_TLS_HANDSHAKE_TIMEOUT=3s
MCP_RESPONSE_HEADER_TIMEOUT=3s
MCP_EXPECT_CONTINUE_TIMEOUT=1s
MCP_REQUEST_TIMEOUT=5s
# OpenID Connect
OIDC_ISSUER_URL=http://keycloak:8080/realms/inference-gateway-realm
OIDC_CLIENT_ID=inference-gateway-client
OIDC_CLIENT_SECRET=
# Server settings
SERVER_HOST=0.0.0.0
SERVER_PORT=8080
SERVER_READ_TIMEOUT=30s
SERVER_WRITE_TIMEOUT=30s
SERVER_IDLE_TIMEOUT=120s
SERVER_TLS_CERT_PATH=
SERVER_TLS_KEY_PATH=
# Client settings
CLIENT_TIMEOUT=30s
CLIENT_MAX_IDLE_CONNS=20
CLIENT_MAX_IDLE_CONNS_PER_HOST=20
CLIENT_IDLE_CONN_TIMEOUT=30s
CLIENT_TLS_MIN_VERSION=TLS12
# Providers
ANTHROPIC_API_URL=https://api.anthropic.com/v1
ANTHROPIC_API_KEY=
CLOUDFLARE_API_URL=https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai
CLOUDFLARE_API_KEY=
COHERE_API_URL=https://api.cohere.ai
COHERE_API_KEY=
GROQ_API_URL=https://api.groq.com/openai/v1
GROQ_API_KEY=
OLLAMA_API_URL=http://ollama:8080/v1
OLLAMA_API_KEY=
OPENAI_API_URL=https://api.openai.com/v1
OPENAI_API_KEY=
DEEPSEEK_API_URL=https://api.deepseek.com
DEEPSEEK_API_KEY=
122 changes: 122 additions & 0 deletions examples/QUICKSTART.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# Quick Start Guide

This guide will help you run all TypeScript SDK examples quickly.

## Prerequisites

1. **Docker** - For running the Inference Gateway
2. **Node.js** - For running the examples
3. **API Key** - For at least one AI provider

## 1. Setup Environment

1. Copy the environment template:

```bash
cp .env.example .env
```

2. Add your API keys to `.env`:
```bash
# Choose one or more providers
GROQ_API_KEY=your_groq_key_here
OPENAI_API_KEY=your_openai_key_here
ANTHROPIC_API_KEY=your_anthropic_key_here
```

## 2. Start Inference Gateway

Choose one of these options:

### Option A: Basic Gateway (for List and Chat examples)

```bash
docker run --rm -p 8080:8080 --env-file .env ghcr.io/inference-gateway/inference-gateway:latest
```

### Option B: Gateway with MCP (for all examples)

```bash
cd mcp
npm run compose:up
```

## 3. Test the Examples

### Quick Test - List Models

```bash
cd list
npm install
npm start
```

### Chat Example

```bash
cd chat
export PROVIDER=groq
export LLM=meta-llama/llama-3.3-70b-versatile
npm install
npm start
```

### MCP Example (requires Docker Compose setup)

```bash
cd mcp
export PROVIDER=groq
export LLM=meta-llama/llama-3.3-70b-versatile
npm install
npm start
```

## 4. Popular Provider/Model Combinations

### Groq (Fast inference)

```bash
export PROVIDER=groq
export LLM=meta-llama/llama-3.3-70b-versatile
```

### OpenAI (High quality)

```bash
export PROVIDER=openai
export LLM=gpt-4o
```

### Anthropic (Strong reasoning)

```bash
export PROVIDER=anthropic
export LLM=claude-3-5-sonnet-20241022
```

## Troubleshooting

### Gateway not responding

- Check if Docker container is running: `docker ps`
- Test health: `curl http://localhost:8080/health`
- Check logs: `docker logs <container_id>`

### Authentication errors

- Verify API key is correct in `.env`
- Ensure the key has sufficient permissions
- Try a different provider

### Model not found

- Use the list example to see available models
- Check if the model name is correct
- Try a different model from the same provider

## Next Steps

1. Explore each example in detail
2. Modify the examples for your use case
3. Build your own applications using the patterns shown
4. Check the main [README](../README.md) for more advanced usage
Loading