config-object.md

Config Object

The config object is used to configure API interactions with various providers. It supports multiple modes such as single provider access, load balancing between providers, and fallback strategies.

The following JSON schema is used to validate the config object:

JSON Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "strategy": {
      "type": "object",
      "properties": {
        "mode": {
          "type": "string",
          "enum": [
            "single",
            "loadbalance",
            "fallback"
          ]
        },
        "on_status_codes": {
          "type": "array",
          "items": {
            "type": "integer"
          },
          "optional": true
        }
      }
    },
    "provider": {
      "type": "string",
      "enum": [
        "openai",
        "anthropic",
        "azure-openai",
        "anyscale",
        "cohere",
        "palm"
      ]
    },
    "resource_name": {
      "type": "string",
      "optional": true
    },
    "deployment_id": {
      "type": "string",
      "optional": true
    },
    "api_version": {
      "type": "string",
      "optional": true
    },
    "override_params": {
      "type": "object"
    },
    "api_key": {
      "type": "string"
    },
    "virtual_key": {
      "type": "string"
    },
    "cache": {
      "type": "object",
      "properties": {
        "mode": {
          "type": "string",
          "enum": [
            "simple",
            "semantic"
          ]
        },
        "max_age": {
          "type": "integer",
          "optional": true
        }
      },
      "required": [
        "mode"
      ]
    },
    "retry": {
      "type": "object",
      "properties": {
        "attempts": {
          "type": "integer"
        },
        "on_status_codes": {
          "type": "array",
          "items": {
            "type": "number"
          },
          "optional": true
        }
      },
      "required": [
        "attempts"
      ]
    },
    "weight": {
      "type": "number"
    },
    "on_status_codes": {
      "type": "array",
      "items": {
        "type": "integer"
      }
    },
    "targets": {
      "type": "array",
      "items": {
        "$ref": "#"
      }
    }
  },
  "anyOf": [
    {
      "required": [
        "provider",
        "api_key"
      ]
    },
    {
      "required": [
        "virtual_key"
      ]
    },
    {
      "required": [
        "strategy",
        "targets"
      ]
    },
    {
      "required": [
        "cache"
      ]
    },
    {
      "required": [
        "retry"
      ]
    }
  ],
  "additionalProperties": false
}

Example Configs

// Simple config with cache and retry
{
  "virtual_key": "***", // Your Virtual Key
  "cache": { // Optional
    "mode": "semantic",
    "max_age": 10000
  },
  "retry": { // Optional
    "attempts": 5,
    "on_status_codes": []
  }
}

// Load balancing with 2 OpenAI keys
{
  "strategy": {
      "mode": "loadbalance"
    },
  "targets": [
    {
      "provider": "openai",
      "api_key": "sk-***"
    },
    {
      "provider": "openai",
      "api_key": "sk-***"
    }
  ]
}

You can find more examples of schemas below.

Schema Details

Key Name	Description	Type	Required	Enum Values	Additional Info
strategy	Operational strategy for the config or any individual target	object	Yes (if no provider or virtual_key)	-	See Strategy Object Details
provider	Name of the service provider	string	Yes (if no mode or virtual_key)	"openai", "anthropic", "azure-openai", "anyscale", "cohere"	-
api_key	API key for the service provider	string	Yes (if provider is specified)	-	-
virtual_key	Virtual key identifier	string	Yes (if no mode or provider)	-	-
cache	Caching configuration	object	No	-	See Cache Object Details
retry	Retry configuration	object	No	-	See Retry Object Details
weight	Weight for load balancing	number	No	-	Used in loadbalance mode
on_status_codes	Status codes triggering fallback	array of strings	No	-	Used in fallback mode
targets	List of target configurations	array	Yes (if mode is specified)	-	Each item follows the config schema
request_timeout	Request timeout configuration	number	No	-	-
custom_host	Route to privately hosted model	string	No	-	Used in combination with provider + api_key
forward_headers	Forward sensitive headers directly	array of strings	No	-	-
override_params	Pass model name and other hyper parameters	object	No	"model", "temperature", "frequency_penalty", "logit_bias", "logprobs", "top_logprobs", "max_tokens", "n", "presence_penalty", "response_format", "seed", "stop", "top_p", etc.	Pass everything that's typically part of the payload

Strategy Object Details

Key Name	Description	Type	Required	Enum Values	Additional Info
mode	strategy mode for the config	string	Yes	"loadbalance", "fallback"
on_status_codes	status codes to apply the strategy. This field is only used when strategy mode is "fallback"	array of numbers	No		Optional

Cache Object Details

Key Name	Description	Type	Required	Enum Values	Additional Info
mode	Cache mode	string	Yes	"simple", "semantic"	-
max_age	Maximum age for cache entries	integer	No	-	Optional

Retry Object Details

Key Name	Description	Type	Required	Enum Values	Additional Info
attempts	Number of retry attempts	integer	Yes	-	-
on_status_codes	Status codes to trigger retries	array of strings	No	-	Optional

Notes

• The strategy mode key determines the operational mode of the config. If strategy mode is not specified, a single provider mode is assumed, requiring either provider and api_key or virtual_key.
• In loadbalance and fallback modes, the targets array specifies the configurations for each target.
• The cache and retry objects provide additional configurations for caching and retry policies, respectively.

Examples

Single Provider with API Key

{
  "provider": "openai",
  "api_key": "sk-***"
}

Passing Model & Hyperparameters with Override Option

{
  "provider": "anthropic",
  "api_key": "xxx",
  "override_params": {
    "model": "claude-3-sonnet-20240229",
    "max_tokens": 512,
    "temperature": 0
  }
}

Single Provider with Virtual Key

{
  "virtual_key": "***"
}

Single Provider with Virtual Key, Cache and Retry

{
  "virtual_key": "***",
  "cache": {
    "mode": "semantic",
    "max_age": 10000
  },
  "retry": {
    "attempts": 5,
    "on_status_codes": [429]
  }
}

Load Balancing with Two OpenAI API Keys

{
  "strategy": {
      "mode": "loadbalance"
    },
  "targets": [
    {
      "provider": "openai",
      "api_key": "sk-***"
    },
    {
      "provider": "openai",
      "api_key": "sk-***"
    }
  ]
}

Load Balancing and Fallback Combination

{
  "strategy": {
      "mode": "loadbalance"
    },
  "targets": [
    {
      "provider": "openai",
      "api_key": "sk-***"
    },
    {
      "strategy": {
          "mode": "fallback",
          "on_status_codes": [429, 241]
        },
      "targets": [
        {
          "virtual_key": "***"
        },
        {
          "virtual_key": "***"
        }
      ]
    }
  ]
}