-
Notifications
You must be signed in to change notification settings - Fork 7
/
aigateway.yaml
106 lines (106 loc) · 3.29 KB
/
aigateway.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
# USAGE: take a copy of this manifest and then in your copy update the provided
# Secret to include your OpenAI API token and/or add tokens for other supported
# providers.
#
# Then `kubectl apply -f $manifest` the manifest and wait for the AIGateway to
# be fully deployed.
#
# FIXME: currently you'll need to find the Gateway resource that was created
# on behalf of the AIGateway to determine the address by which the Kong
# Gateway can be reached. In later iterations we'll have endpoints provided via
# the AIGateway resource status.
# See: https://github.com/Kong/gateway-operator/issues/137
#
# Once you have the endpoint, the path to your backends are based on the
# identifiers provided in the AIGateway below, and then you should be able to
# access them with curl or other tools, e.g.:
#
# curl -X POST -d 'prompt=hi' https://${GATEWAY_ADDR}/devteam-chatgpt
#
# You can also modify the defaultPrompts in the examples below to change the
# overall behavior of the models on responses.
---
apiVersion: v1
kind: Secret
metadata:
name: acme-ai-cloud-providers
type: Opaque
stringData:
openai: "<INSERT TOKEN HERE>"
---
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: kong-ai-gateways
spec:
controllerName: konghq.com/gateway-operator
parametersRef:
group: gateway-operator.konghq.com
kind: GatewayConfiguration
name: kong-ai
namespace: default
---
apiVersion: gateway-operator.konghq.com/v1alpha1
kind: AIGateway
metadata:
name: kong-aigateway
spec:
gatewayClassName: kong-ai-gateways
largeLanguageModels:
cloudHosted:
- identifier: marketing-team-classic-chatgpt
model: gpt-3.5-turbo-instruct
promptType: completions
aiCloudProvider:
name: openai
- identifier: devteam-chatgpt
model: gpt-4
promptType: chat
defaultPrompts:
- role: system
content: "You are a helpful assistant who responds in the style of Sherlock Holmes."
defaultPromptParams:
# temperature: "0.5" # higher confidence predictions
maxTokens: 50 # shorter responses
# topP: "0.9" # higher diversity
aiCloudProvider:
name: openai
cloudProviderCredentials:
name: acme-ai-cloud-providers
---
# TODO: eventually we want to be able to provide GatewayConfiguration resources
# at the individual Gateway level so that these can be created automatically
# and owned by the AIGateway resource.
#
# See: https://github.com/Kong/gateway-operator/issues/1368
apiVersion: gateway-operator.konghq.com/v1beta1
kind: GatewayConfiguration
metadata:
name: kong-ai
spec:
dataPlaneOptions:
deployment:
podTemplateSpec:
spec:
containers:
- name: proxy
image: kong:3.6.0
imagePullPolicy: Always
readinessProbe:
initialDelaySeconds: 1
periodSeconds: 1
env:
- name: KONG_ADMIN_GUI_LISTEN
value: 0.0.0.0:8002
- name: KONG_ADMIN_LISTEN
value: 0.0.0.0:8001, 0.0.0.0:8444 ssl reuseport backlog=16384
controlPlaneOptions:
deployment:
podTemplateSpec:
spec:
containers:
- name: controller
image: kong/kubernetes-ingress-controller:3.1.5
env:
- name: CONTROLLER_LOG_LEVEL
value: debug