From c23da96cefa5712768951274f75821a55e6067c4 Mon Sep 17 00:00:00 2001 From: Jui-Tse Hung Date: Mon, 7 Aug 2023 20:13:16 +0000 Subject: [PATCH 1/6] Add documentation on pointing llmengine client to self-hosted infrastructure --- docs/guides/self_hosting.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/guides/self_hosting.md b/docs/guides/self_hosting.md index 8c6c963b..5dd40803 100644 --- a/docs/guides/self_hosting.md +++ b/docs/guides/self_hosting.md @@ -200,4 +200,7 @@ $ curl -X POST 'http://localhost:5000/v1/llm/completions-sync?model_endpoint_nam You should get a response similar to: ``` {"status":"SUCCESS","outputs":[{"text":". Tell me a joke about AI. Tell me a joke about AI. Tell me a joke about AI. Tell me","num_completion_tokens":30}],"traceback":null} -``` \ No newline at end of file +``` + +### Pointing LLM Engine client to use self-hosted infrastructure +The `llmengine` client makes requests to Scale AI's hosted infrastructure by default. You can have `llmengine` client make requests to your own self-hosted infrastructure by setting the `LLM_ENGINE_BASE_PATH` environment variable to the url of the `llm-engine` pod. The exact url of `llm-engine` pod depends on your Kubernetes cluster networking setup. From 91caba867d424adb6e726f2f29cddb5677de5f60 Mon Sep 17 00:00:00 2001 From: Jui-Tse Hung Date: Mon, 7 Aug 2023 20:15:20 +0000 Subject: [PATCH 2/6] url -> URL --- docs/guides/self_hosting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/self_hosting.md b/docs/guides/self_hosting.md index 5dd40803..502de4d8 100644 --- a/docs/guides/self_hosting.md +++ b/docs/guides/self_hosting.md @@ -203,4 +203,4 @@ You should get a response similar to: ``` ### Pointing LLM Engine client to use self-hosted infrastructure -The `llmengine` client makes requests to Scale AI's hosted infrastructure by default. You can have `llmengine` client make requests to your own self-hosted infrastructure by setting the `LLM_ENGINE_BASE_PATH` environment variable to the url of the `llm-engine` pod. The exact url of `llm-engine` pod depends on your Kubernetes cluster networking setup. +The `llmengine` client makes requests to Scale AI's hosted infrastructure by default. You can have `llmengine` client make requests to your own self-hosted infrastructure by setting the `LLM_ENGINE_BASE_PATH` environment variable to the URL of the `llm-engine` pod. The exact URL of `llm-engine` pod depends on your Kubernetes cluster networking setup. From db1a9def09244cb06c552fa5f2d9e42a2e98c26e Mon Sep 17 00:00:00 2001 From: Jui-Tse Hung Date: Mon, 7 Aug 2023 22:50:49 +0000 Subject: [PATCH 3/6] Add code sample for setting LLM_ENGINE_BASE_PATH --- docs/guides/self_hosting.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/guides/self_hosting.md b/docs/guides/self_hosting.md index 502de4d8..7720dda9 100644 --- a/docs/guides/self_hosting.md +++ b/docs/guides/self_hosting.md @@ -203,4 +203,9 @@ You should get a response similar to: ``` ### Pointing LLM Engine client to use self-hosted infrastructure -The `llmengine` client makes requests to Scale AI's hosted infrastructure by default. You can have `llmengine` client make requests to your own self-hosted infrastructure by setting the `LLM_ENGINE_BASE_PATH` environment variable to the URL of the `llm-engine` pod. The exact URL of `llm-engine` pod depends on your Kubernetes cluster networking setup. +The `llmengine` client makes requests to Scale AI's hosted infrastructure by default. You can have `llmengine` client make requests to your own self-hosted infrastructure by setting the `LLM_ENGINE_BASE_PATH` environment variable to the URL of the `llm-engine` pod. + +The exact URL of `llm-engine` pod depends on your Kubernetes cluster networking setup. The domain is specified in `config.values.infra.dns_host_domain` in the helm chart values config file. Using `charts/llm-engine/values_sample.yaml` as an example, you would +```bash +export LLM_ENGINE_BASE_PATH=https://domain.llm-engine.com +``` \ No newline at end of file From 771b69307a3e09d592a029fd7408da874e53da73 Mon Sep 17 00:00:00 2001 From: Jui-Tse Hung Date: Mon, 7 Aug 2023 22:51:28 +0000 Subject: [PATCH 4/6] Small wording fix --- docs/guides/self_hosting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/self_hosting.md b/docs/guides/self_hosting.md index 7720dda9..338cfeb0 100644 --- a/docs/guides/self_hosting.md +++ b/docs/guides/self_hosting.md @@ -205,7 +205,7 @@ You should get a response similar to: ### Pointing LLM Engine client to use self-hosted infrastructure The `llmengine` client makes requests to Scale AI's hosted infrastructure by default. You can have `llmengine` client make requests to your own self-hosted infrastructure by setting the `LLM_ENGINE_BASE_PATH` environment variable to the URL of the `llm-engine` pod. -The exact URL of `llm-engine` pod depends on your Kubernetes cluster networking setup. The domain is specified in `config.values.infra.dns_host_domain` in the helm chart values config file. Using `charts/llm-engine/values_sample.yaml` as an example, you would +The exact URL of `llm-engine` pod depends on your Kubernetes cluster networking setup. The domain is specified at `config.values.infra.dns_host_domain` in the helm chart values config file. Using `charts/llm-engine/values_sample.yaml` as an example, you would ```bash export LLM_ENGINE_BASE_PATH=https://domain.llm-engine.com ``` \ No newline at end of file From 68dfec45bd5d1bc5426077deab60ce7b4c55921d Mon Sep 17 00:00:00 2001 From: Jui-Tse Hung Date: Tue, 8 Aug 2023 00:24:02 +0000 Subject: [PATCH 5/6] Wording fix --- docs/guides/self_hosting.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/self_hosting.md b/docs/guides/self_hosting.md index 338cfeb0..d9fb26eb 100644 --- a/docs/guides/self_hosting.md +++ b/docs/guides/self_hosting.md @@ -203,9 +203,9 @@ You should get a response similar to: ``` ### Pointing LLM Engine client to use self-hosted infrastructure -The `llmengine` client makes requests to Scale AI's hosted infrastructure by default. You can have `llmengine` client make requests to your own self-hosted infrastructure by setting the `LLM_ENGINE_BASE_PATH` environment variable to the URL of the `llm-engine` pod. +The `llmengine` client makes requests to Scale AI's hosted infrastructure by default. You can have `llmengine` client make requests to your own self-hosted infrastructure by setting the `LLM_ENGINE_BASE_PATH` environment variable to the URL of the `llm-engine` service. -The exact URL of `llm-engine` pod depends on your Kubernetes cluster networking setup. The domain is specified at `config.values.infra.dns_host_domain` in the helm chart values config file. Using `charts/llm-engine/values_sample.yaml` as an example, you would +The exact URL of `llm-engine` service depends on your Kubernetes cluster networking setup. The domain is specified at `config.values.infra.dns_host_domain` in the helm chart values config file. Using `charts/llm-engine/values_sample.yaml` as an example, you would do: ```bash export LLM_ENGINE_BASE_PATH=https://domain.llm-engine.com ``` \ No newline at end of file From a19b460e28f029a82a140d6dbfe054572c6ce0a6 Mon Sep 17 00:00:00 2001 From: Jui-Tse Hung Date: Tue, 8 Aug 2023 19:47:10 +0000 Subject: [PATCH 6/6] Update dns_host_domain --- charts/llm-engine/values_sample.yaml | 2 +- docs/guides/self_hosting.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/charts/llm-engine/values_sample.yaml b/charts/llm-engine/values_sample.yaml index 06d70362..70d740cf 100644 --- a/charts/llm-engine/values_sample.yaml +++ b/charts/llm-engine/values_sample.yaml @@ -96,7 +96,7 @@ config: # k8s_cluster_name [required] is the name of the k8s cluster k8s_cluster_name: main_cluster # dns_host_domain [required] is the domain name of the k8s cluster - dns_host_domain: domain.llm-engine.com + dns_host_domain: llm-engine.domain.com # default_region [required] is the default AWS region for various resources (e.g ECR) default_region: us-east-1 # aws_account_id [required] is the AWS account ID for various resources (e.g ECR) diff --git a/docs/guides/self_hosting.md b/docs/guides/self_hosting.md index d9fb26eb..0c446191 100644 --- a/docs/guides/self_hosting.md +++ b/docs/guides/self_hosting.md @@ -207,5 +207,5 @@ The `llmengine` client makes requests to Scale AI's hosted infrastructure by def The exact URL of `llm-engine` service depends on your Kubernetes cluster networking setup. The domain is specified at `config.values.infra.dns_host_domain` in the helm chart values config file. Using `charts/llm-engine/values_sample.yaml` as an example, you would do: ```bash -export LLM_ENGINE_BASE_PATH=https://domain.llm-engine.com +export LLM_ENGINE_BASE_PATH=https://llm-engine.domain.com ``` \ No newline at end of file