From 1f8c59f3bea6ca9fc090303655745b3c09954431 Mon Sep 17 00:00:00 2001 From: Jui-Tse Hung Date: Tue, 18 Jul 2023 22:06:21 +0000 Subject: [PATCH 1/4] Add a Play With It section in docs/guides/self_hosting.md to document to how test sending a request and getting response via pod port forwarding --- docs/guides/self_hosting.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/guides/self_hosting.md b/docs/guides/self_hosting.md index 027428c9..27b4708c 100644 --- a/docs/guides/self_hosting.md +++ b/docs/guides/self_hosting.md @@ -115,3 +115,29 @@ Below are the configurations to specify in the `values_sample.yaml` file. | config.values.llm_engine.cache_redis_url | The full url for the redis cluster you wish to connect | Yes | | config.values.llm_engine.s3_file_llm_fine_tuning_job_repository | The S3 URI for the S3 bucket/key that you wish to save fine-tuned assets | Yes | | config.values.datadog_trace_enabled | Whether to enable datadog tracing, datadog must be installed in the cluster | No | + +## Play With It +Once `helm install` succeeds, you can forward port 5000 from a `llm-engine` pod and test sending requests to it. + +First, see a list of pods in the namespace that you performed `helm install` in: +``` +$ kubectl get pods -n +NAME READY STATUS RESTARTS AGE +llm-engine-668679554-9q4wj 1/1 Running 0 18m +llm-engine-668679554-xfhxx 1/1 Running 0 18m +llm-engine-cacher-5f8b794585-fq7dj 1/1 Running 0 18m +llm-engine-endpoint-builder-5cd6bf5bbc-sm254 1/1 Running 0 18m +llm-engine-image-cache-a10-sw4pg 1/1 Running 0 18h +``` +Note the pod name you see may be different. + +Forward a port from a `llm-engine` pod: +``` +$ kubectl port-forward pod/llm-engine- 5000:5000 -n +``` + +Then, try sending a request to get LLM model endpoints for `test-user-id`. You should get a response with empty list: +``` +$ curl -X GET -H "Content-Type: application/json" -u "test-user-id:" "http://localhost:5000/v1/llm/model-endpoints" +{"model_endpoints":[]}% +``` \ No newline at end of file From 7ff4cd8f3d9a3c03ba4d22367943d65eb6fd9d4f Mon Sep 17 00:00:00 2001 From: Jui-Tse Hung Date: Tue, 18 Jul 2023 22:08:59 +0000 Subject: [PATCH 2/4] Change llm-engine-image-cache pod age to 18m --- docs/guides/self_hosting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/self_hosting.md b/docs/guides/self_hosting.md index 27b4708c..10a64a18 100644 --- a/docs/guides/self_hosting.md +++ b/docs/guides/self_hosting.md @@ -127,7 +127,7 @@ llm-engine-668679554-9q4wj 1/1 Running 0 llm-engine-668679554-xfhxx 1/1 Running 0 18m llm-engine-cacher-5f8b794585-fq7dj 1/1 Running 0 18m llm-engine-endpoint-builder-5cd6bf5bbc-sm254 1/1 Running 0 18m -llm-engine-image-cache-a10-sw4pg 1/1 Running 0 18h +llm-engine-image-cache-a10-sw4pg 1/1 Running 0 18m ``` Note the pod name you see may be different. From fdf9aa307d48e7f45f02ae1b5de1eec95dcb74c3 Mon Sep 17 00:00:00 2001 From: Jui-Tse Hung Date: Tue, 18 Jul 2023 22:09:44 +0000 Subject: [PATCH 3/4] pod name -> pod names --- docs/guides/self_hosting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/self_hosting.md b/docs/guides/self_hosting.md index 10a64a18..56e19247 100644 --- a/docs/guides/self_hosting.md +++ b/docs/guides/self_hosting.md @@ -129,7 +129,7 @@ llm-engine-cacher-5f8b794585-fq7dj 1/1 Running 0 llm-engine-endpoint-builder-5cd6bf5bbc-sm254 1/1 Running 0 18m llm-engine-image-cache-a10-sw4pg 1/1 Running 0 18m ``` -Note the pod name you see may be different. +Note the pod names you see may be different. Forward a port from a `llm-engine` pod: ``` From 8c7d6fb6db7c65e84d4c8483800695906ed94464 Mon Sep 17 00:00:00 2001 From: Jui-Tse Hung Date: Tue, 18 Jul 2023 23:47:45 +0000 Subject: [PATCH 4/4] - 5000 -> - NAMESPACE_YOU_INSTALL_LLM_ENGINE -> NAMESPACE_WHERE_LLM_ENGINE_IS_INSTALLED --- docs/guides/self_hosting.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/guides/self_hosting.md b/docs/guides/self_hosting.md index 56e19247..23f66579 100644 --- a/docs/guides/self_hosting.md +++ b/docs/guides/self_hosting.md @@ -117,11 +117,11 @@ Below are the configurations to specify in the `values_sample.yaml` file. | config.values.datadog_trace_enabled | Whether to enable datadog tracing, datadog must be installed in the cluster | No | ## Play With It -Once `helm install` succeeds, you can forward port 5000 from a `llm-engine` pod and test sending requests to it. +Once `helm install` succeeds, you can forward port `5000` from a `llm-engine` pod and test sending requests to it. First, see a list of pods in the namespace that you performed `helm install` in: ``` -$ kubectl get pods -n +$ kubectl get pods -n NAME READY STATUS RESTARTS AGE llm-engine-668679554-9q4wj 1/1 Running 0 18m llm-engine-668679554-xfhxx 1/1 Running 0 18m @@ -133,7 +133,7 @@ Note the pod names you see may be different. Forward a port from a `llm-engine` pod: ``` -$ kubectl port-forward pod/llm-engine- 5000:5000 -n +$ kubectl port-forward pod/llm-engine- 5000:5000 -n ``` Then, try sending a request to get LLM model endpoints for `test-user-id`. You should get a response with empty list: