Clarification on recommended setup for mixed OpenAI + self-hosted SGLang

[RafayAK]

First, thanks for the project. I was able to get Shepherd Model Gateway working with both an external OpenAI provider and a self-hosted SGLang worker, but I hit a confusing setup path along the way and wanted to confirm the intended configuration pattern.

My goal was to run SMG as a single gateway for:

• A self-hosted SGLang model
• OpenAI models such as gpt-4.1-mini, gpt-4o-transcripton etc the whole lot.
• The OpenAI-compatible /v1/chat/completions endpoint exposed by SMG

Initial setup that did not work

I first started my self-hosted SGLang model like this:

docker run --restart unless-stopped -d \
  --name sglang \
  --gpus all \
  --privileged \
  -p 30001:30001 \
  --ipc host \
  -v /opt/dlami/nvme/hf-cache:/root/.cache/huggingface \
  -e HF_TOKEN=<my_token> \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  lmsysorg/sglang:latest \
  sglang serve \
  --model-path Qwen/Qwen3.6-35B-A3B-FP8 \
  --host 0.0.0.0 \
  --port 30001 \
  --tp-size 4 \
  --context-length 64000 \
  --mem-fraction-static 0.78 \
  --reasoning-parser qwen3 \
  --mamba-scheduler-strategy extra_buffer \
  --page-size 64

Then I tried to start SMG with both the self-hosted worker and OpenAI in --worker-urls:

docker run -d \
  --name smg \
  -p 30000:30000 \
  -p 29000:29000 \
  -e OPENAI_API_KEY=<my_openai_key> \
  lightseekorg/smg:latest \
  --policy cache_aware \
  --port 30000 \
  --prometheus-port 29000 \
  --worker-urls http://172.17.0.1:30001 https://api.openai.com

When I checked /workers, both workers appeared and OpenAI showed as healthy/ready. However, the OpenAI worker did not appear to advertise any model inventory, while the SGLang worker did.

$ curl http://localhost:30000/workers | jq .
  "workers": [
    {
      "id": "019f23c0-ed6b-7683-9068-19dbbbc9ec90",
      "model_id": "Qwen/Qwen3.6-35B-A3B-FP8",
      "url": "http://172.17.0.1:30001",
      "models": [
        {
          "id": "Qwen/Qwen3.6-35B-A3B-FP8",
          "model_type": [
            "chat",
            "completions",
            "responses",
            "vision",
            "tools"
          ],
          "hf_model_type": "qwen3_5_moe",
          "architectures": [
            "Qwen3_5MoeForConditionalGeneration"
          ],
          "context_length": 64000,
          "tokenizer_path": "Qwen/Qwen3.6-35B-A3B-FP8"
        }
      ],
      "worker_type": "regular",
      "connection_mode": "http",
      "runtime_type": "sglang",
      "labels": {
        "tp_size": "4",
        "model_type": "qwen3_5_moe",
        "served_model_name": "Qwen/Qwen3.6-35B-A3B-FP8",
        "max_model_len": "64000",
        "tokenizer_path": "Qwen/Qwen3.6-35B-A3B-FP8",
        "weight_version": "default",
        "model_path": "Qwen/Qwen3.6-35B-A3B-FP8",
        "architectures": "[\"Qwen3_5MoeForConditionalGeneration\"]",
        "has_image_understanding": "true",
        "is_generation": "true",
        "version": "0.5.14",
        "context_length": "64000",
        "pp_size": "1",
        "is_embedding": "false",
        "load_balance_method": "round_robin",
        "dp_size": "1",
        "has_audio_understanding": "false"
      },
      "priority": 50,
      "cost": 1.0,
      "max_connection_attempts": 20,
      "is_healthy": true,
      "status": "ready",
      "load": 0
    },
    {
      "id": "019f23bf-4f1b-77b2-8135-e520af1c95d5",
      "url": "https://api.openai.com",
      "worker_type": "regular",
      "connection_mode": "http",
      "runtime_type": "external",
      "priority": 50,
      "cost": 1.0,
      "max_connection_attempts": 20,
      "is_healthy": true,
      "status": "ready",
      "load": 0
    }
  ],
  "total": 2,
  "stats": {
    "prefill_count": 0,
    "decode_count": 0,
    "regular_count": 2
  }
}

A request like this failed:

curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-4.1",
    "messages": [{"role": "user", "content": "Say hello and tell me what model you are."}]
  }' | jq .

Error:

{
  "error": {
    "type": "Not Found",
    "code": "model_not_found",
    "message": "No worker available for model 'gpt-4.1'",
    "param": null
  }
}

Similarly, I tried adding the parameter --backend openai to my Docker run command, but it would just treat even the self-hosted model as an external model.

In both the above cases, I noticed the logs :

2026-07-02 16:52:48  INFO smg::workflow::steps::external::discover_models: model_gateway/src/workflow/steps/external/discover_models.rs:281: No API key provided for https://api.openai.com - using wildcard mode (accepts any model). User's Authorization header will be forwarded to backend.
...
2026-07-02 16:52:48  INFO smg::workflow::steps::external::create_workers: model_gateway/src/workflow/steps/external/create_workers.rs:126: Created wildcard worker at https://api.openai.com (accepts any model, user auth forwarded)

But still, calling the model explicitly with this setup would fail even though something like querying all models with a BYOK setup works:

$ curl -s http://localhost:30000/v1/models \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY"
  
{"object":"list","data":[{"id":"text-embedding-ada-002","object":"model","created":0,"owned_by":"openai"},{"id":"whisper-1","object":"model","created":0,"owned_by":"openai"},{"id":"gpt-3.5-turbo","object":"model","created":0,"owned_by":"openai"},{"id":"tts-1","object":"model","created":0,"owned_by":"openai"},{"id":"gpt-3.5-turbo-16k","object":"model","created":0,"owned_by":"openai"},{"id":"gpt-4-0613","object":"model","created":0,"owned_by":"openai"},{"id":"gpt-4","object":"model","created":0,"owned_by":"openai"},{"id":"davinci-002","object":"model","created":0,"owned_by":"openai"},{"id":"babbage-002","object":"model","created":0,"owned_by":"openai"},{"id":"gpt-3.5-turbo-instruct","object":"model","created":0,"owned_by":"openai"},{"id":"gpt-3.5-turbo-instruct-0914","object":"model","created":0,"owned_by":"openai"}...

Setup that worked

What finally worked was starting SMG in plain IGW mode and then registering both workers through the /workers API.

1. Start SMG:

docker run -d \
  --name smg \
  -p 30000:30000 \
  -p 29000:29000 \
  lightseekorg/smg:latest \
  --policy cache_aware \
  --port 30000 \
  --prometheus-port 29000 \
  --enable-igw

2. Register OpenAI:

curl -s -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d "{
    \"url\": \"https://api.openai.com\",
    \"api_key\": \"$OPENAI_API_KEY\",
    \"runtime_type\": \"external\",
    \"provider\": \"openai\"
  }" | jq .

With this, I actually did see an API call to fetch and register all models.

3. Register the self-hosted SGLang worker:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{"url": "http://172.17.0.1:30001"}'

After this, SMG was able to route requests correctly.

Clarification questions

Is this the intended/recommended setup for mixed external provider + self-hosted deployments?

In other words, should users who want both OpenAI and a self-hosted SGLang/vLLM/etc. worker:

1. Start SMG with --enable-igw
2. Avoid putting OpenAI in --worker-urls
3. Register OpenAI explicitly through POST /workers with runtime_type: "external" and provider: "openai"
4. Register the self-hosted worker separately through POST /workers

A few specific things that were unclear to me:

• Is --worker-urls intended only for either self-hosted/OpenAI-compatible inference workers, not when combined with external providers like OpenAI in the same gateway instantiation?
• If https://api.openai.com is passed through --worker-urls, without, let's say, --backend openai, should SMG warn that this is not enough to configure it as an OpenAI provider? Why was it not able to pick up the OPENAI_API_KEY and fetch the models? How is this supposed to work in a BYOK setting where the key is passed as a bearer token in the OpenAI-compatible API call?
• Is it expected that an external OpenAI worker may show as healthy/ready in /workers but still not be routable for gpt-*?

Possible docs improvement

It might help to add a small section to the external providers docs showing the exact mixed deployment pattern:

smg --enable-igw --policy cache_aware

Then:

POST /workers
{
  "url": "https://api.openai.com",
  "api_key": "$OPENAI_API_KEY",
  "runtime_type": "external",
  "provider": "openai"   ## <<< This is some how not listed in worker config options https://lightseek.org/smg/getting-started/multiple-workers/#worker-configuration-options
}

And:

POST /workers
{
  "url": "http://my-sglang-worker:30001"
}

That would make it clearer that external provider registration and self-hosted worker registration are distinct, even though both ultimately appear under /workers.

Happy to open a docs PR if this is indeed the intended pattern. Just wanted to confirm before encoding the wrong behavior in our internal deployment docs.

[RafayAK]

Noticed other discussion threads also tagged you, so 🙃
@slin1237

[slin1237]

This seems like a bug and test we failed to cover.
IGW mode is required for mixed workers
But it shouldn't fail like what you did. I will address this.
Thanks for reporting this.