[BUG] 403 Client Error: Forbidden for url: http://localhost:5000/gateway/chat/invocations

[astan-iq]

Issues Policy acknowledgement

• I have read and agree to submit bug reports in accordance with the issues policy

Where did you encounter this bug?

Local machine

Willingness to contribute

No. I cannot contribute a bug fix at this time.

MLflow version

• Client: 2.8.0
• Tracking server: 2.8.0

System information

• OSX 13.3 (22E252)
• Python 3.8.17

Describe the problem

Hello,

Getting 403 errors when trying to use MLFlow Gateway on on-prem Azure OpenAI deployments.

Trying to run example query code: https://gist.github.com/astan-iq/8da7001dede5b3aa6ecefa309f6e4a37
on a gateway running:
mlflow gateway start --config-path llm_gateway.yaml --port 5000
using the following gateway config: https://gist.github.com/astan-iq/9b264a5f1f06a96c764e2c16517b0987
I get:
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://localhost:5000/gateway/chat/invocations

Hitting the REST API:

curl -X 'POST' \
  'http://127.0.0.1:5000/gateway/chat/invocations' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "messages": [
    {
      "role": "user",
      "content": "hello world"
    }
  ],
  "temperature": 0,
  "max_tokens": 64,
  "stop": [
    "END"
  ],
  "candidate_count": 1
}'

I get:
{"detail":"invalid token provided: token contains an invalid number of segments"}

It would appear that the token is the issue, however I am using the same token as per this gist here: https://gist.github.com/astan-iq/f9486eea5e89e7ab0aff9608f474686f (which works fine).

It is likely that the configuration is at fault however any advise on how to debug the issue would be greatly appreciated.

Tracking information

System information: Darwin Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000
Python version: 3.8.17
MLflow version: 2.8.0
MLflow module location: /Users/USER/miniconda3/envs/dev/lib/python3.8/site-packages/mlflow/init.py
Tracking URI: file:///Users/USER/mlruns
Registry URI: file:///Users/USER/mlruns
MLflow environment variables:
MLFLOW_GATEWAY_URI: http://localhost:5000
MLflow dependencies:
Flask: 2.2.5
Jinja2: 3.0.3
aiohttp: 3.8.5
alembic: 1.11.2
boto3: 1.28.84
click: 8.1.6
cloudpickle: 2.2.1
databricks-cli: 0.17.7
docker: 6.1.3
entrypoints: 0.4
fastapi: 0.104.1
gitpython: 3.1.32
gunicorn: 20.1.0
importlib-metadata: 5.2.0
markdown: 3.4.4
matplotlib: 3.5.3
numpy: 1.23.5
packaging: 21.3
pandas: 1.4.4
prometheus-flask-exporter: 0.22.4
protobuf: 3.19.4
psutil: 5.9.5
pyarrow: 12.0.1
pydantic: 1.10.2
pysftp: 0.2.9
pytz: 2022.7.1
pyyaml: 6.0.1
querystring-parser: 1.2.4
requests: 2.31.0
scikit-learn: 1.0.2
scipy: 1.10.1
sqlalchemy: 1.4.49
sqlparse: 0.4.4
uvicorn: 0.24.0.post1
virtualenv: 20.24.3
watchfiles: 0.21.0

Code to reproduce issue

Client code, expected to work: https://gist.github.com/astan-iq/8da7001dede5b3aa6ecefa309f6e4a37
Gateway config: https://gist.github.com/astan-iq/9b264a5f1f06a96c764e2c16517b0987
What works: https://gist.github.com/astan-iq/f9486eea5e89e7ab0aff9608f474686f

Stack trace

Traceback (most recent call last):
  File "/Users/user/Library/Application Support/JetBrains/PyCharm2023.2/scratches/mlflow_llms_gateway.py", line 25, in <module>
    main()
  File "/Users/user/Library/Application Support/JetBrains/PyCharm2023.2/scratches/mlflow_llms_gateway.py", line 9, in main
    response_chat = query(
  File "/Users/user/miniconda3/envs/dev/lib/python3.8/site-packages/mlflow/gateway/fluent.py", line 250, in query
    return MlflowGatewayClient().query(route, data)
  File "/Users/user/miniconda3/envs/dev/lib/python3.8/site-packages/mlflow/gateway/client.py", line 335, in query
    return self._call_endpoint("POST", query_route, data).json()
  File "/Users/user/miniconda3/envs/dev/lib/python3.8/site-packages/mlflow/gateway/client.py", line 97, in _call_endpoint
    augmented_raise_for_status(response)
  File "/Users/user/miniconda3/envs/dev/lib/python3.8/site-packages/mlflow/utils/request_utils.py", line 39, in augmented_raise_for_status
    raise e
  File "/Users/user/miniconda3/envs/dev/lib/python3.8/site-packages/mlflow/utils/request_utils.py", line 32, in augmented_raise_for_status
    response.raise_for_status()
  File "/Users/user/miniconda3/envs/dev/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://localhost:5000/gateway/chat/invocations

Other info / logs

Gateway outputs

user@laptop scratches % mlflow gateway start --config-path llm_gateway.yaml --port 5000 
[2023-11-13 23:06:17 +0000] [98454] [INFO] Starting gunicorn 20.1.0
[2023-11-13 23:06:17 +0000] [98454] [INFO] Listening at: http://127.0.0.1:5000 (98454)
[2023-11-13 23:06:17 +0000] [98454] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2023-11-13 23:06:17 +0000] [98471] [INFO] Booting worker with pid: 98471
[2023-11-13 23:06:17 +0000] [98472] [INFO] Booting worker with pid: 98472
[2023-11-13 23:06:19 +0000] [98472] [INFO] Started server process [98472]
[2023-11-13 23:06:19 +0000] [98471] [INFO] Started server process [98471]
[2023-11-13 23:06:19 +0000] [98472] [INFO] Waiting for application startup.
[2023-11-13 23:06:19 +0000] [98471] [INFO] Waiting for application startup.
[2023-11-13 23:06:19 +0000] [98472] [INFO] Application startup complete.
[2023-11-13 23:06:19 +0000] [98471] [INFO] Application startup complete.

What component(s) does this bug affect?

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/gateway: AI Gateway service, Gateway client APIs, third-party Gateway integrations
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

What language(s) does this bug affect?

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

What integration(s) does this bug affect?

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

[BenWilson2]

@santiagxf @akshaya-a is there a guide for this on AzureML docs?

[github-actions]

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.