issue: non_ASCII UTF-8 always encoded in payload sent to model (leading to bad tokenization)

[mramendi]

Check Existing Issues

• I have searched for any existing and/or related issues.
• I have searched for any existing and/or related discussions.
• I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.6.30

Ollama Version (if applicable)

No response

Operating System

RHEL 9

Browser (if applicable)

No response

Confirmation

• I have read and followed all instructions in README.md.
• I am using the latest version of both Open WebUI and Ollama.
• I have included the browser console logs.
• I have included the Docker container logs.
• I have provided every relevant configuration, setting, and environment variable used in my setup.
• I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
• I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
• Start with the initial platform/version/OS and dependencies used,
• Specify exact install/launch/configure commands,
• List URLs visited, user input (incl. example values/emails/passwords if needed),
• Describe all options and toggles enabled or changed,
• Include any files or environmental changes,
• Identify the expected and actual result at each stage,
• Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

When I enter a message in non-ASCII UTF-8, for example, Cyrillic, it should be sent to the model unmodified as UTF8, for example, "привет!". Or at least there should be a switch to enable this behaviour.

Actual Behavior

When I enter a message in non-ASCII UTF-8, for example, Cyrillic, it is sent to the model encoded with \u, for example, "\u043f\u0440\u0438\u0432\u0435\u0442!"

Steps to Reproduce

This is my particular container setup. I don't believe this is really caused by Podman/Quadlet, but I am including the relevant config files anyway. The upload failed so I paste them at the end of this message. So, steps in my specific case:

1. Start with RHEL9 - that is what I have, but Fedora, CentOS Stream, Rocky, Alma should all work. Just be sure you're in "Red Hat world" so you have Podman and Quadlet.
2. Make sure the following packages are installed: podman, podman-compose, podman-plugins.
3. Place the files that I pasted below into the $HOME/.config/containers/systemd directory. Modify for file paths and API/database endpoints/keys (note my setup injects API keys and the PostgreSQL password in an start-wrapper.sh script that is intentionally not pasted - it only sets the environment variables OPENAI_API_KEY and DATABASE_URL then execs the normal start.sh)
4. start the service with "systtemctl --user start openwebui-with-playwright
5. Either provide HTTPS with a reverse proxy, or for test purposes modify the files to publish the port on the host
6. Access OpenWebUI through a browser normally and go through the setup
7. Set up some way of intercepting and logging the request body. I use a simple LiteLLM callback; I can shjare it if necessary.
8. Click New chat and select a model (your API endpoint must provide at least one model)
9. Send the folowing message: "привет!"
10. Review the logged message body and notice the message is encoded as "\u043f\u0440\u0438\u0432\u0435\u0442!"

openwebui-with-playwright.yaml

apiVersion: v1
kind: Pod
metadata:
name: openwebui-playwright
spec:
containers:

---------- OpenWebUI ----------

• name: openwebui
  image: ghcr.io/open-webui/open-webui:main
  env:

  • name: OPENAI_API_BASE_URL
    value: "http://host.containers.internal:4000"
  • name: WEB_LOADER_ENGINE
    value: playwright
  • name: PLAYWRIGHT_WS_URL
    value: "ws://localhost:3000" # inside the pod => localhost
  • name: PYTHONPATH
    value: /app/backend/data/site-packages
  • name: USER_AGENT
    value: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36
  • name: PYTHONIOENCODING
    value: utf-8
  • name: JSON_ENSURE_ASCII
    value: False
    command: ["/config/start-wrapper.sh"]

  ports:

  • containerPort: 8080
    volumeMounts:
  • name: data
    mountPath: /app/backend/data
  • name: master-key
    mountPath: /run/secrets/litellm-master.key
    subPath: litellm-master.key
    readOnly: true
  • name: db-pass
    mountPath: /run/secrets/openwebui-db.pass
    subPath: openwebui.pass
    readOnly: true
  • name: config-files
    mountPath: /config/start-wrapper.sh
    subPath: start-wrapper.sh
    readOnly: true

---------- Playwright browser ----------

• name: playwright
  image: mcr.microsoft.com/playwright:v1.49.1-noble
  command: [ "npx", "playwright@1.49.1", "run-server", "--port=3000", "--host=0.0.0.0" ]
  ports:
  • containerPort: 3000 # only inside the pod

volumes:

• name: data
  hostPath:
  path: /srv/hdd/openwebui-files
  type: DirectoryOrCreate
• name: master-key
  hostPath:
  path: /home/admin/.config/ai
  type: Directory
• name: db-pass
  hostPath:
  path: /home/admin/.config/ai
  type: Directory
• name: config-files
  hostPath:
  path: /home/admin/.config/ai
  type: Directory
  ===

openwebui-with-playwright.kube

[Unit]
Description=OpenWebUI + Playwright (internal only)
After=ai-network.service litellm.service postgresql.service
Wants=ai-network.service

[Kube]
Yaml=%h/.config/containers/systemd/openwebui-with-playwright.yaml
Network=ai.network
AutoUpdate=registry

[Install]
WantedBy=default.target

Logs & Screenshots

I see nothing relevant in the pod logs, just a series of GET and POST headers. The browser behaves normally.

Additional Information

No response

[tjbck]

Unable to reproduce, most likely caused by the logging part. If not PR welcome!

[rgaricano]

Related: #15616

"привет!"

Chat Log:

]: 2025-09-24 01:40:29.508 | DEBUG    | open_webui.utils.middleware:process_chat_payload:804 - form_data: {'stream': True, 'model': 'ollama1.cogito:8b', 'messages': [{'role': 'user', 'content': '1.  "привет!"'}], 'features': {'image_generation': False, 'code_interpreter': False, 'web_search': False, 'memory': True}, 'variables': {'{{USER_NAME}}': 'ricardo', '{{USER_LOCATION}}': 'Unknown', '{{CURRENT_DATETIME}}': '2025-09-24 01:40:42', '{{CURRENT_DATE}}': '2025-09-24', '{{CURRENT_TIME}}': '01:40:42', '{{CURRENT_WEEKDAY}}': 'Wednesday', '{{CURRENT_TIMEZONE}}': 'Europe/Madrid', '{{USER_LANGUAGE}}': 'en-US'}, 'metadata': {'user_id': '4ed44e79-97d7-4152-bf26-694cece387df', 'chat_id': 'aeb60ff1-08b6-47a1-bc84-139936a452a8', 'message_id': '46bf8ff1-68b0-4977-b70b-af542084fe5d', 'session_id': 'OIkifXDU2t_MkBXOAABB', 'filter_ids': [], 'tool_ids': None, 'tool_servers': [], 'files': None, 'features': {'image_generation': False, 'code_interpreter': False, 'web_search': False, 'memory': True}, 'variables': {'{{USER_NAME}}': 'ricardo', '{{USER_LOCATION}}': 'Unknown', '{{CURRENT_DATETIME}}': '2025-09-24 01:40:42', '{{CURRENT_DATE}}': '2025-09-24', '{{CURRENT_TIME}}': '01:40:42', '{{CURRENT_WEEKDAY}}': 'Wednesday', '{{CURRENT_TIMEZONE}}': 'Europe/Madrid', '{{USER_LANGUAGE}}': 'en-US'}, 'model': {'id': 'ollama1.cogito:8b', 'name': 'cogito:8b', 'object': 'model', 'created': 1758670811, 'owned_by': 'ollama', 'ollama': {'name': 'cogito:8b', 'model': 'ollama1.cogito:8b', 'modified_at': '2025-09-08T06:11:24.914471522+02:00', 'size': 4920747948, 'digest': '75b508ddece134205c621b91237310024ca983027fb7f724c2032d106419bd1b', 'details': {'parent_model': '', 'format': 'gguf', 'family': 'llama', 'families': ['llama'], 'parameter_size': '8.0B', 'quantization_level': 'Q4_K_M'}, 'connection_type': 'external', 'urls': [1], 'expires_at': 10982042831}, 'connection_type': 'external', 'tags': [], 'info': {'id': 'ollama1.cogito:8b', 'user_id': '4ed44e79-97d7-4152-bf26-694cece387df', 'base_model_id': None, 'name': 'cogito:8b', 'params': {}, 'meta': {'profile_image_url': '/static/favicon.png', 'description': None, 'capabilities': {'vision': True, 'file_upload': True, 'web_search': True, 'image_generation': True, 'code_interpreter': True, 'citations': True, 'status_updates': True}, 'suggestion_prompts': None, 'tags': []}, 'access_control': None, 'is_active': True, 'updated_at': 1757618848, 'created_at': 1757618848}, 'actions': [], 'filters': []}, 'direct': False, 'params': {'stream_delta_chunk_size': None, 'reasoning_tags': None, 'function_calling': 'default'}}, 'options': {}}

chat-Json_tree:

Postgresql_db-chat: