decode error when opening openapi file (dockerized zulip)

[chobeat]

2022-07-10 12:42:14.535 ERR  [django.request] Internal Server Error: /api/send-message2022-07-10 12:42:14.533 ERR  [zerver.middleware.json_error_handler] Traceback (most recent call last):
  File "/srv/zulip-
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/zulip-venv-cache/a30311a4e5879c676d7aa8276dd6be5dcb00b3d1/zulip-py3-venv
    return self.dispatch(request, *args, **kwargs)
  File "/srv/zulip-venv-cache/a30311a4e5879c676d7aa8276dd6be5dcb00b3d1/zulip-py3-venv/lib/python3.8/site-packages/django/views/generic/base.py", line 98, i
    return handler(request, *args, **kwargs)
  File "/home/zulip/deployments/2022-06-21-21-02-46/./zerver/views/documentation.py", line 235, in get
    res
  File "/srv/zulip-venv-cache/a30311a4e5879c676d7aa8276dd6be5dcb00b3d1/zulip-py3-venv/lib/python3.8/site-packages/django/views/generic/base.py", line 159, in get
    context = self.get_context_data(**kwargs)
  File "/home/zulip/deployments/2022-06-21-21-02-46/./zerver/views/documentation.py", line 200, in get_context_data
    endpo
  File "/home/zulip/deployments/2022-06-21-21-02-46/./zerver/openapi/openapi.py", line 318, in get_endpoint_from_operationid
    for endpoint in openapi_spec.openapi()["paths"]:
  File "/home/zulip/d
    self.check_reload()
  File "/home/zulip/deployments/2022-06-21-21-02-46/./zerver/openapi/openapi.py", 
    openapi = yaml.load(f, Loader=yaml.CSafeLoader)
  File "/srv/zulip-ven
    return loader.get_single_data()
  File "/srv/zulip-venv-cache/a303
    node = self.get_single_node()
  File "yaml/_yaml.pyx", line 673, in yaml._
  File "yaml/_yaml.pyx", line 687, in yaml._yaml.CParser._compose_docu
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File 
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 845, in yaml._yaml.CParser._compose_mapping_node
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 845, in yaml._yaml.CParser._compose_mapping_node
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 845, in yaml._yaml.CParser._compose_mapping_node
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 845, in yaml._yaml.CParser._compose_mapping_node
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 845, in yaml._yaml.CParser._compose_mapping_node
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 845, in yaml._yaml.CParser._compose_mapping_node
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 845, in yaml._yaml.CParser._compose_mapping_node
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 845, in yaml._yaml.CParser._compose_mapping_node
  File "yaml/_yaml.pyx", line 729, in yaml._yaml.CParser._compose_node
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4715: ordinal not in range(128)^

I keep getting this kind of errors and I finally managed to trigger them. They happen when I open the docs for the API and the error is clearly an issue with the openapi and the zulip.yaml file. Nonetheless I don't understand if it's a problem with the encoding of the file or what.

To reproduce:

• start the dockerized zulip, latest version
• open any page in the zulip API docs like https://your-host/api/send-message
• it should raise a 404 and the error appears in the log file.

[Pandry]

I've been able to reproduce the issue: it is a matter of LANG environment variables:
The container defaults to C.UTF-8, which seems to cause the error.
Setting the LANG variable to the en_US.UTF-8 solved the issue for me

Alternatively, I've also been able to fix the issue by opening the zulip.yaml file with the encoding="utf8" parameter of the open API

zulip/zerver/openapi/openapi.py

Line 99 in f97fa80

with open(self.openapi_path) as f:

[zulipbot]

Hello @zulip/server-api, @zulip/server-production members, this issue was labeled with the "area: production", "area: documentation (api and integrations)" labels, so you may want to check it out!

[timabbott]

@andersk can you investigate this one?

[andersk]

I can’t reproduce this, on the same version (5.3 based on the venv hash) or latest (5.5), unless I deliberately sabotage the locale configuration:

--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -60,6 +60,7 @@ services:
       - "80:80"
       - "443:443"
     environment:
+      LC_ALL: C
       DB_HOST: "database"
       DB_HOST_PORT: "5432"
       DB_USER: "zulip"

What host system is running your containers? Are you using docker-compose? What version of Docker?

[chobeat]

Hello, hope these infos will help:

docker-compose version 1.27.4
docker version 20.10.17, build 100c701

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[andersk]

Hmm. I set up a CentOS 7 host system with the same versions and I’m still unable to reproduce.

What does this output for you?

docker compose exec zulip env | grep ^L

[chobeat]

Hello.

Sorry for the late reply.

This is the result

docker-compose exec zulip env | grep ^L
LINK_SETTINGS_TO_DATA=True
LANG=en_US.UTF-8```