Server Status 500 when exporting task annotations #7347

alex-bronze-vision · 2024-01-11T10:52:16Z

Actions before raising this issue

I searched the existing issues and did not find anything similar.
I read/searched the docs

Steps to Reproduce

This error does not seem to have a way of reproducing... It happens on some tasks while not in others. It has been commented also by other people in the Discord.

It basically gives an 500 Status code, either from the UI or the SDK, when trying to export a given task:

cvat_sdk.api_client.exceptions.ServiceException: Status Code: 500
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Content-Length': '145', 'Content-Type': 'text/html; charset=utf-8', 'Cross-Origin-Opener-Policy': 'same-origin', 'Date': 'Thu, 11 Jan 2024 10:31:52 GMT', 'Referrer-Policy': 'same-origin', 'Server': 'nginx/1.18.0 (Ubuntu)', 'Vary': 'Origin', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'DENY', 'X-Request-Id': '639981a6-d94f-402c-bd79-1f0cebe8741b'})
HTTP response body: b'\n<!doctype html>\n<html lang="en">\n<head>\n  <title>Server Error (500)</title>\n</head>\n<body>\n  <h1>Server Error (500)</h1><p></p>\n</body>\n</html>\n'

The cvat_server log:

cvat_server                    | 2024-01-10 10:40:49,442 DEBG 'uvicorn-0' stderr output:
cvat_server                    | [2024-01-10 10:40:49,441] ERROR django.request: Internal Server Error: /api/jobs/27/annotations
cvat_server                    | Traceback (most recent call last):
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 534, in thread_handler
cvat_server                    |     raise exc_info[1]
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 42, in inner
cvat_server                    |     response = await get_response(request)
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
cvat_server                    |     response = await wrapped_callback(
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 479, in __call__
cvat_server                    |     ret: _R = await loop.run_in_executor(
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
cvat_server                    |     result = self.fn(*self.args, **self.kwargs)
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 538, in thread_handler
cvat_server                    |     return func(*args, **kwargs)
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
cvat_server                    |     return view_func(*args, **kwargs)
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view
cvat_server                    |     return self.dispatch(request, *args, **kwargs)
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
cvat_server                    |     response = self.handle_exception(exc)
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
cvat_server                    |     self.raise_uncaught_exception(exc)
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
cvat_server                    |     raise exc
cvat_server                    |   File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
cvat_server                    |     response = handler(request, *args, **kwargs)
cvat_server                    |   File "/home/django/cvat/apps/engine/views.py", line 1773, in annotations
cvat_server                    |     return self.export_annotations(
cvat_server                    |   File "/home/django/cvat/apps/engine/mixins.py", line 405, in export_annotations
cvat_server                    |     return export_func(db_instance=self._object,
cvat_server                    |   File "/home/django/cvat/apps/engine/views.py", line 3010, in _export_annotations
cvat_server                    |     if osp.exists(file_path):
cvat_server                    |   File "/usr/lib/python3.10/genericpath.py", line 19, in exists
cvat_server                    |     os.stat(path)
cvat_server                    | TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

Expected Behavior

No response

Possible Solution

I have found that when one goes to the CVAT UI and renewes all the jobs from a task and then refinishes all again the error is somehow solved:

Context

No response

Environment

I am using CVAT on premise with the following Docker Images:

REPOSITORY                                      TAG               IMAGE ID       CREATED         SIZE
cvat.pth.facebookresearch.sam.vit_h             latest            a503dbc4c6a3   2 months ago    11.6GB
cvat/ui                                         dev               6797de82abfc   2 months ago    152MB
cvat/server                                     dev               03e43ed6843a   2 months ago    2.77GB
postgres                                        15-alpine         d43df62c9cb9   3 months ago    237MB
grafana/grafana-oss                             10.1.2            31656ec60d2e   3 months ago    391MB
gcr.io/iguazio/alpine                           3.17              1e0b8b5322fc   5 months ago    7.06MB
quay.io/nuclio/handler-builder-python-onbuild   1.11.24-amd64     94caa75b7738   5 months ago    55.9MB
quay.io/nuclio/dashboard                        1.11.24-amd64     86a4ab0cb6f4   5 months ago    250MB
traefik                                         v2.9              d1e26b5f8193   9 months ago    138MB
clickhouse/clickhouse-server                    22.3-alpine       d4088d71d1dc   9 months ago    773MB
eqalpha/keydb                                   x86_64_v6.3.2     b9c88d8f9d06   11 months ago   129MB
timberio/vector                                 0.26.0-alpine     d8ecc9831523   13 months ago   122MB
openpolicyagent/opa                             0.45.0-rootless   8723f2dc306a   15 months ago   84.3MB
quay.io/nuclio/uhttpc                           0.0.1-amd64       5c59b3d31aa8   5 years ago     3.96MB

The text was updated successfully, but these errors were encountered:

bsekachev · 2024-01-15T07:57:58Z

Hello, we met the same error on previous release.
Try to use the latest one.

It is export problem, to workaround that, update "updated_time" field for a project/task/job (by changing a name, status, assignee, etc) and try to re-export.

alex-bronze-vision · 2024-01-15T10:54:59Z

I have updated CVAT server to latest version and the error is still happening:

ubuntu@ip-172-31-10-149:~/cvat$ docker images -a
REPOSITORY                                      TAG               IMAGE ID       CREATED         SIZE
cvat/ui                                         latest            4b4e76252f16   4 days ago      143MB
cvat/server                                     latest            c782b8c114c2   4 days ago      3.15GB
postgres                                        15-alpine         d492c6ccb0d1   3 weeks ago     240MB
cvat.pth.facebookresearch.sam.vit_h             latest            a503dbc4c6a3   2 months ago    11.6GB
cvat/ui                                         dev               6797de82abfc   2 months ago    152MB
cvat/server                                     dev               03e43ed6843a   2 months ago    2.77GB
postgres                                        <none>            d43df62c9cb9   3 months ago    237MB
grafana/grafana-oss                             10.1.2            31656ec60d2e   3 months ago    391MB
gcr.io/iguazio/alpine                           3.17              1e0b8b5322fc   5 months ago    7.06MB
quay.io/nuclio/handler-builder-python-onbuild   1.11.24-amd64     94caa75b7738   5 months ago    55.9MB
quay.io/nuclio/dashboard                        1.11.24-amd64     86a4ab0cb6f4   5 months ago    250MB
traefik                                         v2.9              d1e26b5f8193   9 months ago    138MB
clickhouse/clickhouse-server                    22.3-alpine       d4088d71d1dc   9 months ago    773MB
eqalpha/keydb                                   x86_64_v6.3.2     b9c88d8f9d06   11 months ago   129MB
timberio/vector                                 0.26.0-alpine     d8ecc9831523   13 months ago   122MB
openpolicyagent/opa                             0.45.0-rootless   8723f2dc306a   15 months ago   84.3MB
quay.io/nuclio/uhttpc                           0.0.1-amd64       5c59b3d31aa8   5 years ago     3.96MB

If this is not solved in cvat_server:latest I don't think that the issue should be closed...

bsekachev · 2024-01-15T11:14:18Z

docker ps?

alex-bronze-vision · 2024-01-15T14:52:05Z

Actually now, after updating I am unable to log into CVAT... I am facing the following issue with cvat_opa container.

Docker images:

ubuntu@ip-172-31-10-149:~/cvat$ docker images
REPOSITORY                                      TAG               IMAGE ID       CREATED         SIZE
cvat/ui                                         latest            4b4e76252f16   5 days ago      143MB
cvat/server                                     latest            c782b8c114c2   5 days ago      3.15GB
postgres                                        15-alpine         d492c6ccb0d1   3 weeks ago     240MB
ghcr.io/bronzetechnologies/training-pipeline    latest            3fc3b3070538   3 weeks ago     9.04GB
cvat.pth.facebookresearch.sam.vit_h             latest            a503dbc4c6a3   2 months ago    11.6GB
grafana/grafana-oss                             10.1.2            31656ec60d2e   3 months ago    391MB
gcr.io/iguazio/alpine                           3.17              1e0b8b5322fc   5 months ago    7.06MB
quay.io/nuclio/handler-builder-python-onbuild   1.11.24-amd64     94caa75b7738   5 months ago    55.9MB
quay.io/nuclio/dashboard                        1.11.24-amd64     86a4ab0cb6f4   5 months ago    250MB
traefik                                         v2.9              d1e26b5f8193   9 months ago    138MB
clickhouse/clickhouse-server                    22.3-alpine       d4088d71d1dc   9 months ago    773MB
eqalpha/keydb                                   x86_64_v6.3.2     b9c88d8f9d06   11 months ago   129MB
timberio/vector                                 0.26.0-alpine     d8ecc9831523   13 months ago   122MB
openpolicyagent/opa                             0.45.0-rootless   8723f2dc306a   15 months ago   84.3MB
quay.io/nuclio/uhttpc                           0.0.1-amd64       5c59b3d31aa8   5 years ago     3.96MB

Docker ps:

ubuntu@ip-172-31-10-149:~/cvat$ docker ps
CONTAINER ID   IMAGE                                      COMMAND                  CREATED          STATUS          PORTS                                                                                          NAMES
39c5e73bcc12   cvat/ui:latest                             "/docker-entrypoint.…"   12 minutes ago   Up 12 minutes   80/tcp                                                                                         cvat_ui
6b544eef9b55   timberio/vector:0.26.0-alpine              "/usr/local/bin/vect…"   12 minutes ago   Up 12 minutes                                                                                                  cvat_vector
4c7ef5a8c9b9   cvat/server:latest                         "./backend_entrypoin…"   12 minutes ago   Up 17 seconds   8080/tcp                                                                                       cvat_utils
296211a38408   cvat/server:latest                         "./backend_entrypoin…"   12 minutes ago   Up 12 minutes   8080/tcp                                                                                       cvat_worker_annotation
7072f3dbdfb0   cvat/server:latest                         "./backend_entrypoin…"   12 minutes ago   Up 12 minutes   8080/tcp                                                                                       cvat_worker_export
b5e1cdc8bf92   cvat/server:latest                         "./backend_entrypoin…"   12 minutes ago   Up 12 minutes   8080/tcp                                                                                       cvat_server
681faeb2071f   cvat/server:latest                         "./backend_entrypoin…"   12 minutes ago   Up 12 minutes   8080/tcp                                                                                       cvat_worker_webhooks
bc828dfbade3   cvat/server:latest                         "./backend_entrypoin…"   12 minutes ago   Up 12 minutes   8080/tcp                                                                                       cvat_worker_import
604810c150ee   cvat/server:latest                         "./backend_entrypoin…"   12 minutes ago   Up 12 minutes   8080/tcp                                                                                       cvat_worker_quality_reports
60945e9fad3d   cvat/server:latest                         "./backend_entrypoin…"   12 minutes ago   Up 12 minutes   8080/tcp                                                                                       cvat_worker_analytics_reports
ac79f4c94ea6   grafana/grafana-oss:10.1.2                 "sh -euc 'mkdir -p /…"   12 minutes ago   Up 12 minutes   3000/tcp                                                                                       cvat_grafana
499ee571ccec   eqalpha/keydb:x86_64_v6.3.2                "docker-entrypoint.s…"   12 minutes ago   Up 12 minutes   6379/tcp                                                                                       cvat_redis
c9bac549c396   traefik:v2.9                               "/entrypoint.sh --pr…"   12 minutes ago   Up 12 minutes   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 80/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp   traefik
72449150996d   clickhouse/clickhouse-server:22.3-alpine   "/entrypoint.sh"         12 minutes ago   Up 12 minutes   8123/tcp, 9000/tcp, 9009/tcp                                                                   cvat_clickhouse
2bf3f3f26804   postgres:15-alpine                         "docker-entrypoint.s…"   12 minutes ago   Up 12 minutes   5432/tcp                                                                                       cvat_db
0b0c5f3eef58   openpolicyagent/opa:0.45.0-rootless        "/opa run --server -…"   12 minutes ago   Up 12 minutes                                                                                                  cvat_opa

cvat_opa log:

ubuntu@ip-172-31-10-149:~/cvat$ docker logs cvat_opa 
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2024-01-15T14:38:48Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2024-01-15T14:38:48Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2024-01-15T14:38:48Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:38:49Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:38:50Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:38:50Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:38:52Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:38:54Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:38:57Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:39:01Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:39:08Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:39:19Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:39:33Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.26.0.10:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-01-15T14:39:51Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": net/http: timeout awaiting response headers","name":"cvat","plugin":"bundle","time":"2024-01-15T14:40:16Z"}

bsekachev · 2024-01-15T14:55:07Z

I do not think you updated correctly.

Container cvat_redis was removed in the latest release.
And containers cvat_redis_inmem and cvat_redis_ondisk appeared instead

bsekachev · 2024-01-15T15:00:06Z

Try doing docker compose down, git fetch origin, git checkout master and then docker compose up.

alex-bronze-vision · 2024-01-15T15:13:47Z

Ok, you were totally right, I was using an old version of the docker_compose file. Now I have been able to update to version 2.10 and the exportation issue seems to be solved:

Thanks for your patience @bsekachev .

alex-bronze-vision added the bug Something isn't working label Jan 11, 2024

bsekachev closed this as completed Jan 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Server Status 500 when exporting task annotations #7347

Server Status 500 when exporting task annotations #7347

alex-bronze-vision commented Jan 11, 2024 •

edited

Loading

bsekachev commented Jan 15, 2024 •

edited

Loading

alex-bronze-vision commented Jan 15, 2024

bsekachev commented Jan 15, 2024

alex-bronze-vision commented Jan 15, 2024 •

edited

Loading

bsekachev commented Jan 15, 2024

bsekachev commented Jan 15, 2024

alex-bronze-vision commented Jan 15, 2024

Server Status 500 when exporting task annotations #7347

Server Status 500 when exporting task annotations #7347

Comments

alex-bronze-vision commented Jan 11, 2024 • edited Loading

Actions before raising this issue

Steps to Reproduce

Expected Behavior

Possible Solution

Context

Environment

bsekachev commented Jan 15, 2024 • edited Loading

alex-bronze-vision commented Jan 15, 2024

bsekachev commented Jan 15, 2024

alex-bronze-vision commented Jan 15, 2024 • edited Loading

bsekachev commented Jan 15, 2024

bsekachev commented Jan 15, 2024

alex-bronze-vision commented Jan 15, 2024

alex-bronze-vision commented Jan 11, 2024 •

edited

Loading

bsekachev commented Jan 15, 2024 •

edited

Loading

alex-bronze-vision commented Jan 15, 2024 •

edited

Loading