Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problems deploying to corporate OpenShift platform #3992

Closed
2 tasks done
nerdinand opened this issue Dec 7, 2021 · 2 comments
Closed
2 tasks done

Problems deploying to corporate OpenShift platform #3992

nerdinand opened this issue Dec 7, 2021 · 2 comments
Labels
question Further information is requested
Projects
Infrastructure

Comments

@nerdinand
Copy link
Contributor

My actions before raising this issue

We encountered various problems trying to deploy the latest version of CVAT to our corporate OpenShift environment.

Expected Behaviour

There is a straightforward and working way to deploy CVAT on OpenShift.

Current Behaviour

There is no easy way of deploying CVAT to OpenShift.

Possible Solution

Test deployment of the Helm chart to OpenShift and make improvements for a seamless installation.

Context

Our company has a corporate OpenShift infrastructure (OpenShift Version 4.7, Kubernetes 1.21.5) on which we would like to run CVAT. Earlier this year, when CVAT version 1.2.0 was the current one, we managed to translate the docker-compose configuration into an OpenShift template with which we could successfully deploy CVAT, albeit with modifications to the Dockerfiles.

In a follow-up project, we aimed to use the official Docker images and Helm chart to deploy the most up-to-date release.
However, there were a couple of new problems, which prevented us from achieving this. Some of the problems could be addressed with changes to:

  • Dockerfiles
  • supervisord configuration
  • Helm chart

But in the end, we did not succeed in deploying.

The frontend application is reachable (the UI loads) but communication with the backend is not possible. I.e. 503 Service Unavailable response when trying to reach:

  • /api/v1/restrictions/user-agreements
  • /api/v1/users/self
  • /api/v1/auth/password/change
  • /api/v1/auth/password/reset

Here's a snippet of the backend log:

Logs from `cvat` container 2021-12-07 09:37:52,839 INFO RPC interface 'supervisor' initialized 2021-12-07 09:37:52,839 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2021-12-07 09:37:52,840 INFO supervisord started with pid 1 2021-12-07 09:37:53,844 INFO spawned: 'ssh-agent' with pid 8 2021-12-07 09:37:53,848 INFO spawned: 'clamav_update' with pid 9 2021-12-07 09:37:53,851 INFO spawned: 'git_status_updater' with pid 11 2021-12-07 09:37:53,853 INFO spawned: 'rqscheduler' with pid 12 2021-12-07 09:37:53,856 INFO spawned: 'rqworker_default_0' with pid 13 2021-12-07 09:37:53,859 INFO spawned: 'rqworker_default_1' with pid 18 2021-12-07 09:37:53,878 INFO spawned: 'rqworker_low' with pid 26 2021-12-07 09:37:53,882 INFO spawned: 'runserver' with pid 33 2021-12-07 09:37:53,883 DEBG fd 10 closed, stopped monitoring (stdout)> 2021-12-07 09:37:53,883 DEBG fd 14 closed, stopped monitoring (stderr)> 2021-12-07 09:37:53,883 INFO exited: clamav_update (exit status 0; not expected) 2021-12-07 09:37:53,885 DEBG received SIGCHLD indicating a child quit 2021-12-07 09:37:53,885 DEBG 'ssh-agent' stdout output: SSH_AUTH_SOCK=/tmp/ssh-agent.sock; export SSH_AUTH_SOCK; echo Agent pid 8;

2021-12-07 09:37:53,885 DEBG 'ssh-agent' stderr output:
debug2: fd 3 setting O_NONBLOCK

2021-12-07 09:37:53,885 DEBG 'git_status_updater' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout
wait-for-it.sh: cvat-redis-master:6379 is available after 0 seconds
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:37:53,885 DEBG 'rqscheduler' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout
wait-for-it.sh: cvat-redis-master:6379 is available after 0 seconds

2021-12-07 09:37:53,885 DEBG 'rqworker_default_0' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout
wait-for-it.sh: cvat-redis-master:6379 is available after 0 seconds
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:37:53,885 DEBG 'rqworker_default_1' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-12-07 09:37:53,885 DEBG 'rqscheduler' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:37:53,977 DEBG 'git_status_updater' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:37:53,977 DEBG 'rqworker_default_0' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:37:53,978 DEBG 'rqscheduler' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:37:53,980 DEBG 'rqworker_default_1' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 0 seconds

2021-12-07 09:37:53,981 DEBG 'rqworker_low' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-12-07 09:37:54,076 DEBG 'rqworker_default_1' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:37:54,078 DEBG 'rqworker_default_1' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:37:54,078 DEBG 'rqworker_low' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 1 seconds

2021-12-07 09:37:54,079 DEBG 'runserver' stderr output:
wait-for-it.sh: waiting for cvat-postgresql:5432 without a timeout

2021-12-07 09:37:54,080 DEBG 'rqworker_low' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:37:54,081 DEBG 'rqworker_low' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:37:54,178 DEBG 'runserver' stderr output:
wait-for-it.sh: cvat-postgresql:5432 is available after 0 seconds

2021-12-07 09:37:54,179 DEBG 'runserver' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:37:54,181 DEBG 'runserver' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:37:55,182 INFO success: ssh-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:37:55,277 INFO spawned: 'clamav_update' with pid 68
2021-12-07 09:37:55,277 INFO success: git_status_updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:37:55,277 INFO success: rqscheduler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:37:55,277 INFO success: rqworker_default_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:37:55,277 INFO success: rqworker_default_1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:37:55,277 INFO success: rqworker_low entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:37:55,277 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:37:55,281 DEBG 'rqscheduler' stderr output:
09:37:55 Registering birth

2021-12-07 09:37:55,283 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:37:55,288 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140269891390960 for <Subprocess at 140269891338784 with name clamav_update in state STARTING> (stdout)>
2021-12-07 09:37:55,288 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140269891092640 for <Subprocess at 140269891338784 with name clamav_update in state STARTING> (stderr)>
2021-12-07 09:37:55,288 INFO exited: clamav_update (exit status 0; not expected)
2021-12-07 09:37:55,288 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:37:55,380 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:37:55,388 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:37:55,390 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:37:55,391 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:37:55,391 DEBG 'git_status_updater' stderr output:
Identity added: /home/django/.ssh/id_rsa (1001570000@cvat-backend-5c78c55764-t97m7)

2021-12-07 09:37:55,396 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:37:55,397 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:37:55,399 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:37:55,479 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:37:55,485 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:37:55,486 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:37:55,489 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:37:55,578 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:37:55,583 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:37:55,584 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:37:55,586 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:37:55,591 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:37:55,679 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:37:55,680 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:37:55,682 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:37:55,777 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:37:57,782 INFO spawned: 'clamav_update' with pid 90
2021-12-07 09:37:57,879 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140269891339456 for <Subprocess at 140269891338784 with name clamav_update in state STARTING> (stdout)>
2021-12-07 09:37:57,879 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140269891092784 for <Subprocess at 140269891338784 with name clamav_update in state STARTING> (stderr)>
2021-12-07 09:37:57,879 INFO exited: clamav_update (exit status 0; not expected)
2021-12-07 09:37:57,879 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:38:00,977 INFO spawned: 'clamav_update' with pid 153
2021-12-07 09:38:01,096 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140269891390960 for <Subprocess at 140269891338784 with name clamav_update in state STARTING> (stdout)>
2021-12-07 09:38:01,096 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140269891092880 for <Subprocess at 140269891338784 with name clamav_update in state STARTING> (stderr)>
2021-12-07 09:38:01,096 INFO exited: clamav_update (exit status 0; not expected)
2021-12-07 09:38:01,096 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:38:02,098 INFO gave up: clamav_update entered FATAL state, too many start retries too quickly
2021-12-07 09:38:34,477 DEBG fd 35 closed, stopped monitoring <POutputDispatcher at 140269891390720 for <Subprocess at 140269891390672 with name rqworker_low in state RUNNING> (stdout)>
2021-12-07 09:38:34,478 DEBG fd 39 closed, stopped monitoring <POutputDispatcher at 140269891094848 for <Subprocess at 140269891390672 with name rqworker_low in state RUNNING> (stderr)>
2021-12-07 09:38:34,478 INFO exited: rqworker_low (terminated by SIGKILL; not expected)
2021-12-07 09:38:34,478 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:38:35,680 INFO spawned: 'rqworker_low' with pid 598
2021-12-07 09:38:35,781 DEBG 'rqworker_low' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-12-07 09:38:35,878 DEBG 'rqworker_low' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 0 seconds

2021-12-07 09:38:35,879 DEBG 'rqworker_low' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:38:35,881 DEBG 'rqworker_low' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:38:36,882 INFO success: rqworker_low entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:38:37,381 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:38:37,383 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:38:37,477 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:38:37,482 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:38:39,579 DEBG fd 25 closed, stopped monitoring <POutputDispatcher at 140269891390432 for <Subprocess at 140269891390384 with name rqworker_default_0 in state RUNNING> (stdout)>
2021-12-07 09:38:39,579 DEBG fd 29 closed, stopped monitoring <POutputDispatcher at 140269891093984 for <Subprocess at 140269891390384 with name rqworker_default_0 in state RUNNING> (stderr)>
2021-12-07 09:38:39,579 INFO exited: rqworker_default_0 (terminated by SIGKILL; not expected)
2021-12-07 09:38:39,579 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:38:40,677 INFO spawned: 'rqworker_default_0' with pid 615
2021-12-07 09:38:40,779 DEBG 'rqworker_default_0' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-12-07 09:38:40,876 DEBG 'rqworker_default_0' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 0 seconds

2021-12-07 09:38:40,878 DEBG 'rqworker_default_0' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:38:40,880 DEBG 'rqworker_default_0' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:38:41,881 INFO success: rqworker_default_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:38:42,880 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:38:42,976 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:38:42,980 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:38:45,380 DEBG 'runserver' stderr output:
bash: [33: 1 (255)] tcsetattr: Inappropriate ioctl for device

2021-12-07 09:38:45,380 DEBG fd 40 closed, stopped monitoring <POutputDispatcher at 140269891390912 for <Subprocess at 140269891390864 with name runserver in state RUNNING> (stdout)>
2021-12-07 09:38:45,380 DEBG fd 44 closed, stopped monitoring <POutputDispatcher at 140269891095280 for <Subprocess at 140269891390864 with name runserver in state RUNNING> (stderr)>
2021-12-07 09:38:45,380 INFO exited: runserver (exit status 137; not expected)
2021-12-07 09:38:45,380 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:38:45,483 INFO spawned: 'runserver' with pid 628
2021-12-07 09:38:45,681 DEBG 'runserver' stderr output:
wait-for-it.sh: waiting for cvat-postgresql:5432 without a timeout

2021-12-07 09:38:45,779 DEBG 'runserver' stderr output:
wait-for-it.sh: cvat-postgresql:5432 is available after 0 seconds

2021-12-07 09:38:45,878 DEBG 'runserver' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:38:45,976 DEBG 'runserver' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:38:46,977 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:38:48,579 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:38:48,680 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:38:48,682 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:38:48,879 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:38:56,887 INFO reaped unknown pid 438
2021-12-07 09:38:56,887 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:39:05,076 INFO reaped unknown pid 472
2021-12-07 09:39:05,077 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:39:08,080 INFO reaped unknown pid 470
2021-12-07 09:39:08,080 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:39:18,776 DEBG 'rqworker_default_1' stderr output:
System check identified some issues:

WARNINGS:
?: (urls.W005) URL namespace 'v1' isn't unique. You may not be able to reverse all URLs in this namespace
DEBUG - 2021-12-07 09:39:18,688 - worker - Registering birth of worker 5ea1fb50d2a740d2830340a84fe4aed6

2021-12-07 09:39:18,779 DEBG 'rqworker_default_1' stderr output:
INFO - 2021-12-07 09:39:18,779 - worker - Worker rq:worker:5ea1fb50d2a740d2830340a84fe4aed6: started, version 1.5.1

2021-12-07 09:39:18,779 DEBG 'rqworker_default_1' stderr output:
INFO - 2021-12-07 09:39:18,779 - worker - *** Listening on default...

2021-12-07 09:39:18,780 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-12-07 09:39:18,779 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-12-07 09:39:18,780 DEBG 'rqworker_default_1' stderr output:
INFO - 2021-12-07 09:39:18,780 - worker - Cleaning registries for queue: default

2021-12-07 09:39:18,782 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-12-07 09:39:18,781 - worker - *** Listening on default...

2021-12-07 09:39:18,782 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-12-07 09:39:18,782 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-12-07 09:39:22,480 DEBG fd 30 closed, stopped monitoring <POutputDispatcher at 140269891390528 for <Subprocess at 140269891390480 with name rqworker_default_1 in state RUNNING> (stdout)>
2021-12-07 09:39:22,480 DEBG fd 34 closed, stopped monitoring <POutputDispatcher at 140269891094416 for <Subprocess at 140269891390480 with name rqworker_default_1 in state RUNNING> (stderr)>
2021-12-07 09:39:22,480 INFO exited: rqworker_default_1 (terminated by SIGKILL; not expected)
2021-12-07 09:39:22,480 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:39:23,581 INFO spawned: 'rqworker_default_1' with pid 1013
2021-12-07 09:39:23,676 DEBG 'rqworker_default_1' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-12-07 09:39:23,689 DEBG 'rqworker_default_1' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 0 seconds

2021-12-07 09:39:23,690 DEBG 'rqworker_default_1' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:39:23,777 DEBG 'rqworker_default_1' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:39:24,778 INFO success: rqworker_default_1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:39:24,979 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:39:24,982 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:39:24,984 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:39:25,078 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:39:30,279 DEBG fd 15 closed, stopped monitoring <POutputDispatcher at 140269891530656 for <Subprocess at 140269891390000 with name git_status_updater in state RUNNING> (stdout)>
2021-12-07 09:39:30,279 DEBG fd 19 closed, stopped monitoring <POutputDispatcher at 140269891093120 for <Subprocess at 140269891390000 with name git_status_updater in state RUNNING> (stderr)>
2021-12-07 09:39:30,279 INFO exited: git_status_updater (terminated by SIGKILL; not expected)
2021-12-07 09:39:30,279 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:39:31,285 INFO spawned: 'git_status_updater' with pid 1090
2021-12-07 09:39:31,478 DEBG 'git_status_updater' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-12-07 09:39:31,483 DEBG 'git_status_updater' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 0 seconds

2021-12-07 09:39:31,577 DEBG 'git_status_updater' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:39:31,579 DEBG 'git_status_updater' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:39:32,580 INFO success: git_status_updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:39:32,781 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:39:32,876 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:39:32,879 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:39:32,884 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:39:38,977 DEBG 'rqworker_low' stderr output:
System check identified some issues:

WARNINGS:
?: (urls.W005) URL namespace 'v1' isn't unique. You may not be able to reverse all URLs in this namespace
DEBUG - 2021-12-07 09:39:38,881 - worker - Registering birth of worker 5c8d9528f8b64e27b5284c95053e3dc2

2021-12-07 09:39:38,979 DEBG 'rqworker_low' stderr output:
INFO - 2021-12-07 09:39:38,979 - worker - Worker rq:worker:5c8d9528f8b64e27b5284c95053e3dc2: started, version 1.5.1

2021-12-07 09:39:38,979 DEBG 'rqworker_low' stderr output:
INFO - 2021-12-07 09:39:38,979 - worker - *** Listening on low...

2021-12-07 09:39:38,980 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-12-07 09:39:38,979 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-12-07 09:39:38,981 DEBG 'rqworker_low' stderr output:
INFO - 2021-12-07 09:39:38,981 - worker - Cleaning registries for queue: low

2021-12-07 09:39:38,982 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-12-07 09:39:38,982 - worker - *** Listening on low...

2021-12-07 09:39:38,983 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-12-07 09:39:38,983 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-12-07 09:39:53,179 DEBG 'rqworker_default_0' stderr output:
System check identified some issues:

WARNINGS:
?: (urls.W005) URL namespace 'v1' isn't unique. You may not be able to reverse all URLs in this namespace
DEBUG - 2021-12-07 09:39:53,178 - worker - Registering birth of worker 85fd83ba56464724aa8087948cdd0b2f

2021-12-07 09:39:53,276 DEBG 'rqworker_default_0' stderr output:
INFO - 2021-12-07 09:39:53,276 - worker - Worker rq:worker:85fd83ba56464724aa8087948cdd0b2f: started, version 1.5.1

2021-12-07 09:39:53,276 DEBG 'rqworker_default_0' stderr output:
INFO - 2021-12-07 09:39:53,276 - worker - *** Listening on default...

2021-12-07 09:39:53,277 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-12-07 09:39:53,277 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-12-07 09:39:53,278 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-12-07 09:39:53,278 - worker - *** Listening on default...

2021-12-07 09:39:53,278 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-12-07 09:39:53,278 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-12-07 09:39:55,277 DEBG fd 17 closed, stopped monitoring <POutputDispatcher at 140269891390720 for <Subprocess at 140269891390384 with name rqworker_default_0 in state RUNNING> (stdout)>
2021-12-07 09:39:55,278 DEBG fd 25 closed, stopped monitoring <POutputDispatcher at 140269891094128 for <Subprocess at 140269891390384 with name rqworker_default_0 in state RUNNING> (stderr)>
2021-12-07 09:39:55,278 INFO exited: rqworker_default_0 (terminated by SIGKILL; not expected)
2021-12-07 09:39:55,278 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:39:56,283 INFO spawned: 'rqworker_default_0' with pid 1339
2021-12-07 09:39:56,378 DEBG 'rqworker_default_0' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-12-07 09:39:56,382 DEBG 'rqworker_default_0' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 0 seconds

2021-12-07 09:39:56,384 DEBG 'rqworker_default_0' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:39:56,385 DEBG 'rqworker_default_0' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:39:57,277 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:39:57,279 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:39:57,281 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:39:57,281 INFO success: rqworker_default_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:39:57,285 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-12-07 09:40:05,380 DEBG 'runserver' stderr output:
bash: [628: 1 (255)] tcsetattr: Inappropriate ioctl for device

2021-12-07 09:40:05,381 DEBG fd 29 closed, stopped monitoring <POutputDispatcher at 140269891390432 for <Subprocess at 140269891390864 with name runserver in state RUNNING> (stdout)>
2021-12-07 09:40:05,381 DEBG fd 39 closed, stopped monitoring <POutputDispatcher at 140269891095424 for <Subprocess at 140269891390864 with name runserver in state RUNNING> (stderr)>
2021-12-07 09:40:05,381 INFO exited: runserver (exit status 137; not expected)
2021-12-07 09:40:05,381 DEBG received SIGCHLD indicating a child quit
2021-12-07 09:40:05,476 INFO spawned: 'runserver' with pid 1419
2021-12-07 09:40:05,578 DEBG 'runserver' stderr output:
wait-for-it.sh: waiting for cvat-postgresql:5432 without a timeout

2021-12-07 09:40:05,583 DEBG 'runserver' stderr output:
wait-for-it.sh: cvat-postgresql:5432 is available after 0 seconds

2021-12-07 09:40:05,584 DEBG 'runserver' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-12-07 09:40:05,676 DEBG 'runserver' stderr output:
groups: cannot find name for group ID 1001570000

2021-12-07 09:40:06,677 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-12-07 09:40:06,680 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-12-07 09:40:06,682 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-12-07 09:40:06,685 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:3Wb+ERbyC9zhJ111sBIMulVrMw4hpSjtSpbXRNBQwnY "1001570000@cvat-backend-5c78c55764-t97m7" (life: 0) (confirm: 0) (provider: none)

2021-12-07 09:40:06,778 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

The logs are quite hard to read due to the large number of different processes. It's hard to debug where the real problem lies, but it seems the runserver process is stuck in a restart loop for some reason. Also, clamav doesn't seem to work, as indicated by supervisorctl:

$ supervisorctl status
clamav_update                         FATAL     Exited too quickly (process log may have details)
git_status_updater                    RUNNING   pid 4657, uptime 0:00:48
rqscheduler                           RUNNING   pid 12, uptime 0:08:47
rqworker_default:rqworker_default_0   RUNNING   pid 4869, uptime 0:00:26
rqworker_default:rqworker_default_1   RUNNING   pid 4578, uptime 0:00:56
rqworker_low                          RUNNING   pid 598, uptime 0:08:05
runserver                             RUNNING   pid 4947, uptime 0:00:17
ssh-agent                             RUNNING   pid 8, uptime 0:08:47

Your Environment

  • Git hash: 967b0fee2c1c5028ad5f8e888b4eb5c138fe2b1e (v1.7.0)
  • Docker 20.10.11

Next steps

You may join our Gitter channel for community support.
@azhavoro
Copy link
Contributor

azhavoro commented Dec 7, 2021

Hi, have you set any memory limits? It look likes as OOM issue...

@nerdinand
Copy link
Contributor Author

@azhavoro How embarrassing, you are completely right. I thought I'd tried with more memory, but apparently I was wrong. Seems to work now...

@nerdinand nerdinand mentioned this issue Dec 7, 2021
Merged
5 tasks
@nmanovic nmanovic added the question Further information is requested label Dec 8, 2021
@nmanovic nmanovic added this to To do in Infrastructure via automation Dec 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
Infrastructure
To do
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nerdinand @bsekachev @nmanovic @azhavoro