404 page not found when loading CVAT UI or Traefik dashboard

[ricber]

My actions before raising this issue

• Read/searched the docs
• Searched past issues. In particular I read issues 404 Not found error when loading the browser UI for CVAT #4322 and 404 page not found error #4676

Steps to Reproduce (for bugs)

1. Install CVAT following the official guide
2. Run CVAT with docker compose up -d
3. Open CVAT UI at myIP:8081 (myIP is a real IP that is censored here. I've set the environment variable CVAT_HOST to myIP and I've changed the external port from 8080 to 8081 in the docker.compose.yml file)
4. Get error 404 page not found (I get the same error if I connect to the traefik dashboard at myIP:8090)

Possible Solution

I tried to set http proxy environment variables to empty as described in this comment, but it didn't solve the problem.

Your Environment

• Git hash commit (git log -1):

commit 7a7267d35c4c76a473f4bd24c804a8442483b6e6 (HEAD -> develop, origin/develop, origin/HEAD)
Author: Anastasia Yasakova <yasakova_anastasiya@mail.ru>
Date:   Fri Apr 7 13:30:27 2023 +0300

    Update REST API tests documentation (#5993)

• Docker version: Docker version 23.0.1
• Are you using Docker Swarm or Kubernetes? No
• Operating System and version: Ubuntu 20.04.5 LTS

docker-compose.yml:

# Copyright (C) 2018-2022 Intel Corporation
#
# SPDX-License-Identifier: MIT

services:
  cvat_db:
    container_name: cvat_db
    image: postgres:15-alpine
    restart: always
    environment:
      POSTGRES_USER: root
      POSTGRES_DB: cvat
      POSTGRES_HOST_AUTH_METHOD: trust
    volumes:
      - cvat_db:/var/lib/postgresql/data
    networks:
      - cvat

  cvat_redis:
    container_name: cvat_redis
    image: redis:4.0-alpine
    restart: always
    networks:
      - cvat

  cvat_server:
    container_name: cvat_server
    image: cvat/server:${CVAT_VERSION:-dev}
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
      - cvat_opa
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ''
      ALLOWED_HOSTS: '*'
      CVAT_REDIS_HOST: 'cvat_redis'
      CVAT_POSTGRES_HOST: 'cvat_db'
      ADAPTIVE_AUTO_ANNOTATION: 'false'
      IAM_OPA_BUNDLE: '1'
      no_proxy: clickhouse,grafana,vector,nuclio,opa,${no_proxy:-}
      NUMPROCS: 1
      DJANGO_LOG_SERVER_HOST: vector
      DJANGO_LOG_SERVER_PORT: 80
      CLICKHOUSE_HOST: clickhouse
      CVAT_ANALYTICS: 1
      CVAT_BASE_URL:
    command: -c supervisord/server.conf
    labels:
      - traefik.enable=true
      - traefik.http.services.cvat.loadbalancer.server.port=8080
      - traefik.http.routers.cvat.rule=Host(`${CVAT_HOST:-localhost}`) &&
        PathPrefix(`/api/`, `/git/`, `/opencv/`, `/static/`, `/admin`, `/documentation/`, `/django-rq`)
      - traefik.http.routers.cvat.entrypoints=web
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      cvat:
        aliases:
          - cvat-server

  cvat_utils:
    container_name: cvat_utils
    image: cvat/server:${CVAT_VERSION:-dev}
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
      - cvat_opa
    environment:
      CVAT_REDIS_HOST: 'cvat_redis'
      CVAT_REDIS_PASSWORD: ''
      CVAT_POSTGRES_HOST: 'cvat_db'
      CLICKHOUSE_HOST: clickhouse
      DJANGO_LOG_SERVER_HOST: vector
      DJANGO_LOG_SERVER_PORT: 80
      no_proxy: clickhouse,grafana,vector,nuclio,opa,${no_proxy:-}
    command: -c supervisord/utils.conf
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_worker_import:
    container_name: cvat_worker_import
    image: cvat/server:${CVAT_VERSION:-dev}
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
    environment:
      CVAT_REDIS_HOST: 'cvat_redis'
      CVAT_POSTGRES_HOST: 'cvat_db'
      DJANGO_LOG_SERVER_HOST: vector
      DJANGO_LOG_SERVER_PORT: 80
      no_proxy: clickhouse,grafana,vector,nuclio,opa,${no_proxy:-}
      NUMPROCS: 2
    command: -c supervisord/worker.import.conf
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_worker_export:
    container_name: cvat_worker_export
    image: cvat/server:${CVAT_VERSION:-dev}
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
    environment:
      CVAT_REDIS_HOST: 'cvat_redis'
      CVAT_POSTGRES_HOST: 'cvat_db'
      CLICKHOUSE_HOST: clickhouse
      DJANGO_LOG_SERVER_HOST: vector
      DJANGO_LOG_SERVER_PORT: 80
      no_proxy: clickhouse,grafana,vector,nuclio,opa,${no_proxy:-}
      NUMPROCS: 2
    command: -c supervisord/worker.export.conf
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_worker_annotation:
    container_name: cvat_worker_annotation
    image: cvat/server:${CVAT_VERSION:-dev}
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
      - cvat_opa
    environment:
      CVAT_REDIS_HOST: 'cvat_redis'
      CVAT_POSTGRES_HOST: 'cvat_db'
      DJANGO_LOG_SERVER_HOST: vector
      DJANGO_LOG_SERVER_PORT: 80
      no_proxy: clickhouse,grafana,vector,nuclio,opa,${no_proxy:-},--log.level=DEBUG
      NUMPROCS: 1
    command: -c supervisord/worker.annotation.conf
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_worker_webhooks:
    container_name: cvat_worker_webhooks
    image: cvat/server:${CVAT_VERSION:-dev}
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
      - cvat_opa
    environment:
      CVAT_REDIS_HOST: 'cvat_redis'
      CVAT_POSTGRES_HOST: 'cvat_db'
      DJANGO_LOG_SERVER_HOST: vector
      DJANGO_LOG_SERVER_PORT: 80
      no_proxy: clickhouse,grafana,vector,nuclio,opa,${no_proxy:-}
      NUMPROCS: 1
    command: -c supervisord/worker.webhooks.conf
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

  cvat_ui:
    container_name: cvat_ui
    image: cvat/ui:${CVAT_VERSION:-dev}
    restart: always
    depends_on:
      - cvat_server
    labels:
      - traefik.enable=true
      - traefik.http.services.cvat-ui.loadbalancer.server.port=80
      - traefik.http.routers.cvat-ui.rule=Host(`${CVAT_HOST:-localhost}`)
      - traefik.http.routers.cvat-ui.entrypoints=web
    networks:
      - cvat

  traefik:
    image: traefik:v2.9
    container_name: traefik
    restart: always
    command:
      - '--providers.docker.exposedByDefault=false'
      - '--providers.docker.network=cvat'
      - '--entryPoints.web.address=:8080'
      - '--providers.file.directory=/etc/traefik/rules'
    # Uncomment to get Traefik dashboard
      - "--entryPoints.dashboard.address=:8090"
      - "--api.dashboard=true"
      - "--log.level=DEBUG"
    labels:
      - traefik.enable=true
      - traefik.http.routers.dashboard.entrypoints=dashboard
      - traefik.http.routers.dashboard.service=api@internal
      - traefik.http.routers.dashboard.rule=Host(`${CVAT_HOST:-localhost}`)
    ports:
      - 8081:8080
      - 8090:8090
    environment:
      CVAT_HOST: ${CVAT_HOST:-localhost}
      DJANGO_LOG_VIEWER_HOST: grafana
      DJANGO_LOG_VIEWER_PORT: 3000
      http_proxy: ""
      https_proxy: ""
      HTTP_PROXY: ""
      HTTPS_PROXY: ""
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./components/analytics/grafana_conf.yml:/etc/traefik/rules/grafana_conf.yml:ro
    networks:
      - cvat

  cvat_opa:
    container_name: cvat_opa
    image: openpolicyagent/opa:0.45.0-rootless
    restart: always
    networks:
      cvat:
        aliases:
          - opa
    command:
      - run
      - --server
      - --log-level=error
      - --set=services.cvat.url=http://cvat-server:8080
      - --set=bundles.cvat.service=cvat
      - --set=bundles.cvat.resource=/api/auth/rules
      - --set=bundles.cvat.polling.min_delay_seconds=5
      - --set=bundles.cvat.polling.max_delay_seconds=15

  cvat_clickhouse:
    container_name: cvat_clickhouse
    image: clickhouse/clickhouse-server:22.3-alpine
    restart: always
    environment:
      - CLICKHOUSE_DB=cvat
      - CLICKHOUSE_USER=user
      - CLICKHOUSE_PASSWORD=user
    networks:
      cvat:
        aliases:
          - clickhouse
    volumes:
      - ./components/analytics/clickhouse/init.sh:/docker-entrypoint-initdb.d/init.sh:ro
      - cvat_events_db:/var/lib/clickhouse/

  cvat_vector:
    container_name: cvat_vector
    image: timberio/vector:0.26.0-alpine
    restart: always
    depends_on:
      - cvat_clickhouse
    environment:
      - CLICKHOUSE_DB=cvat
      - CLICKHOUSE_USER=user
      - CLICKHOUSE_PASSWORD=user
      - CLICKHOUSE_HOST=clickhouse
    networks:
      cvat:
        aliases:
          - vector
    volumes:
      - ./components/analytics/vector/vector.toml:/etc/vector/vector.toml:ro

  cvat_grafana:
    image: grafana/grafana-oss:9.3.6
    container_name: cvat_grafana
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_BASIC_ENABLED=false
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
      - GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=grafana-clickhouse-datasource
      - GF_SERVER_ROOT_URL=http://${CVAT_HOST:-localhost}/analytics
      - GF_INSTALL_PLUGINS=https://github.com/grafana/clickhouse-datasource/releases/download/v2.0.7/grafana-clickhouse-datasource-2.0.7.linux_amd64.zip;grafana-clickhouse-datasource
      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/all_events.json
    volumes:
      - ./components/analytics/grafana/dashboards/:/var/lib/grafana/dashboards/:ro
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
          - name: 'ClickHouse'
            type: 'grafana-clickhouse-datasource'
            isDefault: true
            jsonData:
              defaultDatabase: cvat
              port: 9000
              server: clickhouse
              username: user
              tlsSkipVerify: false
            secureJsonData:
              password: user
            editable: true
        EOF
        mkdir -p /etc/grafana/provisioning/dashboards
        cat <<EOF > /etc/grafana/provisioning/dashboards/dashboard.yaml
        apiVersion: 1
        providers:
          - name: cvat-logs
            type: file
            updateIntervalSeconds: 30
            options:
              path:  /var/lib/grafana/dashboards
              foldersFromFilesStructure: true
        EOF
        exec /run.sh
    networks:
      cvat:
        aliases:
          - grafana

volumes:
  cvat_db:
  cvat_data:
  cvat_keys:
  cvat_logs:
  cvat_events_db:

networks:
  cvat:

Other diagnostic information / logs:

• Health check output:

  docker exec -t cvat_server python manage.py health_check
  Cache backend: default   ... working
  Cache backend: media     ... working
  DatabaseBackend          ... working
  DiskUsage                ... working
  MemoryUsage              ... working
  MigrationsHealthCheck    ... working
  OPAHealthCheck           ... working
  

• Logs from traefik container:

time="2023-04-12T08:06:24Z" level=info msg="Configuration loaded from flags."
time="2023-04-12T08:06:24Z" level=info msg="Traefik version 2.9.10 built on 2023-04-06T16:15:08Z"
time="2023-04-12T08:06:24Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"dashboard\":{\"address\":\":8090\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"cvat\",\"swarmModeRefreshSeconds\":\"15s\"},\"file\":{\"directory\":\"/etc/traefik/rules\",\"watch\":true}},\"api\":{\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"}}"
time="2023-04-12T08:06:24Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2023-04-12T08:06:24Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-04-12T08:06:24Z" level=debug msg="Starting TCP Server" entryPointName=web
time="2023-04-12T08:06:24Z" level=debug msg="Starting TCP Server" entryPointName=dashboard
time="2023-04-12T08:06:24Z" level=info msg="Starting provider *file.Provider"
time="2023-04-12T08:06:24Z" level=debug msg="*file.Provider provider configuration: {\"directory\":\"/etc/traefik/rules\",\"watch\":true}"
time="2023-04-12T08:06:24Z" level=info msg="Starting provider *traefik.Provider"
time="2023-04-12T08:06:24Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-04-12T08:06:24Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-04-12T08:06:24Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-04-12T08:06:24Z" level=info msg="Starting provider *docker.Provider"
time="2023-04-12T08:06:24Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"cvat\",\"swarmModeRefreshSeconds\":\"15s\"}"
time="2023-04-12T08:06:24Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"grafana\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"analytics-auth\",\"strip-prefix\"],\"service\":\"grafana\",\"rule\":\"Host(`myIP`) \\u0026\\u0026 PathPrefix(`/analytics`)\"},\"grafana_https\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"analytics-auth\",\"strip-prefix\"],\"service\":\"grafana\",\"rule\":\"Host(`myIP`) \\u0026\\u0026 PathPrefix(`/analytics`)\",\"tls\":{}}},\"services\":{\"grafana\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://grafana:3000\"}],\"passHostHeader\":false}}},\"middlewares\":{\"analytics-auth\":{\"forwardAuth\":{\"address\":\"http://cvat_server:8080/analytics\",\"authRequestHeaders\":[\"Cookie\",\"Authorization\"]}},\"strip-prefix\":{\"stripPrefix\":{\"prefixes\":[\"/analytics\"],\"forceSlash\":true}}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=file
time="2023-04-12T08:06:24Z" level=debug msg="Configuration received: {\"http\":{\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2023-04-12T08:06:24Z" level=debug msg="Provider connection established with docker 23.0.1 (API 1.42)" providerName=docker
time="2023-04-12T08:06:24Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-04-12T08:06:24Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web routerName=grafana@file serviceName=grafana middlewareName=pipelining
time="2023-04-12T08:06:24Z" level=debug msg="Creating load-balancer" routerName=grafana@file serviceName=grafana entryPointName=web
time="2023-04-12T08:06:24Z" level=debug msg="Creating server 0 http://grafana:3000" serverName=0 entryPointName=web routerName=grafana@file serviceName=grafana
time="2023-04-12T08:06:24Z" level=debug msg="child http://grafana:3000 now UP"
time="2023-04-12T08:06:24Z" level=debug msg="Propagating new UP status"
time="2023-04-12T08:06:24Z" level=debug msg="Added outgoing tracing middleware grafana" routerName=grafana@file entryPointName=web middlewareType=TracingForwarder middlewareName=tracing
time="2023-04-12T08:06:24Z" level=debug msg="Creating middleware" routerName=grafana@file middlewareType=StripPrefix middlewareName=strip-prefix@file entryPointName=web
time="2023-04-12T08:06:24Z" level=debug msg="Adding tracing to middleware" middlewareName=strip-prefix@file entryPointName=web routerName=grafana@file
time="2023-04-12T08:06:24Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=analytics-auth@file middlewareType=ForwardedAuthType routerName=grafana@file
time="2023-04-12T08:06:24Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-04-12T08:06:24Z" level=error msg="entryPoint \"websecure\" doesn't exist" entryPointName=websecure routerName=grafana_https@file
time="2023-04-12T08:06:24Z" level=error msg="no valid entryPoint for this router" routerName=grafana_https@file
time="2023-04-12T08:06:24Z" level=error msg="entryPoint \"websecure\" doesn't exist" routerName=grafana_https@file entryPointName=websecure
time="2023-04-12T08:06:24Z" level=error msg="no valid entryPoint for this router" routerName=grafana_https@file

• Logs from cvat_grafana container:

[...]
logger=plugin.loader t=2023-04-12T08:08:11.525539043Z level=info msg="Plugin registered" pluginID=input
logger=plugin.loader t=2023-04-12T08:08:11.617260769Z level=info msg="Plugin registered" pluginID=grafana-clickhouse-datasource
logger=plugin.grafana-clickhouse-datasource t=2023-04-12T08:08:11.637458012Z level=info msg=Profiler enabled=false
logger=secrets t=2023-04-12T08:08:11.640151582Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=query_data t=2023-04-12T08:08:11.645661542Z level=info msg="Query Service initialization"
logger=live.push_http t=2023-04-12T08:08:11.652723106Z level=info msg="Live Push Gateway initialization"
logger=infra.usagestats.collector t=2023-04-12T08:08:12.027922486Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=provisioning.datasources t=2023-04-12T08:08:12.028862138Z level=info msg="inserting datasource from configuration " name=ClickHouse uid=PDEE91DDB90597936
logger=provisioning.alerting t=2023-04-12T08:08:12.145789806Z level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2023-04-12T08:08:12.14582473Z level=info msg="finished to provision alerting"
logger=grafanaStorageLogger t=2023-04-12T08:08:12.146051628Z level=info msg="storage starting"
logger=ngalert.state.manager t=2023-04-12T08:08:12.146325631Z level=info msg="Warming state cache for startup"
logger=http.server t=2023-04-12T08:08:12.14878016Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl=/analytics socket=
logger=sqlstore.transactions t=2023-04-12T08:08:12.157986766Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked"
logger=sqlstore.transactions t=2023-04-12T08:08:12.282947914Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked"
logger=ngalert.state.manager t=2023-04-12T08:08:12.597489565Z level=info msg="State cache has been initialized" states=0 duration=451.162049ms
logger=ticker t=2023-04-12T08:08:12.5975596Z level=info msg=starting first_tick=2023-04-12T08:08:20Z
logger=ngalert.multiorg.alertmanager t=2023-04-12T08:08:12.597548613Z level=info msg="starting MultiOrg Alertmanager"
logger=provisioning t=2023-04-12T08:08:12.598465457Z level=error msg="Failed to provision dashboard" error="Failed to provision dashboards: failed to provision config cvat-logs: open /var/lib/grafana/dashboards: permission denied"
logger=server t=2023-04-12T08:08:12.598531324Z level=error msg="Stopped background service" service=*provisioning.ProvisioningServiceImpl reason="Failed to provision dashboards: failed to provision config cvat-logs: open /var/lib/grafana/dashboards: permission denied"
logger=tracing t=2023-04-12T08:08:12.598606048Z level=info msg="Closing tracing"
logger=ticker t=2023-04-12T08:08:12.598681429Z level=info msg=stopped last_tick=2023-04-12T08:08:10Z
logger=sqlstore.transactions t=2023-04-12T08:08:12.60952642Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked"
logger=plugin.grafana-clickhouse-datasource t=2023-04-12T08:08:12.660044372Z level=info msg=Profiler enabled=false
logger=datasources t=2023-04-12T08:08:12.913069058Z level=error msg="Failed getting data source" err="context canceled" uid= id=1 name= orgId=1
logger=secret.migration t=2023-04-12T08:08:12.913581581Z level=error msg="Stopped secret migration service" service=*migrations.DataSourceSecretMigrationService reason="context canceled"
logger=infra.lockservice t=2023-04-12T08:08:12.913728236Z level=error msg="Failed to release the lock" error="context canceled"
logger=server t=2023-04-12T08:08:12.913771687Z level=error msg="Server shutdown" error="*provisioning.ProvisioningServiceImpl run error: Failed to provision dashboards: failed to provision config cvat-logs: open /var/lib/grafana/dashboards: permission denied"
*provisioning.ProvisioningServiceImpl run error: Failed to provision dashboards: failed to provision config cvat-logs: open /var/lib/grafana/dashboards: permission denied

• Logs from cvat_opa container:

{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:24Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:24Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:25Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:25Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:25Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:26Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:27Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:29Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.13:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:34Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.13:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:38Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.13:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:44Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.13:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-04-12T08:06:56Z"}

[dmunro890]

I am also experiencing a very similar issue. Log outputs are very similar, other than the db ones, I don't have permission denied errors, but everything else is presenting similarly.

[fernandorovai]

Any news on this? I'm facing the same problem when trying to expose to external IP in another port.

[amine-ammor]

Hello, I had exactly the same issue , I fixed it by opening the terminal as root , it resolved the "permission denied error", and enabled me to get the login page.

I think that in the "Dockerfile" some commands that must be executed as root, are not prepended with sudo.

[ricber]

@amine-ammor thank you for the suggestion! Indeed, running docker compose with sudo solved the problem. However, I've recently checkout to the last release (v2.4.3) and I didn't need to run docker compose with sudo.

[ntdat017]

I have the same problem, and run docker compose with sudo but it's not work

[amine-ammor]

Hi,
In fact, the proposed solution, was to start the terminal as root, or equivalently, to type the 'su' command, (to have root privileges for the whole session), then to follow the installation's instructions.

I didn't read the source code, I can't specify the exact reason, why it worked, it was just a guess.
Good luck.

[zhentai-SMU]

Hi, I had the same issue and was able to resolve it. Hopefully, this solution will work for you too. I believe the reason for the "404 Not Found" error is that I only set a temporary environment variable during the installation of CVAT. When I restarted the Docker container or took some other wrong steps, the CVAT_HOST stopped working. To fix this, I simply reset the CVAT_HOST and saved it in the config file.

[InoriYuzuriha0]

Any solution for this issue? I had exactly same from this issue

[Mactarvish]

same issue

[shashwat-kalpvaig]

Still facing the same issue with nginx proxy_pass it to port 8080. The header on title bar comes up but all the assets and js are not loaded properly.