Clickhouse Connection Refused

[CactusChaos]

Hi,
I am unable to access the web portal after install. On checking the logs I am getting:

poeticmetric-worker-1  | goroutine 1 [running]:
poeticmetric-worker-1  | log.Panicln({0x40003dfea0?, 0x2a?, 0x400050e240?})
poeticmetric-worker-1  |        /usr/local/go/src/log/log.go:446 +0x60
poeticmetric-worker-1  | main.main()
poeticmetric-worker-1  |        /usr/src/poeticmetric/cmd/migrator/main.go:69 +0x360
poeticmetric-worker-1  | 2023/08/27 01:30:44 dial tcp 172.24.0.3:9000: connect: connection refused
poeticmetric-worker-1  | panic: dial tcp 172.24.0.3:9000: connect: connection refused

I have a couple of the external ports remapped due to already existing services running on them.

My .env file:

# CLICKHOUSE
CLICKHOUSE_DATABASE=poeticmetric
CLICKHOUSE_HOST=clickhouse
CLICKHOUSE_PASSWORD=##
CLICKHOUSE_TCP_PORT=9000
CLICKHOUSE_USER=poeticmetric

# DOCKER COMPOSE
COMPOSE_PROJECT_NAME=poeticmetric

# GOOGLE
GOOGLE_CLIENT_ID=
GOOGLE_CLIENT_SECRET=

# POETICMETRIc
FRONTEND_BASE_URL=http://localhost:8000
REST_API_BASE_URL=http://localhost:8001
VERSION=latest
WORKER_COUNT=2

# POSTGRES
POSTGRES_DATABASE=poeticmetric
POSTGRES_HOST=postgres
POSTGRES_PASSWORD=##
POSTGRES_PORT=5432
POSTGRES_USER=poeticmetric

# RABBITMQ
RABBITMQ_HOST=rabbitmq
RABBITMQ_PASSWORD=##
RABBITMQ_PORT=5672
RABBITMQ_USER=poeticmetric
RABBITMQ_VHOST=poeticmetric

# REDIS
REDIS_HOST=redis
REDIS_PASSWORD=##
REDIS_PORT=6379

# SMTP
SMTP_FROM=poeticmetric
SMTP_HOST=smtp.zoho.com.au
SMTP_PASSWORD=##
SMTP_PORT=465
SMTP_USER=##

Docker compose:

version: "3.9"

x-backend-common: &backend-common
  depends_on:
    - clickhouse
    - postgres
    - rabbitmq
    - redis
  image: "poeticmetric/backend:${VERSION}"
  environment: &backend-common-environment
    CLICKHOUSE_DATABASE: ${CLICKHOUSE_DATABASE}
    CLICKHOUSE_HOST: ${CLICKHOUSE_HOST}
    CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
    CLICKHOUSE_TCP_PORT: ${CLICKHOUSE_TCP_PORT}
    CLICKHOUSE_USER: ${CLICKHOUSE_USER}
    FRONTEND_BASE_URL: ${FRONTEND_BASE_URL}
    POSTGRES_DATABASE: ${POSTGRES_DATABASE}
    POSTGRES_HOST: ${POSTGRES_HOST}
    POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    POSTGRES_PORT: ${POSTGRES_PORT}
    POSTGRES_USER: ${POSTGRES_USER}
    RABBITMQ_HOST: ${RABBITMQ_HOST}
    RABBITMQ_PASSWORD: ${RABBITMQ_PASSWORD}
    RABBITMQ_PORT: ${RABBITMQ_PORT}
    RABBITMQ_USER: ${RABBITMQ_USER}
    RABBITMQ_VHOST: ${RABBITMQ_VHOST}
    REDIS_HOST: ${REDIS_HOST}
    REDIS_PASSWORD: ${REDIS_PASSWORD}
    REDIS_PORT: ${REDIS_PORT}
    REST_API_BASE_URL: ${REST_API_BASE_URL}
    SMTP_FROM: ${SMTP_FROM}
    SMTP_HOST: ${SMTP_HOST}
    SMTP_PASSWORD: ${SMTP_PASSWORD}
    SMTP_PORT: ${SMTP_PORT}
    SMTP_USER: ${SMTP_USER}
    WORKER_COUNT: ${WORKER_COUNT}
  restart: unless-stopped
  tty: true

networks:
  default:
    name: ${COMPOSE_PROJECT_NAME}

volumes:
  postgres:
  clickhouse:

services:
  clickhouse:
    environment:
      CLICKHOUSE_DB: ${CLICKHOUSE_DATABASE}
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
      CLICKHOUSE_USER: ${CLICKHOUSE_USER}
    image: clickhouse/clickhouse-server:latest-alpine
    volumes:
      - clickhouse:/var/lib/clickhouse
    ports:
      - "9002:9000"

  frontend:
    environment:
      FRONTEND_BASE_URL: ${FRONTEND_BASE_URL}
      REST_API_BASE_URL: ${REST_API_BASE_URL}
    image: "poeticmetric/frontend:${VERSION}"
    ports:
      - "8002:8000"

  postgres:
    environment:
      POSTGRES_DB: ${POSTGRES_DATABASE}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_USER: ${POSTGRES_USER}
    image: postgres:15-alpine
    volumes:
      - postgres:/var/lib/postgresql/data
    ports:
      - "5434:5432"

  rabbitmq:
    environment:
      RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
      RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
      RABBITMQ_DEFAULT_VHOST: ${RABBITMQ_VHOST}
    image: rabbitmq:3-management-alpine

  redis:
    command:
      - "redis-server"
      - "--requirepass"
      - "${REDIS_PASSWORD}"
    image: redis:7-alpine

  rest-api:
    <<: *backend-common
    environment:
      <<: *backend-common-environment
      INSTANCE: rest-api
    ports:
      - "8003:8001"

  scheduler:
    <<: *backend-common
    depends_on:
      - rabbitmq
    environment:
      <<: *backend-common-environment
      INSTANCE: scheduler

  worker:
    <<: *backend-common
    environment:
      <<: *backend-common-environment
      INSTANCE: worker

NGINX:

server {
        server_name poeticmetric.##;
        location / {
        proxy_pass http://localhost:8002/;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
        }
        location /api/ {
        proxy_pass http://localhost:8003/;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
        }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/poeticmetric.##/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/poeticmetric.##/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = poeticmetric.##) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    server_name poeticmetric.##;
    listen 80;
    return 404; # managed by Certbot
}

Note ## is just redacted information :)

[th0th]

What do you see in the logs of the clickhouse container?

[CactusChaos]

docker container logs poeticmetric-clickhouse-1
gives me

/entrypoint.sh: create database 'poeticmetric'
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Logging trace to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
Processing configuration file '/etc/clickhouse-server/users.xml'.
Merging configuration file '/etc/clickhouse-server/users.d/default-user.xml'.
Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.

Doesn't appear to be anything in /var/log relating to clubhouse. Checked all the other logs and the only thing funky seems:

panic: dial tcp 172.24.0.3:9000: connect: connection refused
panic: dial tcp 172.24.0.2:5672: connect: connection refused`

[th0th]

Oh I think I get it. You changed ClickHouse's port from 9000 to 9003 in the port mapping, but you didn't change the environment variable CLICKHOUSE_TCP_PORT. Update it as 9003 as well, and it should work.

[CactusChaos]

No luck on the restart. Anymore ideas on how I can track down the problem?

[CactusChaos]

Here are all the logs from docker compose:

 /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
frontend_1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
frontend_1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/40-generate-config.sh
frontend_1  | /docker-entrypoint.sh: Configuration complete; ready for start up
frontend_1  | 2023/08/30 20:29:16 [notice] 1#1: using the "epoll" event method
frontend_1  | 2023/08/30 20:29:16 [notice] 1#1: nginx/1.23.4
frontend_1  | 2023/08/30 20:29:16 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r4)
frontend_1  | 2023/08/30 20:29:16 [notice] 1#1: OS: Linux 5.15.0-82-generic
frontend_1  | 2023/08/30 20:29:16 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
frontend_1  | 2023/08/30 20:29:16 [notice] 1#1: start worker processes
frontend_1  | 2023/08/30 20:29:16 [notice] 1#1: start worker process 31
frontend_1  | 2023/08/30 20:29:16 [notice] 1#1: start worker process 32

 /entrypoint.sh: create database 'poeticmetric'
clickhouse_1  | Processing configuration file '/etc/clickhouse-server/config.xml'.
clickhouse_1  | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
clickhouse_1  | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
clickhouse_1  | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
clickhouse_1  | Processing configuration file '/etc/clickhouse-server/config.xml'.
clickhouse_1  | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
clickhouse_1  | Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
clickhouse_1  | Processing configuration file '/etc/clickhouse-server/users.xml'.
clickhouse_1  | Merging configuration file '/etc/clickhouse-server/users.d/default-user.xml'.
clickhouse_1  | Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.

 | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1  |
postgres_1  |
postgres_1  | 2023-08-30 20:29:15.732 UTC [1] LOG:  starting PostgreSQL 15.4 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit
postgres_1  | 2023-08-30 20:29:15.733 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1  | 2023-08-30 20:29:15.733 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1  | 2023-08-30 20:29:15.742 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1  | 2023-08-30 20:29:15.750 UTC [24] LOG:  database system was shut down at 2023-08-30 20:26:11 UTC
postgres_1  | 2023-08-30 20:29:15.776 UTC [1] LOG:  database system is ready to accept connections
postgres_1  | 2023-08-30 20:34:15.833 UTC [22] LOG:  checkpoint starting: time
postgres_1  | 2023-08-30 20:34:15.842 UTC [22] LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.003 s, sync=0.002 s,
total=0.010 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB

rabbitmq_1  |   Starting broker...2023-08-30 20:29:20.489075+00:00 [info] <0.230.0> ^[[0m
rabbitmq_1  | 2023-08-30 20:29:20.489075+00:00 [info] <0.230.0>  node           : rabbit@14613349b98b^[[0m
rabbitmq_1  | 2023-08-30 20:29:20.489075+00:00 [info] <0.230.0>  home dir       : /var/lib/rabbitmq^[[0m
rabbitmq_1  | 2023-08-30 20:29:20.489075+00:00 [info] <0.230.0>  config file(s) : /etc/rabbitmq/conf.d/10-defaults.conf^[[0m
rabbitmq_1  | 2023-08-30 20:29:20.489075+00:00 [info] <0.230.0>  cookie hash    : 9CKrB1MDCT8dr/DQm1sAcA==^[[0m
rabbitmq_1  | 2023-08-30 20:29:20.489075+00:00 [info] <0.230.0>  log(s)         : <stdout>^[[0m
rabbitmq_1  | 2023-08-30 20:29:20.489075+00:00 [info] <0.230.0>  data dir       : /var/lib/rabbitmq/mnesia/rabbit@14613349b98b^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.939335+00:00 [info] <0.230.0> Running boot step pre_boot defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.939402+00:00 [info] <0.230.0> Running boot step rabbit_global_counters defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.939676+00:00 [info] <0.230.0> Running boot step rabbit_osiris_metrics defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.939778+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.940279+00:00 [info] <0.230.0> Running boot step rabbit_alarm defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.944546+00:00 [info] <0.380.0> Memory high watermark set to 1525 MiB (1599714099 bytes) of 3814 MiB (3999285248 bytes) total^>
rabbitmq_1  | 2023-08-30 20:29:23.947703+00:00 [info] <0.382.0> Enabling free disk space monitoring (disk free space: 25431265280, total memory: 3999285248)^[>
rabbitmq_1  | 2023-08-30 20:29:23.947856+00:00 [info] <0.382.0> Disk free limit set to 50MB^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.949764+00:00 [info] <0.230.0> Running boot step code_server_cache defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.949873+00:00 [info] <0.230.0> Running boot step file_handle_cache defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.950297+00:00 [info] <0.385.0> Limiting to approx 1048479 file handles (943629 sockets)^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.950421+00:00 [info] <0.386.0> FHC read buffering: OFF^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.950557+00:00 [info] <0.386.0> FHC write buffering: ON^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.951457+00:00 [info] <0.230.0> Running boot step worker_pool defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.951528+00:00 [info] <0.363.0> Will use 2 processes for default worker pool^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.951563+00:00 [info] <0.363.0> Starting worker pool 'worker_pool' with 2 processes in it^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.952134+00:00 [info] <0.230.0> Running boot step database defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.953720+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.955639+00:00 [info] <0.230.0> Successfully synced tables from a peer^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.959972+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.960244+00:00 [info] <0.230.0> Successfully synced tables from a peer^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.960282+00:00 [info] <0.230.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, ski>
rabbitmq_1  | 2023-08-30 20:29:23.960567+00:00 [info] <0.230.0> Running boot step tracking_metadata_store defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.960648+00:00 [info] <0.398.0> Setting up a table for connection tracking on this node: tracked_connection^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.960690+00:00 [info] <0.398.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhos>
rabbitmq_1  | 2023-08-30 20:29:23.960735+00:00 [info] <0.398.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user^>
rabbitmq_1  | 2023-08-30 20:29:23.960794+00:00 [info] <0.398.0> Setting up a table for channel tracking on this node: tracked_channel^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.960833+00:00 [info] <0.398.0> Setting up a table for channel tracking on this node: tracked_channel_per_user^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.960934+00:00 [info] <0.230.0> Running boot step networking_metadata_store defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961012+00:00 [info] <0.230.0> Running boot step feature_flags defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961317+00:00 [info] <0.230.0> Running boot step codec_correctness_check defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961376+00:00 [info] <0.230.0> Running boot step external_infrastructure defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961400+00:00 [info] <0.230.0> Running boot step rabbit_event defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961594+00:00 [info] <0.230.0> Running boot step rabbit_registry defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961672+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961727+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961765+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961797+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_direct defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961854+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961894+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_headers defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961943+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_topic defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.961991+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.962029+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.962066+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.962109+00:00 [info] <0.230.0> Running boot step rabbit_priority_queue defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.962130+00:00 [info] <0.230.0> Priority queues enabled, real BQ is rabbit_variable_queue^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.962181+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_client_local defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.962234+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.964248+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_random defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.964376+00:00 [info] <0.230.0> Running boot step kernel_ready defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.964438+00:00 [info] <0.230.0> Running boot step rabbit_sysmon_minder defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.964575+00:00 [info] <0.230.0> Running boot step rabbit_epmd_monitor defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.965326+00:00 [info] <0.406.0> epmd monitor knows us, inter-node communication (distribution) port: 25672^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.965455+00:00 [info] <0.230.0> Running boot step guid_generator defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.968199+00:00 [info] <0.230.0> Running boot step rabbit_node_monitor defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.968505+00:00 [info] <0.410.0> Starting rabbit_node_monitor^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.968715+00:00 [info] <0.230.0> Running boot step delegate_sup defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.969235+00:00 [info] <0.230.0> Running boot step rabbit_memory_monitor defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.969478+00:00 [info] <0.230.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.969590+00:00 [info] <0.230.0> Running boot step core_initialized defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.969631+00:00 [info] <0.230.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.969686+00:00 [info] <0.230.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.969726+00:00 [info] <0.230.0> Running boot step rabbit_definitions_hashing defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.969791+00:00 [info] <0.230.0> Running boot step rabbit_exchange_parameters defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.970177+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.970541+00:00 [info] <0.230.0> Running boot step rabbit_policies defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.970888+00:00 [info] <0.230.0> Running boot step rabbit_policy defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.970979+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_validator defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.971174+00:00 [info] <0.230.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.971276+00:00 [info] <0.230.0> Running boot step rabbit_stream_coordinator defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.971411+00:00 [info] <0.230.0> Running boot step rabbit_vhost_limit defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.971561+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.971607+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.971645+00:00 [info] <0.230.0> Management plugin: using rates mode 'basic'^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.972062+00:00 [info] <0.230.0> Running boot step recovery defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.976943+00:00 [info] <0.446.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@14613349b98b/msg_stores/vhosts/CMP>
rabbitmq_1  | 2023-08-30 20:29:23.981958+00:00 [info] <0.446.0> Starting message stores for vhost 'poeticmetric'^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.982474+00:00 [info] <0.456.0> Message store "CMP16TOTDH1X1H5NB7OY6KAIQ/msg_store_transient": using rabbit_msg_store_ets_inde>
rabbitmq_1  | 2023-08-30 20:29:23.984278+00:00 [info] <0.446.0> Started message store of type transient for vhost 'poeticmetric'^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.984727+00:00 [info] <0.460.0> Message store "CMP16TOTDH1X1H5NB7OY6KAIQ/msg_store_persistent": using rabbit_msg_store_ets_ind>
rabbitmq_1  | 2023-08-30 20:29:23.988107+00:00 [info] <0.446.0> Started message store of type persistent for vhost 'poeticmetric'^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.998485+00:00 [info] <0.446.0> Recovering 12 queues of type rabbit_classic_queue took 20ms^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.998558+00:00 [info] <0.446.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms^[[0m
rabbitmq_1  | 2023-08-30 20:29:23.998773+00:00 [info] <0.446.0> Recovering 0 queues of type rabbit_stream_queue took 0ms^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.004819+00:00 [info] <0.230.0> Running boot step empty_db_check defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.004901+00:00 [info] <0.230.0> Will not seed default virtual host and user: have definitions to load...^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.005028+00:00 [info] <0.230.0> Running boot step rabbit_observer_cli defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.005112+00:00 [info] <0.230.0> Running boot step rabbit_looking_glass defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.005176+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics_gc defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.005536+00:00 [info] <0.230.0> Running boot step background_gc defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.005769+00:00 [info] <0.230.0> Running boot step routing_ready defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.005809+00:00 [info] <0.230.0> Running boot step pre_flight defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.005836+00:00 [info] <0.230.0> Running boot step notify_cluster defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.005912+00:00 [info] <0.230.0> Running boot step networking defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.006080+00:00 [info] <0.230.0> Running boot step definition_import_worker_pool defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.006313+00:00 [info] <0.363.0> Starting worker pool 'definition_import_pool' with 2 processes in it^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.006704+00:00 [info] <0.230.0> Running boot step cluster_name defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.006790+00:00 [info] <0.230.0> Running boot step direct_client defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.006915+00:00 [info] <0.230.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.007079+00:00 [info] <0.230.0> Creating table rabbit_node_maintenance_states for maintenance mode status^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.007620+00:00 [info] <0.230.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.007781+00:00 [info] <0.556.0> Resetting node maintenance status^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.024090+00:00 [info] <0.615.0> Management plugin: HTTP (non-TLS) listener started on port 15672^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.024495+00:00 [info] <0.643.0> Statistics database started.^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.024659+00:00 [info] <0.642.0> Starting worker pool 'management_worker_pool' with 3 processes in it^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.035994+00:00 [info] <0.657.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.036451+00:00 [info] <0.556.0> Ready to start client connection listeners^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.038276+00:00 [info] <0.701.0> started TCP listener on [::]:5672^[[0m
rabbitmq_1  |  completed with 4 plugins.
rabbitmq_1  | 2023-08-30 20:29:24.107743+00:00 [info] <0.556.0> Server startup complete; 4 plugins started.^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.107743+00:00 [info] <0.556.0>  * rabbitmq_prometheus^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.107743+00:00 [info] <0.556.0>  * rabbitmq_management^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.107743+00:00 [info] <0.556.0>  * rabbitmq_management_agent^[[0m
rabbitmq_1  | 2023-08-30 20:29:24.107743+00:00 [info] <0.556.0>  * rabbitmq_web_dispatch^[[0m
rabbitmq_1  | 2023-08-30 20:29:26.476446+00:00 [info] <0.705.0> accepting AMQP connection <0.705.0> (192.168.0.7:44350 -> 192.168.0.4:5672)^[[0m
rabbitmq_1  | 2023-08-30 20:29:26.484333+00:00 [info] <0.705.0> connection <0.705.0> (192.168.0.7:44350 -> 192.168.0.4:5672): user 'poeticmetric' authenticate>

redis_1  | 1:C 30 Aug 2023 20:29:15.387 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1  | 1:C 30 Aug 2023 20:29:15.390 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1  | 1:C 30 Aug 2023 20:29:15.390 * Redis version=7.2.0, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1  | 1:C 30 Aug 2023 20:29:15.390 * Configuration loaded
redis_1  | 1:M 30 Aug 2023 20:29:15.391 * monotonic clock: POSIX clock_gettime
redis_1  | 1:M 30 Aug 2023 20:29:15.401 * Running mode=standalone, port=6379.
redis_1  | 1:M 30 Aug 2023 20:29:15.402 * Server initialized
redis_1  | 1:M 30 Aug 2023 20:29:15.403 * Loading RDB produced by version 7.2.0
redis_1  | 1:M 30 Aug 2023 20:29:15.403 * RDB age 184 seconds
redis_1  | 1:M 30 Aug 2023 20:29:15.403 * RDB memory usage when created 1.02 Mb
redis_1  | 1:M 30 Aug 2023 20:29:15.403 * Done loading RDB, keys loaded: 0, keys expired: 0.
redis_1  | 1:M 30 Aug 2023 20:29:15.403 * DB loaded from disk: 0.002 seconds
redis_1  | 1:M 30 Aug 2023 20:29:15.403 * Ready to accept connections tcp

| goroutine 1 [running]:
rest-api_1  | log.Panicln({0x40003dfea0?, 0x400028a7d0?, 0xd63580?})
rest-api_1  |   /usr/local/go/src/log/log.go:446 +0x60
rest-api_1  | main.main()
rest-api_1  |   /usr/src/poeticmetric/cmd/migrator/main.go:57 +0x26c
rest-api_1  | 2023/08/30 20:29:26 📝 Running PostgreSQL migrations...
rest-api_1  | 2023/08/30 20:29:26 📝 Running PostgreSQL migrations... ✅
rest-api_1  | 2023/08/30 20:29:26 📊 Running ClickHouse migrations...
rest-api_1  | 2023/08/30 20:29:26 📊 Running ClickHouse migrations... ✅
rest-api_1  |
rest-api_1  | ^[[90m^[[90m ┌───────────────────────────────────────────────────┐
rest-api_1  | ^[[90m │                   ^[[96mFiber v2.48.0^[[90m                   │
rest-api_1  | ^[[90m │                http://127.0.0.1:80                │
rest-api_1  | ^[[90m │        (bound on host 0.0.0.0 and port 80)        │
rest-api_1  | ^[[90m │                                                   │
rest-api_1  | ^[[90m │ Handlers ........... ^[[96m158^[[90m  Processes ........... ^[[96m1^[[90m │
rest-api_1  | ^[[90m │ Prefork ....... Disabled  PID ................. ^[[96m1^[[90m │
rest-api_1  | ^[[90m └───────────────────────────────────────────────────┘^[[0m
rest-api_1  |

rker_1  | goroutine 1 [running]:
worker_1  | log.Panicln({0x40003dfea0?, 0x4000289360?, 0xd63580?})
worker_1  |     /usr/local/go/src/log/log.go:446 +0x60
worker_1  | main.main()
worker_1  |     /usr/src/poeticmetric/cmd/migrator/main.go:57 +0x26c
worker_1  | 2023/08/30 20:28:25 dial tcp: lookup redis on 127.0.0.11:53: server misbehaving
worker_1  | panic: dial tcp: lookup redis on 127.0.0.11:53: server misbehaving
worker_1  |
worker_1  |
worker_1  | goroutine 1 [running]:
worker_1  | log.Panicln({0x40003f7ea0?, 0x400007f450?, 0xd63580?})
worker_1  |     /usr/local/go/src/log/log.go:446 +0x60
worker_1  | main.main()
worker_1  |     /usr/src/poeticmetric/cmd/migrator/main.go:57 +0x26c
worker_1  | 2023/08/30 20:29:26 📝 Running PostgreSQL migrations...
worker_1  | 2023/08/30 20:29:26 📝 Running PostgreSQL migrations... ✅
worker_1  | 2023/08/30 20:29:26 📊 Running ClickHouse migrations...
worker_1  | 2023/08/30 20:29:26 📊 Running ClickHouse migrations... ✅
worker_1  | 2023/08/30 20:29:26 Queues:
worker_1  | 2023/08/30 20:29:26 * createEvent
worker_1  | 2023/08/30 20:29:26 * endTrials
worker_1  | 2023/08/30 20:29:26 * notifyAboutEndingTrials
worker_1  | 2023/08/30 20:29:26 * notifyAboutTrialMidways
worker_1  | 2023/08/30 20:29:26 * sendEmail
worker_1  | 2023/08/30 20:29:26 * sendWebhook
worker_1  | 2023/08/30 20:29:26 Starting to listen...

scheduler_1  | goroutine 1 [running]:
scheduler_1  | log.Panicln({0x40003dfea0?, 0x4000288b90?, 0xd63580?})
scheduler_1  |  /usr/local/go/src/log/log.go:446 +0x60
scheduler_1  | main.main()
scheduler_1  |  /usr/src/poeticmetric/cmd/migrator/main.go:57 +0x26c
scheduler_1  | 2023/08/30 20:28:25 dial tcp: lookup redis on 127.0.0.11:53: server misbehaving
scheduler_1  | panic: dial tcp: lookup redis on 127.0.0.11:53: server misbehaving
scheduler_1  |
scheduler_1  |
scheduler_1  | goroutine 1 [running]:
scheduler_1  | log.Panicln({0x40003f7ea0?, 0x400009c7d0?, 0xd63580?})
scheduler_1  |  /usr/local/go/src/log/log.go:446 +0x60
scheduler_1  | main.main()
scheduler_1  |  /usr/src/poeticmetric/cmd/migrator/main.go:57 +0x26c
scheduler_1  | 2023/08/30 20:29:26 📝 Running PostgreSQL migrations...
scheduler_1  | 2023/08/30 20:29:26 📝 Running PostgreSQL migrations... ✅
scheduler_1  | 2023/08/30 20:29:26 📊 Running ClickHouse migrations...
scheduler_1  | 2023/08/30 20:29:26 📊 Running ClickHouse migrations... ✅
scheduler_1  | 2023/08/30 20:29:26 ⌛ Starting scheduler...

[CactusChaos]

@th0th
Ok I think I might have worked it out but not 100% sure how to solve it. It appears my volumes/containers are being named differently:

poeticmetric-clickhouse-1   clickhouse/clickhouse-server:latest-alpine   "/entrypoint.sh"         clickhouse          About a minute ago   Up About a minute   8123/tcp, 9009/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp
poeticmetric-frontend-1     poeticmetric/frontend:latest                 "/docker-entrypoint.…"   frontend            About a minute ago   Up About a minute   80/tcp, 0.0.0.0:8002->8000/tcp, :::8002->8000/tcp
poeticmetric-postgres-1     postgres:15-alpine                           "docker-entrypoint.s…"   postgres            About a minute ago   Up About a minute   0.0.0.0:5434->5432/tcp, :::5434->5432/tcp
poeticmetric-rabbitmq-1     rabbitmq:3-management-alpine                 "docker-entrypoint.s…"   rabbitmq            About a minute ago   Up About a minute   4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 15691-15692/tcp, 25672/tcp
poeticmetric-redis-1        redis:7-alpine                               "docker-entrypoint.s…"   redis               About a minute ago   Up About a minute   6379/tcp
poeticmetric-rest-api-1     poeticmetric/backend:latest                  "docker-entrypoint.sh"   rest-api            About a minute ago   Up About a minute   0.0.0.0:8003->8001/tcp, :::8003->8001/tcp
poeticmetric-scheduler-1    poeticmetric/backend:latest                  "docker-entrypoint.sh"   scheduler           About a minute ago   Up About a minute
poeticmetric-worker-1       poeticmetric/backend:latest                  "docker-entrypoint.sh"   worker              About a minute ago   Up About a minute

Where in .env they are named directly eg:
CLICKHOUSE_HOST=clickhouse
I have tried updating the ones that within the .env but not sure about how to update the ones that are not in .env. Does this sound right?

[th0th]

I am happy to hear that you figured it out 💐 What change you did made it work?

[CactusChaos]

Didn't end up having any luck unfortunately so switched over to Umami. Cheers for your help!