Docker versions of ClickHouse server since 24.1.4 ignore initialization

[clamoris]

Describe the issue
Latest version of Docker image fails to run initialisation scripts, shows "ClickHouse Database directory appears to contain a database; Skipping initialization" in logs. Either a bug or documentation not updated to reflect changes.

How to reproduce
Issue appears in clickhouse/clickhouse-server:24.1.4, still present in 24.1.5. Volume mounted to /docker-entrypoint-initdb.d/ does not initialise the database, but should, as stated here - https://hub.docker.com/r/clickhouse/clickhouse-server

[darwin67]

Same here. Our CI started failing from a day or so again all of a sudden, and can also confirm the problem started from 24.1.4 where setting the environment variable CLICKHOUSE_DB doesn't actually initializes the expected database.

[ronnix]

Might be related to #59991

[den-crane]

• Option to run init db scripts every time #59808
• Clickhouse server no longer initializes default database #59989 Run init scripts when option is enabled rather than disabled #59991

[abraithwaite]

Having the same issue. Here's initialization log and what's contained in the directory. Maybe the check on the data directory not being empty is too simple?

Logs from docker-compose up -d --build clickhouse:

15:24:03 $ dc logs clickhouse
clickhouse-1  | /entrypoint.sh: create new user 'default' instead 'default'
clickhouse-1  | ClickHouse Database directory appears to contain a database; Skipping initialization
clickhouse-1  | Processing configuration file '/etc/clickhouse-server/config.xml'.
clickhouse-1  | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
clickhouse-1  | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
clickhouse-1  | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log

15:24:41 $ dc exec clickhouse /bin/bash
root@6381428b4a5d:/# cd /var/lib/clickhouse/
root@6381428b4a5d:/var/lib/clickhouse# ls
access  dictionaries_lib  format_schemas  metadata_dropped   preprocessed_configs  store  user_defined  user_scripts
data    flags             metadata        named_collections  status                tmp    user_files    uuid
root@6381428b4a5d:/var/lib/clickhouse# cd data/
root@6381428b4a5d:/var/lib/clickhouse/data# ls
default  system
root@6381428b4a5d:/var/lib/clickhouse/data# cd default/
root@6381428b4a5d:/var/lib/clickhouse/data/default# ls
root@6381428b4a5d:/var/lib/clickhouse/data/default# ls -lah
total 0
drwxr-x---. 2 clickhouse clickhouse  6 Feb 20 23:24 .
drwxr-x---. 4 clickhouse clickhouse 35 Feb 20 23:24 ..
root@6381428b4a5d:/var/lib/clickhouse/data/default# cd ../system/
root@6381428b4a5d:/var/lib/clickhouse/data/system# ls -lah
total 0
drwxr-x---. 2 clickhouse clickhouse 72 Feb 20 23:24 .
drwxr-x---. 4 clickhouse clickhouse 35 Feb 20 23:24 ..
lrwxrwxrwx. 1 clickhouse clickhouse 67 Feb 20 23:24 asynchronous_metric_log -> /var/lib/clickhouse/store/b35/b3576987-db1c-4934-b8a6-64af5fc84c41/
lrwxrwxrwx. 1 clickhouse clickhouse 67 Feb 20 23:24 metric_log -> /var/lib/clickhouse/store/b57/b5761b58-6f89-4c4e-b9d8-9dba586c1050/
lrwxrwxrwx. 1 clickhouse clickhouse 67 Feb 20 23:24 trace_log -> /var/lib/clickhouse/store/b35/b3554d75-2eea-44b6-9b6e-48a97cc63a6c/

Docker compose config:

  clickhouse:
    image: docker.io/clickhouse/clickhouse-server:24
    environment:
      - CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
    volumes:
      - ./api/ch/migrations:/docker-entrypoint-initdb.d/
    ports:
      - "9000:9000"
      - "8123:8123"

Noteworthy: we do not want to always run init scripts, so the new option doesn't help us too much.

Edit: setting CLICKHOUSE_ALWAYS_RUN_INITDB_SCRIPTS is ineffective also anyways. The initscripts are still not run when I have that set.

[Algunenano]

Fixed in #59991

[abraithwaite]

I don't believe that to be the case. As I mentioned in my comment, the data directory already exists and always will exist with the docker setup as it is currently, meaning that the init scripts will still never get run

https://github.com/ClickHouse/ClickHouse/pull/59991/files#diff-65b30e60f953b0f91424fca7c7b9f257fe4cc1e6f69c0cf65bf931015912370cR123-R131

How was this tested out of curiosity? Would love to replicate on my end. @Algunenano

[ZenVoich]

I confirm, setting CLICKHOUSE_ALWAYS_RUN_INITDB_SCRIPTS in the latest version 24.1.6.52 doesn't run init scirpts

[Zach-Johnson]

This is now broken on the latest 23.x versions as well 😭

[martinformi]

Ok, how to fix this without updating to 24.x ?