install.sh fails due to slow clickhouse startup process

[pharindoko]

Version

2.1.70

Steps to Reproduce

1. go to onpremise folder
2. execute bash -x install.sh --no-user-prompt
3. wait until clickhouse get`s started in correlation with kafka and snuba

2021-07-18 11:45:33,696 Attempting to connect to Kafka (attempt 0)...
2021-07-18 11:45:35,701 Attempting to connect to Kafka (attempt 1)...
2021-07-18 11:45:37,704 Attempting to connect to Kafka (attempt 2)...
2021-07-18 11:45:37,718 Connected to Kafka on attempt 2
2021-07-18 11:45:37,718 Creating Kafka topics...
++ docker-compose --no-ansi run --rm snuba-api migrations migrate --force
--no-ansi option is deprecated and will be removed in future versions. Use `--ansi never` instead.
Creating sentry_onpremise_snuba-api_run ...
Creating sentry_onpremise_snuba-api_run ... done
Failed to connect to clickhouse:9000
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 260, in connect
    return self._init_connection(host, port)
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 226, in _init_connection
    self.socket = self._create_socket(host, port)
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 221, in _create_socket
    raise err
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 212, in _create_socket
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Failed to connect to clickhouse:9000
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 260, in connect
    return self._init_connection(host, port)
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 226, in _init_connection
    self.socket = self._create_socket(host, port)
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 221, in _create_socket
    raise err
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 212, in _create_socket
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Connection to Clickhouse cluster clickhouse:9000 failed (attempt 0)

4. the connection attempt fail and the reason for this is the slow startup of clickhouse entrypoint.sh
   it does recursively chown through the clickhouse folders
   https://github.com/ClickHouse/ClickHouse/blob/9f529b0c25f1e092651355775355b5cee57c259f/docker/server/entrypoint.sh#L75

5. using
   tree -a <clickhouse-directory>
   I get 1078 directories, 76755 files, ca. 4.6 GB

6. Manually starting entrypoint.sh in the docker container takes me minimum 5 Minutes waiting for chown to finish.

7. Quite interesting that clickhouse knows about that problem ... that`s why they put following code at the top of the entrypoint.sh file
   https://github.com/ClickHouse/ClickHouse/blob/9f529b0c25f1e092651355775355b5cee57c259f/docker/server/entrypoint.sh#L7

8. What helped to fix this error immediately was to add environment variable CLICKHOUSE_DO_NOT_CHOWN with value 1 into docker-compose file.
   https://github.com/getsentry/onpremise/blob/34812ce837b29f6ff204638b7fd24caeaddb411c/docker-compose.yml#L172
   Means no permissions will be set in this long running process.
   Correct permissions could be handled in another task triggered by install.sh ...

Expected Result

clickhouse starts up immediately as expected in container environments without doing that much background work and connections will not fail due to slow upcoming clickhouse-server.

Actual Result

Traceback (most recent call last):
  File "/usr/src/snuba/snuba/migrations/connect.py", line 30, in check_clickhouse_connections
    check_clickhouse(clickhouse)
  File "/usr/src/snuba/snuba/migrations/connect.py", line 49, in check_clickhouse
    ver = clickhouse.execute("SELECT version()")[0][0]
  File "/usr/src/snuba/snuba/clickhouse/native.py", line 96, in execute
    raise ClickhouseError(e.code, e.message) from e
snuba.clickhouse.errors.ClickhouseError: [210] Connection refused (clickhouse:9000)
Failed to connect to clickhouse:9000

[chadwhitacre]

ca. 4.6 GB

This sounds to me like you're beyond the usage limits of what onpremise is designed to handle. If that's the case our recommended next step is to migrate to SaaS. Is that an option for you?

If not, as at #1034 (comment) I would recommend addressing the issue through automation on your side since you are already using a custom setup afaict. I don't see that it makes sense to add the CLICKHOUSE_DO_NOT_CHOWN hack to onpremise as our goal here is to support simple, low-volume and proof-of-concept deploys.

[pharindoko]

Hey @chadwhitacre , @BYK

I`m not happy with this answer and I would like you to discuss that internally.

What I see here is a race condition that will occur independent of the size of clickhouse (even with less size).

And what you call a hack (CLICKHOUSE_DO_NOT_CHOWN) is in my eyes a good fix for this problem that clickhouse itself offers in their entrypoint.sh (dockerfile)

I had no a custom setup until I came to this point. I just changed the config.yml and the sentry.conf.py (db connection) with standard properties you described - that`s it - no fancy hacks.

I can create a PR with a proposal if you want to.

br,

pharindoko

[chadwhitacre]

I had no a custom setup until I came to this point. I just changed the config.yml and the sentry.conf.py (db connection) with standard properties you described - that`s it - no fancy hacks.

Fair enough. Sorry for the faulty assumption.

What I see here is a race condition that will occur independent of the size of clickhouse (even with less size).

Really? Doesn't the race condition get worse as the number of files that Clickhouse wants to chown increases? If this were to happen with regularity at smaller sizes I would expect to have heard about it by now.

Correct permissions could be handled in another task triggered by install.sh ...

If correcting permissions after the fact is in fact necessary, then won't the ClickHouse entrypoint fail, causing install.sh to fail as well? That's how I'm reading the elif:

    if [ "$DO_CHOWN" = "1" ]; then
        # ensure proper directories permissions
        chown -R "$USER:$GROUP" "$dir"
    elif ! $gosu test -d "$dir" -a -w "$dir" -a -r "$dir"; then
        echo "Necessary directory '$dir' isn't accessible by user with id '$USER'"
        exit 1
    fi

Though I suppose that's only checking the top-level directory, it's not recursing. Do we know that we need to correct permissions after the fact?

If all you need to do is set the CLICKHOUSE_DO_NOT_CHOWN environment variable, can you do that easily in your setup with a docker-compose.override.yml?

[chadwhitacre]

P.S. BYK is on [a well-earned] vacation this week. Reopening so we can discuss when he returns.

[BYK]

I'd say the best thing we can do here is to add a wait mechanism to the migrate command in Snuba. That said I'm also not sure if we really need the chown command in our setup as we just use docker volumes, and they should be fine? We can certainly try and see how using CLICKHOUSE_DO_NOT_CHOWN works without anything extra in the install script. Wdyt @chadwhitacre?

[chadwhitacre]

Wdyt @chadwhitacre?

If all that needs to be done is setting CLICKHOUSE_DO_NOT_CHOWN, that should be doable via docker-compose.override.yml (right?).

If something more needs to be done than setting an environment variable, then I'm not sure we want to take on the added complexity.

[BYK]

If all that needs to be done is setting CLICKHOUSE_DO_NOT_CHOWN, that should be doable via docker-compose.override.yml (right?).

Yes, but my point was making this the default as it should be harmless for our use case.

[chadwhitacre]

making this the default as it should be harmless for our use case.

If this is indeed true then I'm fine with it.

Means no permissions will be set in this long running process.

Has this been a problem for you, @pharindoko?

[pharindoko]

Hmm guys, I guess one issue is that I`m using AWS EFS (network share). This might have an influence why it takes so long - so I will switch to AWS ebs - block storage ssd ...

setting CLICKHOUSE_DO_NOT_CHOWN can be set in docker-compose.yml but would mean that all files have to be owned correctly (that`s an additional check clickhouse does).

as consequence it means that in the install.sh there must be an additional task that will be done in advance to chown through all files. (docker-compose run ... clickhouse ...)

I guess it could be easier to add a healthcheck in docker-compose.yml that waits for clickhouse to be really ready (by curl against localhost:9000) before snuba starts tries to connect ... @BYK , @chadwhitacre
What`s your opinion ?

br,

pharindoko

[chadwhitacre]

I`m using AWS EFS (network share). This might have an influence why it takes so long - so I will switch to AWS ebs - block storage ssd ...

🤞

add a healthcheck in docker-compose.yml that waits for clickhouse to be really ready (by curl against localhost:9000) before snuba starts tries to connect

I'm open to this, since we already have healthchecks for other services. If I'm reading it right we wait for like ~90 seconds. For your system @pharindoko we would need to bump that up to 5+ minutes, I think? You okay with that @BYK?