How to gracefully shutdown service when systemctl stop will issue signal SIGKILL

[dddping]

Company or project name

No response

Question

I follow the installation guide to install ClickHouse at ubuntu 26.04 inside WSL2. After installation completed, I run systemctl stop to terminate the ClickHouse service.
But the status log show ubuntu just kill the process with signal SIGKILL. I wonder is there a better approach to shutdown ClickHouse service and avoid data lost?

I search the doc and only find something like SYSTEM FLUSH ASYNC INSERT QUEUE and async_insert_queue_flush_on_shutdown. Some other source said it need to monitor process by using SHOW PROCESSLIST;.

So my questions is, given that ubuntu 26.04 doesn't seem to handle shutdown gracefully, what is the manual steps require to shutdown ClickHouse service?

I install and try systemctl stop at both ubuntu 26.04 and CentOS Stream 10 and both do signal SIGKILL...

The log of systemctl status

clickhouse-server[1878]: Processing configuration file '/etc/clickhouse-server/config.xml'.
clickhouse-server[1878]: Logging trace to /var/log/clickhouse-server/clickhouse-server.log
clickhouse-server[1878]: Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
systemd[1]: clickhouse-server.service: Supervising process 1886 which is not our child. We'll most likely not notice when it exits.
systemd[1]: Started clickhouse-server.service - ClickHouse Server (analytic DBMS for big data).
systemd[1]: Stopping clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)...
systemd[1]: clickhouse-server.service: Killing process 1886 (n/a) with signal SIGKILL.
systemd[1]: clickhouse-server.service: Deactivated successfully.
systemd[1]: Stopped clickhouse-server.service - ClickHouse Server (analytic DBMS for big data).
systemd[1]: clickhouse-server.service: Consumed 2.719s CPU time over 35.604s wall clock time, 362.7M memory peak.

[skuznetsov-clickhouse]

Good news: your shutdown is almost certainly already graceful — the SIGKILL line in systemctl status is a cosmetic artifact, not the signal that stopped the server. Here is what happens.

Why systemd prints this

The packaged clickhouse-server.service starts the binary, which forks: the parent becomes a supervisor (clickhouse-watchdog, PID 1878 in your log — it exists to record OOM/SIGKILL traces into clickhouse-server.log) and the child (PID 1886) is the actual server. The watchdog tells systemd MAINPID=1886. Since 1886 is a grandchild of systemd, not a direct child, systemd prints "Supervising process 1886 which is not our child. We'll most likely not notice when it exits."

On systemctl stop, systemd sends SIGTERM (the default KillSignal; the shipped unit does not override it, and it sets TimeoutStopSec=infinity, so systemd never force-kills a live server on timeout). ClickHouse handles SIGTERM by shutting down gracefully: it stops listening sockets, waits for backups/restores, flushes the async-insert queue (async_insert_queue_flush_on_shutdown is true by default), and shuts down storages. After the server and watchdog have exited, systemd — which could not observe the exit of the "not our child" MainPID — fires one last cleanup SIGKILL at the recorded PID. The (n/a) where the process name should be means the process was already gone at that moment: systemd prints that line before attempting the kill, and (n/a) is what it shows when it can no longer read the process's name from /proc. The SIGKILL hit nothing. Hence "Deactivated successfully".

How to verify on your machine

grep -E "Received termination signal|Closed all listening sockets|Closed connections|Shutting down storages|Application: shutting down" \
    /var/log/clickhouse-server/clickhouse-server.log | tail

You should see the full graceful sequence. Conversely, if the server had really been killed by SIGKILL, the watchdog would have logged a fatal line: Child process was terminated by signal 9 (KILL). — grep for that; you won't find it for your systemctl stop.

If you want the confusing messages gone

Disable the watchdog fork so systemd supervises the real server process directly (this is even suggested in a comment inside the shipped unit file):

sudo systemctl edit clickhouse-server

[Service]
Environment=CLICKHOUSE_WATCHDOG_ENABLE=0

then sudo systemctl daemon-reload && sudo systemctl restart clickhouse-server. Trade-off: you lose the watchdog's OOM/SIGKILL trace lines in clickhouse-server.log and rely on journalctl instead.

Tuning graceful shutdown (optional)

By default, on shutdown ClickHouse cancels still-running queries and waits up to 5 seconds for connections to close. If you want it to let running queries finish first, create /etc/clickhouse-server/config.d/graceful_shutdown.xml:

<clickhouse>
    <!-- wait for running queries to finish instead of cancelling them (default: 0) -->
    <shutdown_wait_unfinished_queries>1</shutdown_wait_unfinished_queries>
    <!-- how long to wait, in seconds (default: 5) -->
    <shutdown_wait_unfinished>60</shutdown_wait_unfinished>
</clickhouse>

No systemd change is needed for this: TimeoutStopSec=infinity in the shipped unit already lets the server take as long as it needs.

Manual alternatives — none are required. clickhouse-client -q "SYSTEM SHUTDOWN" triggers the same graceful shutdown (it sends SIGTERM to the server's process group), but note that under systemd the packaged unit has Restart=always, so the server will be restarted about 30 seconds later — systemctl stop is the right way to stop the service. And data written by committed inserts is already on disk in immutable parts regardless — a graceful shutdown is about in-flight queries and in-memory queues (async inserts, Buffer tables), which the sequence above handles.