How to Run ClickHouse Export Queries Asynchronously Without Waiting for Results

[LarsAlmgren]

Hey, I wonder if someone know a good way of achieving the following.

We are copying data from ClickHouse to GCS with GCS function. Sometimes these queries run for a while. We trigger the queries from processes running on preemtible nodes, so basically the process can be killed whenever. So if we sit and wait for the query to finish and the process is shutdown, the query is cancelled and we would need to retry the whole query. Hence, I would like to do these "exports" asynchronous instead of waiting as I anyway don't expect anything more than 👍 or 👎 for the query.

The workflow would be something like this:

1. Trigger query with custom query id and don't wait
2. Poll query_log until query have finished
3. Use the data in GCS for whatever I want to do

Good to know info:

• We are using http to connect to ClickHouse

I haven't really understood how (or if) I can trigger a query and then just not wait for the result.

[LarsAlmgren]

Gemini suggested the following test, but it was blocking

curl -i "http://localhost:8123/?query_id=api_test_1&wait_end_of_query=0&cancel_http_readonly_queries_on_client_close=0" \
  --data-binary "SELECT sleep(3)"

[LarsAlmgren]

Found this

Ran the following and cancelled the request, which worked:

curl -i "http://localhost:8123/?query_id=api_test_1" --data-binary "SELECT sleep(10) Format Null"

Verified by running the following query:

SELECT event_time, query_id, type FROM system.query_log WHERE query_id LIKE 'fire_forget_test_%' ORDER BY event_time DESC LIMIT 10

SELECT
    event_time,
    query_id,
    type
FROM system.query_log
WHERE query_id LIKE 'fire_forget_test_%'
ORDER BY event_time DESC
LIMIT 10

Query id: 282cb3a9-feda-48aa-ae27-546ec172dc54

    ┌──────────event_time─┬─query_id─────────────────────────┬─type─────────────────────┐
 1. │ 2025-12-19 15:10:23 │ fire_forget_test_20251219_161023 │ QueryStart               │
 2. │ 2025-12-19 15:05:57 │ fire_forget_test_20251219_160532 │ QueryFinish              │

Made an implementation in Golang which works as well:

func main() {
	client := &http.Client{}

	params := url.Values{}

	id := runID()
	queryID := fmt.Sprintf("fire_forget_test_%s", id)
	params.Add("query_id", queryID)

	query := "SELECT sleep(10) Format Null"
	endpoint := fmt.Sprintf("http://localhost:8123/?%s&query=%s", params.Encode(), url.QueryEscape(query))

	t.Logf("Running query: %s", query)

	ctx, cancelFunc := context.WithCancel(context.Background())
	req, _ := http.NewRequestWithContext(ctx, "GET", endpoint, nil)

	go func() {
		resp, err := client.Do(req)
		if err != nil {
			fmt.Printf("Error: %s\n", err)
			os.Exit(1)
		}
		resp.Body.Close()
	}()

	for {
		time.Sleep(1 * time.Second)
		q := "SELECT COUNT(1) FROM system.processes WHERE query_id = '" + queryID + "'"
		e := fmt.Sprintf("http://localhost:8123/?query=%s", url.QueryEscape(q))
		r, _ := http.NewRequest("GET", e, nil)
		resp, _ := client.Do(r)
		if resp.StatusCode == 200 {
			fmt.Println("Query is running.")
			cancelFunc()
			break
		}
		resp.Body.Close()
	}

	fmt.Println("Success! The Go process is now exiting, but the query should still be running.")
}