Any way to get a correct X-ClickHouse-Summary without send_progress_in_http_headers?

[iameugenejo]

I have an application sending a big query to clickhouse using http interface, and I need to get the summary of the query at the end. (I cannot use JSON format for it)

The X-ClickHouse-Summary header from the request has all the fields set to 0, which is incorrect -

 X-ClickHouse-Summary: {"read_rows":"0","read_bytes":"0","written_rows":"0","written_bytes":"0","total_rows_to_read":"0"}

Only way I get the correct summary is to pass send_progress_in_http_headers=1

The problem is, http client modules I have tried throw an error for having too many headers in the response.

python requests -
requests.exceptions.ConnectionError: ('Connection aborted.', HTTPException('got more than 100 headers',))
nodejs requests -
{ [Error: Parse Error] bytesParsed: 153, code: 'HPE_HEADER_OVERFLOW'}

Is there any way to get the correct summary output without the progress headers?

[iameugenejo]

I just found a workaround, adding a huge http_headers_progress_interval_ms value to suppress the progress header for a long time seems to work for my case. http_headers_progress_interval_ms=3600000.

I would still like to know if there's a better or a correct way to do this.

[alexey-milovidov]

The way you have found is Ok: you want to get only one X-ClickHouse-Summary header at the end of query execution (if no data was sent at this moment), so you set progress interval to very large value. Looks appropriate.

I don't know any better way to do it.

[alexey-milovidov]

If data is sent too early, you can also provide wait_end_of_query=1 parameter in URL that will buffer all the data.