grpc query hangs in ClickHouse 21.3 and 21.6

[zhicwu]

Sending simple queries like insert into test_insert(i) values(<random number>) or select toUInt32(<random number>) as n via grpc(non-ssl) using multiple threads, often some of them hang in ClickHouse for ever. Using JDBC driver, which uses http protocol, won't run into this issue.

SHOW PROCESSLIST

Query id: 5026d02f-b13b-4ffa-a9bc-f7b764691617

┌─is_initial_query─┬─user────┬─query_id─────────────────────────────┬─address───────────────┬──port─┬─initial_user─┬─initial_query_id─────────────────────┬─initial_address───────┬─initial_port─┬─interface─┬─os_user─┬─client_hostname─┬─client_name─┬─client_revision─┬─client_version_major─┬─client_version_minor─┬─client_version_patch─┬─http_method─┬─http_user_agent─┬─http_referer─┬─forwarded_for─┬─quota_key─┬──────────elapsed─┬─is_cancelled─┬─read_rows─┬─read_bytes─┬─total_rows_approx─┬─written_rows─┬─written_bytes─┬─memory_usage─┬─peak_memory_usage─┬─query─────────────────────────────┬─thread_ids────┬─ProfileEvents.Names────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─ProfileEvents.Values───────────────────────────────────┬─Settings.Names────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─Settings.Values──────────────────┬─current_database─┐
│                1 │ default │ 0fb974b6-fbf8-4a96-a211-c6148c613ab7 │ ::ffff:192.168.84.249 │ 51284 │ default      │ 0fb974b6-fbf8-4a96-a211-c6148c613ab7 │ ::ffff:192.168.84.249 │        51284 │         3 │         │                 │             │               0 │                    0 │                    0 │                    0 │           0 │                 │              │               │           │ 338897.503978826 │            0 │         0 │          0 │                 0 │            1 │             9 │            0 │                 0 │ insert into test_insert(i) values │ [141]         │ ['Query','InsertQuery','QueryTimeMicroseconds','InsertQueryTimeMicroseconds','IOBufferAllocs','IOBufferAllocBytes','FunctionExecute','InsertedRows','InsertedBytes','SelectedRows','SelectedBytes','ContextLock','RWLockAcquiredReadLocks','RealTimeMicroseconds','UserTimeMicroseconds','SystemTimeMicroseconds'] │ [1,1,438,438,1,1048591,2,1,9,1,9,18,2,571,237,177]     │ ['max_threads','use_uncompressed_cache','aggregation_memory_efficient_merge_threads','max_execution_time','max_memory_usage'] │ ['4','0','0','300','6442450944'] │ system           │
│                1 │ default │ f6140ae8-8f77-49a9-bc26-d79471495334 │ ::ffff:192.168.84.249 │ 52600 │ default      │ f6140ae8-8f77-49a9-bc26-d79471495334 │ ::ffff:192.168.84.249 │        52600 │         3 │         │                 │             │               0 │                    0 │                    0 │                    0 │           0 │                 │              │               │           │ 338838.453987025 │            1 │         1 │          1 │                 0 │            0 │             0 │            0 │                 0 │ select toUInt32(5593) as n        │ [140,137]     │ ['Query','SelectQuery','QueryTimeMicroseconds','SelectQueryTimeMicroseconds','SelectedRows','SelectedBytes','ContextLock','RWLockAcquiredReadLocks','RealTimeMicroseconds','UserTimeMicroseconds','SoftPageFaults']                                        │ [1,1,983,983,1,1,20,1,1317,1010,1]                     │ ['max_threads','use_uncompressed_cache','aggregation_memory_efficient_merge_threads','max_execution_time','max_memory_usage'] │ ['4','0','0','300','6442450944'] │ system           │
└──────────────────┴─────────┴──────────────────────────────────────┴───────────────────────┴───────┴──────────────┴──────────────────────────────────────┴───────────────────────┴──────────────┴───────────┴─────────┴─────────────────┴─────────────┴─────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴─────────────┴─────────────────┴──────────────┴───────────────┴───────────┴──────────────────┴──────────────┴───────────┴────────────┴───────────────────┴──────────────┴───────────────┴──────────────┴───────────────────┴───────────────────────────────────┴───────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────┴──────────────────┘

SELECT *
FROM system.query_log
WHERE initial_query_id IN ('0fb974b6-fbf8-4a96-a211-c6148c613ab7', 'f6140ae8-8f77-49a9-bc26-d79471495334')

Query id: ee33fecd-6fd5-44b7-b22e-8eed795910d3

┌─type────────┬─event_date─┬──────────event_time─┬────event_time_microseconds─┬────query_start_time─┬─query_start_time_microseconds─┬─query_duration_ms─┬─read_rows─┬─read_bytes─┬─written_rows─┬─written_bytes─┬─result_rows─┬─result_bytes─┬─memory_usage─┬─current_database─┬─query─────────────────────────────┬─normalized_query_hash─┬─query_kind─┬─databases──┬─tables─────────────────┬─columns─┬─projections─┬─exception_code─┬─exception─┬─stack_trace─┬─is_initial_query─┬─user────┬─query_id─────────────────────────────┬─address───────────────┬──port─┬─initial_user─┬─initial_query_id─────────────────────┬─initial_address───────┬─initial_port─┬─interface─┬─os_user─┬─client_hostname─┬─client_name─┬─client_revision─┬─client_version_major─┬─client_version_minor─┬─client_version_patch─┬─http_method─┬─http_user_agent─┬─http_referer─┬─forwarded_for─┬─quota_key─┬─revision─┬─log_comment─┬─thread_ids─┬─ProfileEvents.Names────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─ProfileEvents.Values───────────────────────┬─Settings.Names────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─Settings.Values──────────────────┬─used_aggregate_functions─┬─used_aggregate_function_combinators─┬─used_database_engines─┬─used_data_type_families─┬─used_dictionaries─┬─used_formats─┬─used_functions─┬─used_storages─┬─used_table_functions─┐
│ QueryStart  │ 2021-07-26 │ 2021-07-26 13:45:37 │ 2021-07-26 13:45:37.063429 │ 2021-07-26 13:45:37 │    2021-07-26 13:45:37.063429 │                 0 │         0 │          0 │            0 │             0 │           0 │            0 │            0 │ system           │ insert into test_insert(i) values │  15736145110192260209 │ Insert     │ ['system'] │ ['system.test_insert'] │ []      │ []          │              0 │           │             │                1 │ default │ 0fb974b6-fbf8-4a96-a211-c6148c613ab7 │ ::ffff:192.168.84.249 │ 51284 │ default      │ 0fb974b6-fbf8-4a96-a211-c6148c613ab7 │ ::ffff:192.168.84.249 │        51284 │         3 │         │                 │             │               0 │                    0 │                    0 │                    0 │           0 │                 │              │               │           │    54451 │             │ []         │ []                                                                                                                                                                                                                                                         │ []                                         │ ['max_threads','use_uncompressed_cache','aggregation_memory_efficient_merge_threads','max_execution_time','max_memory_usage'] │ ['4','0','0','300','6442450944'] │ []                       │ []                                  │ []                    │ []                      │ []                │ []           │ []             │ []            │ []                   │
│ QueryFinish │ 2021-07-26 │ 2021-07-26 13:45:37 │ 2021-07-26 13:45:37.063913 │ 2021-07-26 13:45:37 │    2021-07-26 13:45:37.063429 │                 0 │         0 │          0 │            1 │             9 │           1 │            9 │            0 │ system           │ insert into test_insert(i) values │  15736145110192260209 │ Insert     │ ['system'] │ ['system.test_insert'] │ []      │ []          │              0 │           │             │                1 │ default │ 0fb974b6-fbf8-4a96-a211-c6148c613ab7 │ ::ffff:192.168.84.249 │ 51284 │ default      │ 0fb974b6-fbf8-4a96-a211-c6148c613ab7 │ ::ffff:192.168.84.249 │        51284 │         3 │         │                 │             │               0 │                    0 │                    0 │                    0 │           0 │                 │              │               │           │    54451 │             │ [141]      │ ['Query','InsertQuery','IOBufferAllocs','IOBufferAllocBytes','FunctionExecute','InsertedRows','InsertedBytes','SelectedRows','SelectedBytes','ContextLock','RWLockAcquiredReadLocks','RealTimeMicroseconds','UserTimeMicroseconds','SystemTimeMicroseconds'] │ [1,1,1,1048591,2,1,9,1,9,16,2,571,237,177] │ ['max_threads','use_uncompressed_cache','aggregation_memory_efficient_merge_threads','max_execution_time','max_memory_usage'] │ ['4','0','0','300','6442450944'] │ []                       │ []                                  │ []                    │ []                      │ []                │ ['Values']   │ ['replicate']  │ []            │ []                   │
└─────────────┴────────────┴─────────────────────┴────────────────────────────┴─────────────────────┴───────────────────────────────┴───────────────────┴───────────┴────────────┴──────────────┴───────────────┴─────────────┴──────────────┴──────────────┴──────────────────┴───────────────────────────────────┴───────────────────────┴────────────┴────────────┴────────────────────────┴─────────┴─────────────┴────────────────┴───────────┴─────────────┴──────────────────┴─────────┴──────────────────────────────────────┴───────────────────────┴───────┴──────────────┴──────────────────────────────────────┴───────────────────────┴──────────────┴───────────┴─────────┴─────────────────┴─────────────┴─────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴─────────────┴─────────────────┴──────────────┴───────────────┴───────────┴──────────┴─────────────┴────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────┴──────────────────────────┴─────────────────────────────────────┴───────────────────────┴─────────────────────────┴───────────────────┴──────────────┴────────────────┴───────────────┴──────────────────────┘
┌─type────────┬─event_date─┬──────────event_time─┬────event_time_microseconds─┬────query_start_time─┬─query_start_time_microseconds─┬─query_duration_ms─┬─read_rows─┬─read_bytes─┬─written_rows─┬─written_bytes─┬─result_rows─┬─result_bytes─┬─memory_usage─┬─current_database─┬─query──────────────────────┬─normalized_query_hash─┬─query_kind─┬─databases──┬─tables─────────┬─columns──────────────┬─projections─┬─exception_code─┬─exception─┬─stack_trace─┬─is_initial_query─┬─user────┬─query_id─────────────────────────────┬─address───────────────┬──port─┬─initial_user─┬─initial_query_id─────────────────────┬─initial_address───────┬─initial_port─┬─interface─┬─os_user─┬─client_hostname─┬─client_name─┬─client_revision─┬─client_version_major─┬─client_version_minor─┬─client_version_patch─┬─http_method─┬─http_user_agent─┬─http_referer─┬─forwarded_for─┬─quota_key─┬─revision─┬─log_comment─┬─thread_ids─┬─ProfileEvents.Names───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─ProfileEvents.Values───────┬─Settings.Names────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─Settings.Values──────────────────┬─used_aggregate_functions─┬─used_aggregate_function_combinators─┬─used_database_engines─┬─used_data_type_families─┬─used_dictionaries─┬─used_formats───────────────────┬─used_functions─┬─used_storages─┬─used_table_functions─┐
│ QueryStart  │ 2021-07-26 │ 2021-07-26 13:46:36 │ 2021-07-26 13:46:36.113378 │ 2021-07-26 13:46:36 │    2021-07-26 13:46:36.113378 │                 0 │         0 │          0 │            0 │             0 │           0 │            0 │            0 │ system           │ select toUInt32(5593) as n │  10364793771821666952 │ Select     │ ['system'] │ ['system.one'] │ ['system.one.dummy'] │ []          │              0 │           │             │                1 │ default │ f6140ae8-8f77-49a9-bc26-d79471495334 │ ::ffff:192.168.84.249 │ 52600 │ default      │ f6140ae8-8f77-49a9-bc26-d79471495334 │ ::ffff:192.168.84.249 │        52600 │         3 │         │                 │             │               0 │                    0 │                    0 │                    0 │           0 │                 │              │               │           │    54451 │             │ []         │ []                                                                                                                                                            │ []                         │ ['max_threads','use_uncompressed_cache','aggregation_memory_efficient_merge_threads','max_execution_time','max_memory_usage'] │ ['4','0','0','300','6442450944'] │ []                       │ []                                  │ []                    │ []                      │ []                │ []                             │ []             │ []            │ []                   │
│ QueryFinish │ 2021-07-26 │ 2021-07-26 13:46:36 │ 2021-07-26 13:46:36.114464 │ 2021-07-26 13:46:36 │    2021-07-26 13:46:36.113378 │                 0 │         1 │          1 │            0 │             0 │           1 │           72 │            0 │ system           │ select toUInt32(5593) as n │  10364793771821666952 │ Select     │ ['system'] │ ['system.one'] │ ['system.one.dummy'] │ []          │              0 │           │             │                1 │ default │ f6140ae8-8f77-49a9-bc26-d79471495334 │ ::ffff:192.168.84.249 │ 52600 │ default      │ f6140ae8-8f77-49a9-bc26-d79471495334 │ ::ffff:192.168.84.249 │        52600 │         3 │         │                 │             │               0 │                    0 │                    0 │                    0 │           0 │                 │              │               │           │    54451 │             │ [140,137]  │ ['Query','SelectQuery','SelectedRows','SelectedBytes','ContextLock','RWLockAcquiredReadLocks','RealTimeMicroseconds','UserTimeMicroseconds','SoftPageFaults'] │ [1,1,1,1,18,1,1317,1010,1] │ ['max_threads','use_uncompressed_cache','aggregation_memory_efficient_merge_threads','max_execution_time','max_memory_usage'] │ ['4','0','0','300','6442450944'] │ []                       │ []                                  │ []                    │ []                      │ []                │ ['RowBinaryWithNamesAndTypes'] │ ['toUInt32']   │ []            │ []                   │
└─────────────┴────────────┴─────────────────────┴────────────────────────────┴─────────────────────┴───────────────────────────────┴───────────────────┴───────────┴────────────┴──────────────┴───────────────┴─────────────┴──────────────┴──────────────┴──────────────────┴────────────────────────────┴───────────────────────┴────────────┴────────────┴────────────────┴──────────────────────┴─────────────┴────────────────┴───────────┴─────────────┴──────────────────┴─────────┴──────────────────────────────────────┴───────────────────────┴───────┴──────────────┴──────────────────────────────────────┴───────────────────────┴──────────────┴───────────┴─────────┴─────────────────┴─────────────┴─────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴─────────────┴─────────────────┴──────────────┴───────────────┴───────────┴──────────┴─────────────┴────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────┴──────────────────────────┴─────────────────────────────────────┴───────────────────────┴─────────────────────────┴───────────────────┴────────────────────────────────┴────────────────┴───────────────┴──────────────────────┘

SELECT *
FROM system.query_thread_log
WHERE initial_query_id IN ('0fb974b6-fbf8-4a96-a211-c6148c613ab7', 'f6140ae8-8f77-49a9-bc26-d79471495334')

Query id: 4e5fe055-25a6-4033-a055-1872f8fb6c3e

┌─event_date─┬──────────event_time─┬────event_time_microseconds─┬────query_start_time─┬─query_start_time_microseconds─┬─query_duration_ms─┬─read_rows─┬─read_bytes─┬─written_rows─┬─written_bytes─┬─memory_usage─┬─peak_memory_usage─┬─thread_name─────┬─thread_id─┬─master_thread_id─┬─current_database─┬─query─────────────────────────────┬─normalized_query_hash─┬─is_initial_query─┬─user────┬─query_id─────────────────────────────┬─address───────────────┬──port─┬─initial_user─┬─initial_query_id─────────────────────┬─initial_address───────┬─initial_port─┬─interface─┬─os_user─┬─client_hostname─┬─client_name─┬─client_revision─┬─client_version_major─┬─client_version_minor─┬─client_version_patch─┬─http_method─┬─http_user_agent─┬─http_referer─┬─forwarded_for─┬─quota_key─┬─revision─┬─ProfileEvents.Names────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─ProfileEvents.Values───────────────────────┐
│ 2021-07-26 │ 2021-07-26 13:45:37 │ 2021-07-26 13:45:37.063891 │ 2021-07-26 13:45:37 │    2021-07-26 13:45:37.063305 │                 0 │         0 │          0 │            1 │             9 │            0 │                 0 │ QueryPipelineEx │       141 │              141 │ system           │ insert into test_insert(i) values │  15736145110192260209 │                1 │ default │ 0fb974b6-fbf8-4a96-a211-c6148c613ab7 │ ::ffff:192.168.84.249 │ 51284 │ default      │ 0fb974b6-fbf8-4a96-a211-c6148c613ab7 │ ::ffff:192.168.84.249 │        51284 │         3 │         │                 │             │               0 │                    0 │                    0 │                    0 │           0 │                 │              │               │           │    54451 │ ['Query','InsertQuery','IOBufferAllocs','IOBufferAllocBytes','FunctionExecute','InsertedRows','InsertedBytes','SelectedRows','SelectedBytes','ContextLock','RWLockAcquiredReadLocks','RealTimeMicroseconds','UserTimeMicroseconds','SystemTimeMicroseconds'] │ [1,1,1,1048591,2,1,9,1,9,16,2,571,237,177] │
└────────────┴─────────────────────┴────────────────────────────┴─────────────────────┴───────────────────────────────┴───────────────────┴───────────┴────────────┴──────────────┴───────────────┴──────────────┴───────────────────┴─────────────────┴───────────┴──────────────────┴──────────────────┴───────────────────────────────────┴───────────────────────┴──────────────────┴─────────┴──────────────────────────────────────┴───────────────────────┴───────┴──────────────┴──────────────────────────────────────┴───────────────────────┴──────────────┴───────────┴─────────┴─────────────────┴─────────────┴─────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴─────────────┴─────────────────┴──────────────┴───────────────┴───────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────┘
┌─event_date─┬──────────event_time─┬────event_time_microseconds─┬────query_start_time─┬─query_start_time_microseconds─┬─query_duration_ms─┬─read_rows─┬─read_bytes─┬─written_rows─┬─written_bytes─┬─memory_usage─┬─peak_memory_usage─┬─thread_name─────┬─thread_id─┬─master_thread_id─┬─current_database─┬─query──────────────────────┬─normalized_query_hash─┬─is_initial_query─┬─user────┬─query_id─────────────────────────────┬─address───────────────┬──port─┬─initial_user─┬─initial_query_id─────────────────────┬─initial_address───────┬─initial_port─┬─interface─┬─os_user─┬─client_hostname─┬─client_name─┬─client_revision─┬─client_version_major─┬─client_version_minor─┬─client_version_patch─┬─http_method─┬─http_user_agent─┬─http_referer─┬─forwarded_for─┬─quota_key─┬─revision─┬─ProfileEvents.Names────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─ProfileEvents.Values───┐
│ 2021-07-26 │ 2021-07-26 13:46:36 │ 2021-07-26 13:46:36.114308 │ 2021-07-26 13:46:36 │    2021-07-26 13:46:36.114169 │                 0 │         1 │          1 │            0 │             0 │            0 │                 0 │ QueryPipelineEx │       137 │              140 │ system           │ select toUInt32(5593) as n │  10364793771821666952 │                1 │ default │ f6140ae8-8f77-49a9-bc26-d79471495334 │ ::ffff:192.168.84.249 │ 52600 │ default      │ f6140ae8-8f77-49a9-bc26-d79471495334 │ ::ffff:192.168.84.249 │        52600 │         3 │         │                 │             │               0 │                    0 │                    0 │                    0 │           0 │                 │              │               │           │    54451 │ ['SelectedRows','SelectedBytes','ContextLock','RealTimeMicroseconds']                                                          │ [1,1,2,107]            │
│ 2021-07-26 │ 2021-07-26 13:46:36 │ 2021-07-26 13:46:36.114441 │ 2021-07-26 13:46:36 │    2021-07-26 13:46:36.113216 │                 1 │         0 │          0 │            0 │             0 │            0 │                 0 │ QueryPipelineEx │       140 │              140 │ system           │ select toUInt32(5593) as n │  10364793771821666952 │                1 │ default │ f6140ae8-8f77-49a9-bc26-d79471495334 │ ::ffff:192.168.84.249 │ 52600 │ default      │ f6140ae8-8f77-49a9-bc26-d79471495334 │ ::ffff:192.168.84.249 │        52600 │         3 │         │                 │             │               0 │                    0 │                    0 │                    0 │           0 │                 │              │               │           │    54451 │ ['Query','SelectQuery','ContextLock','RWLockAcquiredReadLocks','RealTimeMicroseconds','UserTimeMicroseconds','SoftPageFaults'] │ [1,1,16,1,1210,1010,1] │
└────────────┴─────────────────────┴────────────────────────────┴─────────────────────┴───────────────────────────────┴───────────────────┴───────────┴────────────┴──────────────┴───────────────┴──────────────┴───────────────────┴─────────────────┴───────────┴──────────────────┴──────────────────┴────────────────────────────┴───────────────────────┴──────────────────┴─────────┴──────────────────────────────────────┴───────────────────────┴───────┴──────────────┴──────────────────────────────────────┴───────────────────────┴──────────────┴───────────┴─────────┴─────────────────┴─────────────┴─────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴─────────────┴─────────────────┴──────────────┴───────────────┴───────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┘

I posted some more information at here but not sure if that helps.

Expected Behavior
No query hangs in server - if we have to live with that, at least they can be killed.

[vitlibar]

Can you please provide a bit more information about how you called ClickHouse via gRPC? There could be multiple ways to do that, I mean in the service defined in clickhouse_grpc.proto we have four methods, which one were you using? Can you check if that hanging can be reproduced with clickhouse-grpc-client.py?

[zhicwu]

Can you please provide a bit more information about how you called ClickHouse via gRPC?

Sorry I forgot to mention here. I use ExecuteQueryWithStreamIO with a single QueryInfo(because I was testing simple queries). Also I modified clickhouse_grpc.proto by changing type of 3 fields(data, input_data, and output) from string to bytes - these two types are the same in C++ but they are very different in Java.

Can you check if that hanging can be reproduced with clickhouse-grpc-client.py?

I didn't get a chance to try that, as I'm still testing the Java client. You can use below docker image to reproduce the issue(assume your clickhouse server is my_server and the insecure grpc port is 9100):

docker run --rm -it zhicwu/clickhouse-java-test bash
# usage:
# -t <threads>
# -p connection=<reuse|new> whether to reuse connection for queries or not
java -DdbHost="my-server" -jar benchmarks.jar jmh -t 2 -p connection=reuse Simple
# in case you need to review the modified IDL
unzip -p benchmarks.jar clickhouse_grpc.proto

[vitlibar]

You can use below docker image to reproduce the issue(assume your clickhouse server is my_server and the insecure grpc port is 9100):

docker run --rm -it zhicwu/clickhouse-java-test bash
# usage:
# -t <threads>
# -p connection=<reuse|new> whether to reuse connection for queries or not
java -DdbHost="my-server" -jar benchmarks.jar jmh -t 2 -p connection=reuse Simple
# in case you need to review the modified IDL
unzip -p benchmarks.jar clickhouse_grpc.proto

Your benchmark was very useful, it helped me to reproduce the issue. So now I know how to fix that.

[vitlibar]

Also I modified clickhouse_grpc.proto by changing type of 3 fields(data, input_data, and output) from string to bytes - these two types are the same in C++ but they are very different in Java.

I've changed type of those and some other fields from string to bytes in our repository too, thanks for the idea. #27431

[zhicwu]

Confirmed that this has been fixed. Below is the test results(using 16 threads) against 21.7.6.39:

Benchmark                                   (client)   (connection)   Mode  Cnt    Score    Error  Units
Query.select10kStringRows     clickhouse-grpc-client          reuse  thrpt   20  235.880 ± 22.463  ops/s
Query.select10kStringRows     clickhouse-grpc-client            new  thrpt   20  221.185 ± 20.566  ops/s
Query.select10kTimestampRows  clickhouse-grpc-client          reuse  thrpt   20  257.300 ± 12.860  ops/s
Query.select10kTimestampRows  clickhouse-grpc-client            new  thrpt   20  218.527 ± 33.856  ops/s
Query.select10kUInt64Rows     clickhouse-grpc-client          reuse  thrpt   20  268.650 ± 20.244  ops/s
Query.select10kUInt64Rows     clickhouse-grpc-client            new  thrpt   20  249.088 ± 27.682  ops/s

Query.select10kStringRows     clickhouse-jdbc                 reuse  thrpt   20  232.634 ± 23.928  ops/s
Query.select10kTimestampRows  clickhouse-jdbc                 reuse  thrpt   20   94.483 ± 29.697  ops/s
Query.select10kUInt64Rows     clickhouse-jdbc                 reuse  thrpt   20  107.952 ± 75.091  ops/s
Query.select10kStringRows     clickhouse4j                    reuse  thrpt   20  217.680 ± 21.593  ops/s
Query.select10kTimestampRows  clickhouse4j                    reuse  thrpt   20  177.652 ± 17.755  ops/s
Query.select10kUInt64Rows     clickhouse4j                    reuse  thrpt   20  231.947 ± 14.831  ops/s
Query.select10kStringRows     clickhouse-native-jdbc-shaded   reuse  thrpt   20  134.540 ± 11.052  ops/s
Query.select10kTimestampRows  clickhouse-native-jdbc-shaded   reuse  thrpt   20  132.243 ± 18.975  ops/s
Query.select10kUInt64Rows     clickhouse-native-jdbc-shaded   reuse  thrpt   20  132.965 ± 10.164  ops/s
Query.select10kStringRows     mysql-connector-java            reuse  thrpt   20   95.085 ± 10.749  ops/s
Query.select10kTimestampRows  mysql-connector-java            reuse  thrpt   20   38.540 ±  5.320  ops/s
Query.select10kUInt64Rows     mysql-connector-java            reuse  thrpt   20   97.415 ± 15.821  ops/s
Query.select10kStringRows     mariadb-java-client             reuse  thrpt   20   76.576 ± 10.753  ops/s
Query.select10kTimestampRows  mariadb-java-client             reuse  thrpt   20   42.612 ±  6.034  ops/s
Query.select10kUInt64Rows     mariadb-java-client             reuse  thrpt   20   67.680 ±  6.833  ops/s
Query.select10kStringRows     postgresql-jdbc                 reuse  thrpt   20   45.994 ±  8.306  ops/s
Query.select10kTimestampRows  postgresql-jdbc                 reuse  thrpt   20   26.973 ±  8.505  ops/s
Query.select10kUInt64Rows     postgresql-jdbc                 reuse  thrpt   20   64.671 ± 10.700  ops/s