Describe the situation
I have a RabbitMQ queue which receives messages at a rate about 20,000s.
CREATE TABLE default.Queue
(
`Data` String
)
ENGINE = RabbitMQ
SETTINGS rabbitmq_host_port = 'host:5672',
rabbitmq_exchange_name = 'Exchange',
rabbitmq_format = 'JSONAsString',
rabbitmq_exchange_type = 'direct',
rabbitmq_routing_key_list = 'messages',
rabbitmq_queue_settings_list = 'x-max-length=2000000, x-overflow=reject-publish',
rabbitmq_flush_interval_ms = 100,
date_time_input_format = 'best_effort';
In materialized view I parse the message with simpleJSONExtractString functions and push the data to AggregatingMergeTree.
In RabbitMQ my queue fill up pretty quickly:
but in the log I see that CH quite often "processed 0 rows" despite there are plenty of messages in the queue:
Not sure how what to do to fix it:
- Tried tweaking
rabbitmq_flush_interval_ms - but the queue still ends up full
- Tried adding more consumers - that will work for some time but at some point both consumers disconnect and never reconnect again.
Describe the situation
I have a RabbitMQ queue which receives messages at a rate about 20,000s.
In materialized view I parse the message with
simpleJSONExtractStringfunctions and push the data toAggregatingMergeTree.In RabbitMQ my queue fill up pretty quickly:
but in the log I see that CH quite often "processed 0 rows" despite there are plenty of messages in the queue:
Not sure how what to do to fix it:
rabbitmq_flush_interval_ms- but the queue still ends up full