Queue Full on producer

[pratikbatua]

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

I have tried multiple settings mentioned in other issues however the problem still persist. Producer rate is ~100K messages per second. Below are the config settings I am using -

(metadata.broker.list;localhost:9092);
(statistics.interval.ms;10000);
(queue.buffering.max.ms;0);
(queue.buffering.max.messages;10000000);
(queue.buffering.max.kbytes;2097151);
(linger.ms;5);
(batch.num.messages;1);
(fetch.wait.max.ms;0);
(request.required.acks;0)

Increasing the buffersize only delay the problem.

How to reproduce

I have 3G size file which I read and use to generate the message and continuously publish each message. The message volume is~1.4 million. The producer breaks neat 700K after applying above settings.

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 0.11

• Apache Kafka version: 2.11-2.1.0

• librdkafka client configuration:

(metadata.broker.list;localhost:9092);
(statistics.interval.ms;10000);
(queue.buffering.max.ms;0);
(queue.buffering.max.messages;10000000);
(queue.buffering.max.kbytes;2097151);
(linger.ms;5);
(batch.num.messages;1);
(fetch.wait.max.ms;0);
(request.required.acks;0)

• Operating system: x86_64 GNU/Linux RHEL 7

• Provide logs (with debug=.. as necessary) from librdkafka

Queue Full

• Provide broker log excerpts

• Critical issue

[edenhill]

queue.buffering.max.ms and linger.ms are the same thing (the latter aliases the former).
For thruput, increase linger.ms (from its default of 0) so that batches have a chance to accumulate.
Larger batches -> less overhead -> higher thruput.

In your case I'd suggest starting at 10ms and going upwards in increments of 10 until you find the sweet spot between latency and throughput.

Also, when you hit ERR__QUEUE_FULL you should call poll(some timeout) and then retry sending.
The reason for this is that all messages inside librdkafka are accounted for when queue.buffering.max.messages is enforced, including messages that have been delivered but whose delivery callback/event has not been fired yet - and those are fired by polling.

[pratikbatua]

Thanks for the response.

With suggested changes , I am able to delay the Queue Full error however it hits somehow.

Regarding introducing delay, this is not desirable solution as I am planning to use kakfa to consume real-time market data which very high frequency data. so even a wait for few milli second is not an ideal solution.

Also note that program hits Queue full error around 850K messages from the total of 1.4 mm messages I am sending. Below is the settings I am using as per you suggestion before.

(metadata.broker.list;localhost:9092);
(statistics.interval.ms;10000);
(queue.buffering.max.ms;40);
(queue.buffering.max.messages;10000000);
(queue.buffering.max.kbytes;2097151);
(batch.num.messages;10);
(fetch.wait.max.ms;0);
(request.required.acks;0)

ERROR :
'Local: Queue full

[edenhill]

I suggest you enable statistics by setting a stats_cb and setting statistics.interval.ms to e.g., 1000 ms.
Then monitor msg_and msg_size in https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md#top-level, to see which limit you are hitting.

[edenhill]

Why are you setting batch.num.messages=10, btw? You want as large batches as possible, leave it at the default value.

You also never want to use request.required.acks=0, you might as well just throw away your messages since you will have no indication if they were delivered or not.
Strongly suggest acks=all, which is the proper reliability guarantee, but at least acks=1.

[pratikbatua]

I was playing around batch size and acks to see if that helps to eliminate the error. I have taken a snapshot for 10 seconds before producer got stuck with Queue Full error.

(metadata.broker.list;localhost:9092);
(statistics.interval.ms;1000);
(queue.buffering.max.ms;40);
(queue.buffering.max.messages;10000000);
(queue.buffering.max.kbytes;2097151);
(fetch.wait.max.ms;0);
(request.required.acks;0)

I don't see anything problematic.

https://gist.github.com/pratikbatua/af9885e1d832129bcfc0f3ff1b467d60

[edenhill]

You're still using acks=0 (fire and forget), highly recommend all or 1.

[pratikbatua]

I was using it to ensure that there is no backlog of messages on producer for which ack is pending.

I ran the test with below props. I can see that the after ~40 seconds the producer queue message size reaches max capacity.

I really don't have an option to wait in between on producer side as can't delay the market data flow .

what would be your suggestion ?

(metadata.broker.list;localhost:9092);
(statistics.interval.ms;1000);
(queue.buffering.max.ms;40);
(queue.buffering.max.messages;10000000);
(queue.buffering.max.kbytes;2097151);
(fetch.wait.max.ms;0);
(request.required.acks;1)

type	ts	time	replyq	msg_cnt	msg_size	msg_max	msg_size_max	simple_cnt	metadata_cache_cnt	tx	tx_bytes	rx	rx_bytes	txmsgs	txmsg_bytes	rxmsgs	rxmsg_bytes
producer	1970-01-20D21:06:26.387708000	2019-03-12D18:58:50.000000000	2	0	0	1.00E+07	2.15E+09	0	1	41	1937	41	10278	0	0	0	0
producer	1970-01-20D21:06:27.387813000	2019-03-12D18:58:51.000000000	3	0	0	1.00E+07	2.15E+09	0	1	42	1986	42	10527	0	0	0	0
producer	1970-01-20D21:06:28.387922000	2019-03-12D18:58:52.000000000	4	0	0	1.00E+07	2.15E+09	0	1	43	2035	43	10776	0	0	0	0
producer	1970-01-20D21:06:29.388036000	2019-03-12D18:58:53.000000000	5	0	0	1.00E+07	2.15E+09	0	1	44	2084	44	11025	0	0	0	0
producer	1970-01-20D21:06:30.388217000	2019-03-12D18:58:54.000000000	6	0	0	1.00E+07	2.15E+09	0	1	45	2133	45	11274	0	0	0	0
producer	1970-01-20D21:06:31.388327000	2019-03-12D18:58:55.000000000	7	0	0	1.00E+07	2.15E+09	0	1	46	2182	46	11523	0	0	0	0
producer	1970-01-20D21:06:32.388440000	2019-03-12D18:58:56.000000000	8	0	0	1.00E+07	2.15E+09	0	1	47	2231	47	11772	0	0	0	0
producer	1970-01-20D21:06:33.388566000	2019-03-12D18:58:57.000000000	9	0	0	1.00E+07	2.15E+09	0	1	47	2231	47	11772	0	0	0	0
producer	1970-01-20D21:06:34.388639000	2019-03-12D18:58:58.000000000	10	0	0	1.00E+07	2.15E+09	0	1	49	2329	49	12270	0	0	0	0
producer	1970-01-20D21:06:35.388773000	2019-03-12D18:58:59.000000000	11	0	0	1.00E+07	2.15E+09	0	1	50	2378	50	12519	0	0	0	0
producer	1970-01-20D21:06:36.388898000	2019-03-12D18:59:00.000000000	12	0	0	1.00E+07	2.15E+09	0	1	51	2427	51	12768	0	0	0	0
producer	1970-01-20D21:06:37.389120000	2019-03-12D18:59:01.000000000	13	0	0	1.00E+07	2.15E+09	0	1	52	2476	52	13017	0	0	0	0
producer	1970-01-20D21:06:38.389231000	2019-03-12D18:59:02.000000000	14	0	0	1.00E+07	2.15E+09	0	1	53	2525	53	13266	0	0	0	0
producer	1970-01-20D21:06:39.389344000	2019-03-12D18:59:03.000000000	15	0	0	1.00E+07	2.15E+09	0	1	54	2574	54	13515	0	0	0	0
producer	1970-01-20D21:06:40.389451000	2019-03-12D18:59:04.000000000	16	0	0	1.00E+07	2.15E+09	0	1	55	2623	55	13764	0	0	0	0
producer	1970-01-20D21:06:41.389566000	2019-03-12D18:59:05.000000000	17	0	0	1.00E+07	2.15E+09	0	1	56	2672	56	14013	0	0	0	0
producer	1970-01-20D21:06:42.389672000	2019-03-12D18:59:06.000000000	18	0	0	1.00E+07	2.15E+09	0	1	57	2721	57	14262	0	0	0	0
producer	1970-01-20D21:06:43.389804000	2019-03-12D18:59:07.000000000	19	0	0	1.00E+07	2.15E+09	0	1	58	2770	58	14511	0	0	0	0
producer	1970-01-20D21:06:44.389927000	2019-03-12D18:59:08.000000000	20	0	0	1.00E+07	2.15E+09	0	1	59	2819	59	14760	0	0	0	0
producer	1970-01-20D21:06:45.390044000	2019-03-12D18:59:09.000000000	21	0	0	1.00E+07	2.15E+09	0	1	60	2868	60	15009	0	0	0	0
producer	1970-01-20D21:06:46.390146000	2019-03-12D18:59:10.000000000	22	0	0	1.00E+07	2.15E+09	0	1	61	2917	61	15258	0	0	0	0
producer	1970-01-20D21:06:47.390251000	2019-03-12D18:59:11.000000000	23	0	0	1.00E+07	2.15E+09	0	1	62	2966	61	15258	0	0	0	0
producer	1970-01-20D21:06:48.390332000	2019-03-12D18:59:12.000000000	24	0	0	1.00E+07	2.15E+09	0	1	63	3015	63	15756	0	0	0	0
producer	1970-01-20D21:06:49.390497000	2019-03-12D18:59:13.000000000	25	0	0	1.00E+07	2.15E+09	0	1	64	3064	64	16005	0	0	0	0
producer	1970-01-20D21:06:50.390676000	2019-03-12D18:59:14.000000000	26	0	0	1.00E+07	2.15E+09	0	1	65	3113	65	16254	0	0	0	0
producer	1970-01-20D21:06:51.390788000	2019-03-12D18:59:15.000000000	27	0	0	1.00E+07	2.15E+09	0	1	66	3162	66	16503	0	0	0	0
producer	1970-01-20D21:06:52.391068000	2019-03-12D18:59:16.000000000	28	0	0	1.00E+07	2.15E+09	0	1	67	3211	67	16752	0	0	0	0
producer	1970-01-20D21:06:53.391199000	2019-03-12D18:59:17.000000000	29	0	0	1.00E+07	2.15E+09	0	1	68	3260	68	17001	0	0	0	0
producer	1970-01-20D21:06:54.395034000	2019-03-12D18:59:18.000000000	30	0	0	1.00E+07	2.15E+09	0	1	69	3309	69	17250	0	0	0	0
producer	1970-01-20D21:06:55.395983000	2019-03-12D18:59:19.000000000	31	0	0	1.00E+07	2.15E+09	0	1	70	3358	70	17499	0	0	0	0
producer	1970-01-20D21:06:56.396109000	2019-03-12D18:59:20.000000000	32	0	0	1.00E+07	2.15E+09	0	1	71	3407	71	17748	0	0	0	0
producer	1970-01-20D21:06:57.396221000	2019-03-12D18:59:21.000000000	33	0	0	1.00E+07	2.15E+09	0	1	72	3456	72	17997	0	0	0	0
producer	1970-01-20D21:06:58.396581000	2019-03-12D18:59:22.000000000	34	0	0	1.00E+07	2.15E+09	0	1	73	3505	73	18246	0	0	0	0
producer	1970-01-20D21:06:59.397822000	2019-03-12D18:59:23.000000000	35	0	0	1.00E+07	2.15E+09	0	1	74	3554	74	18495	0	0	0	0
producer	1970-01-20D21:07:00.398149000	2019-03-12D18:59:24.000000000	36	0	0	1.00E+07	2.15E+09	0	1	75	3603	75	18744	0	0	0	0
producer	1970-01-20D21:07:01.399524000	2019-03-12D18:59:25.000000000	37	0	0	1.00E+07	2.15E+09	0	1	76	3652	75	18744	0	0	0	0
producer	1970-01-20D21:07:02.399599000	2019-03-12D18:59:26.000000000	38	0	0	1.00E+07	2.15E+09	0	1	77	3701	77	19242	0	0	0	0
producer	1970-01-20D21:07:03.399824000	2019-03-12D18:59:27.000000000	39	0	0	1.00E+07	2.15E+09	0	1	78	3750	78	19491	0	0	0	0
producer	1970-01-20D21:07:04.399933000	2019-03-12D18:59:28.000000000	40	0	0	1.00E+07	2.15E+09	0	1	79	3799	79	19740	0	0	0	0
producer	1970-01-20D21:07:05.400061000	2019-03-12D18:59:29.000000000	41	0	0	1.00E+07	2.15E+09	0	1	80	3848	80	19989	0	0	0	0
producer	1970-01-20D21:07:06.400246000	2019-03-12D18:59:30.000000000	42	0	0	1.00E+07	2.15E+09	0	1	80	3848	80	19989	0	0	0	0
producer	1970-01-20D21:07:07.400326000	2019-03-12D18:59:31.000000000	43	0	0	1.00E+07	2.15E+09	0	1	82	3946	82	20487	0	0	0	0
producer	1970-01-20D21:07:08.400435000	2019-03-12D18:59:32.000000000	44	0	0	1.00E+07	2.15E+09	0	1	83	3995	83	20736	0	0	0	0
producer	1970-01-20D21:06:24.313957000	2019-03-12D18:58:48.000000000	0	0	0	1.00E+07	2.15E+09	0	1	8	320	8	2061	0	0	0	0
producer	1970-01-20D21:06:25.314094000	2019-03-12D18:58:49.000000000	1	0	0	1.00E+07	2.15E+09	0	1	9	369	9	2310	0	0	0	0
producer	1970-01-20D21:06:26.314213000	2019-03-12D18:58:50.000000000	2	0	0	1.00E+07	2.15E+09	0	1	11	467	10	2559	0	0	0	0
producer	1970-01-20D21:06:27.314337000	2019-03-12D18:58:51.000000000	3	0	0	1.00E+07	2.15E+09	0	1	11	467	11	2808	0	0	0	0
producer	1970-01-20D21:06:28.314632000	2019-03-12D18:58:52.000000000	4	0	0	1.00E+07	2.15E+09	0	1	13	565	12	3057	0	0	0	0
producer	1970-01-20D21:06:29.314747000	2019-03-12D18:58:53.000000000	5	0	0	1.00E+07	2.15E+09	0	1	14	614	14	3555	0	0	0	0
producer	1970-01-20D21:06:30.314892000	2019-03-12D18:58:54.000000000	6	0	0	1.00E+07	2.15E+09	0	1	14	614	14	3555	0	0	0	0
producer	1970-01-20D21:06:31.314963000	2019-03-12D18:58:55.000000000	7	0	0	1.00E+07	2.15E+09	0	1	16	712	16	4053	0	0	0	0
producer	1970-01-20D21:06:32.315118000	2019-03-12D18:58:56.000000000	8	0	0	1.00E+07	2.15E+09	0	1	17	761	17	4302	0	0	0	0
producer	1970-01-20D21:06:33.315236000	2019-03-12D18:58:57.000000000	9	0	0	1.00E+07	2.15E+09	0	1	18	810	18	4551	0	0	0	0
producer	1970-01-20D21:06:34.315341000	2019-03-12D18:58:58.000000000	10	0	0	1.00E+07	2.15E+09	0	1	19	859	19	4800	0	0	0	0
producer	1970-01-20D21:06:35.315444000	2019-03-12D18:58:59.000000000	11	0	0	1.00E+07	2.15E+09	0	1	20	908	20	5049	0	0	0	0
producer	1970-01-20D21:06:36.315539000	2019-03-12D18:59:00.000000000	12	0	0	1.00E+07	2.15E+09	0	1	21	957	21	5298	0	0	0	0
producer	1970-01-20D21:06:37.315650000	2019-03-12D18:59:01.000000000	13	0	0	1.00E+07	2.15E+09	0	1	22	1006	22	5547	0	0	0	0
producer	1970-01-20D21:06:38.315750000	2019-03-12D18:59:02.000000000	14	0	0	1.00E+07	2.15E+09	0	1	23	1055	23	5796	0	0	0	0
producer	1970-01-20D21:06:39.316017000	2019-03-12D18:59:03.000000000	15	0	0	1.00E+07	2.15E+09	0	1	24	1104	24	6045	0	0	0	0
producer	1970-01-20D21:06:40.316128000	2019-03-12D18:59:04.000000000	16	0	0	1.00E+07	2.15E+09	0	1	25	1153	25	6294	0	0	0	0
producer	1970-01-20D21:06:41.316246000	2019-03-12D18:59:05.000000000	17	0	0	1.00E+07	2.15E+09	0	1	25	1153	25	6294	0	0	0	0
producer	1970-01-20D21:06:42.316322000	2019-03-12D18:59:06.000000000	18	0	0	1.00E+07	2.15E+09	0	1	27	1251	27	6792	0	0	0	0
producer	1970-01-20D21:06:43.316437000	2019-03-12D18:59:07.000000000	19	0	0	1.00E+07	2.15E+09	0	1	28	1300	28	7041	0	0	0	0
producer	1970-01-20D21:06:44.316534000	2019-03-12D18:59:08.000000000	20	0	0	1.00E+07	2.15E+09	0	1	29	1349	29	7290	0	0	0	0
producer	1970-01-20D21:06:45.316637000	2019-03-12D18:59:09.000000000	21	0	0	1.00E+07	2.15E+09	0	1	30	1398	30	7539	0	0	0	0
producer	1970-01-20D21:06:46.316743000	2019-03-12D18:59:10.000000000	189	51283	1.26E+08	1.00E+07	2.15E+09	0	1	681	1.20E+08	198	18627	49120	1.21E+08	0	0
producer	1970-01-20D21:06:47.317693000	2019-03-12D18:59:11.000000000	382	103613	2.70E+08	1.00E+07	2.15E+09	0	1	1461	2.58E+08	391	31731	98897	2.57E+08	0	0
producer	1970-01-20D21:06:48.323435000	2019-03-12D18:59:12.000000000	562	156844	4.05E+08	1.00E+07	2.15E+09	0	1	2190	3.89E+08	570	43545	150447	3.88E+08	0	0
producer	1970-01-20D21:06:49.325614000	2019-03-12D18:59:13.000000000	725	202179	5.20E+08	1.00E+07	2.15E+09	0	1	2809	4.98E+08	734	54735	192997	4.96E+08	0	0
producer	1970-01-20D21:06:50.326615000	2019-03-12D18:59:14.000000000	883	245073	6.30E+08	1.00E+07	2.15E+09	0	1	3389	6.02E+08	892	65346	233472	6.00E+08	0	0
producer	1970-01-20D21:06:51.327714000	2019-03-12D18:59:15.000000000	1047	295651	7.58E+08	1.00E+07	2.15E+09	0	1	4059	7.25E+08	1056	76353	282608	7.25E+08	0	0
producer	1970-01-20D21:06:52.328788000	2019-03-12D18:59:16.000000000	1196	337228	8.64E+08	1.00E+07	2.15E+09	0	1	4632	8.31E+08	1205	86370	322918	8.28E+08	0	0
producer	1970-01-20D21:06:53.329702000	2019-03-12D18:59:17.000000000	1351	380392	9.76E+08	1.00E+07	2.15E+09	0	1	5230	9.38E+08	1360	96783	364298	9.35E+08	0	0
producer	1970-01-20D21:06:54.330909000	2019-03-12D18:59:18.000000000	1532	433047	1.11E+09	1.00E+07	2.15E+09	0	1	5929	1.07E+09	1542	108978	414758	1.06E+09	0	0
producer	1970-01-20D21:06:55.332108000	2019-03-12D18:59:19.000000000	1696	481374	1.23E+09	1.00E+07	2.15E+09	0	1	6581	1.19E+09	1705	119919	461062	1.18E+09	0	0
producer	1970-01-20D21:06:56.333311000	2019-03-12D18:59:20.000000000	1871	529750	1.36E+09	1.00E+07	2.15E+09	0	1	7262	1.31E+09	1880	131652	507668	1.30E+09	0	0
producer	1970-01-20D21:06:57.335384000	2019-03-12D18:59:21.000000000	2047	578769	1.48E+09	1.00E+07	2.15E+09	0	1	7927	1.43E+09	2056	143451	554831	1.42E+09	0	0
producer	1970-01-20D21:06:58.336514000	2019-03-12D18:59:22.000000000	2216	625652	1.60E+09	1.00E+07	2.15E+09	0	1	8557	1.54E+09	2225	154788	599057	1.53E+09	0	0
producer	1970-01-20D21:06:59.337471000	2019-03-12D18:59:23.000000000	2377	669993	1.71E+09	1.00E+07	2.15E+09	0	1	9149	1.65E+09	2386	165597	642454	1.64E+09	0	0
producer	1970-01-20D21:07:00.338503000	2019-03-12D18:59:24.000000000	2541	714642	1.83E+09	1.00E+07	2.15E+09	0	1	9787	1.76E+09	2550	176604	685813	1.76E+09	0	0
producer	1970-01-20D21:07:01.339582000	2019-03-12D18:59:25.000000000	2712	761123	1.96E+09	1.00E+07	2.15E+09	0	1	10503	1.89E+09	2721	188073	730170	1.88E+09	0	0
producer	1970-01-20D21:07:02.340535000	2019-03-12D18:59:26.000000000	2876	805534	2.08E+09	1.00E+07	2.15E+09	0	1	11145	2.01E+09	2885	199080	772738	2.00E+09	0	0
**producer	1970-01-20D21:07:03.341604000	2019-03-12D18:59:27.000000000	2971	829228	2.15E+09	1.00E+07	2.15E+09	0	1	11503	2.07E+09	2980	205533	795920	2.06E+09	0	0
producer	1970-01-20D21:07:04.342482000	2019-03-12D18:59:28.000000000	2972	829228	2.15E+09	1.00E+07	2.15E+09	0	1	11504	2.07E+09	2981	205782	795920	2.06E+09	0	0
producer	1970-01-20D21:07:05.342596000	2019-03-12D18:59:29.000000000	2973	829228	2.15E+09	1.00E+07	2.15E+09	0	1	11505	2.07E+09	2982	206031	795920	2.06E+09	0	0
producer	1970-01-20D21:07:06.342781000	2019-03-12D18:59:30.000000000	2974	829228	2.15E+09	1.00E+07	2.15E+09	0	1	11506	2.07E+09	2983	206280	795920	2.06E+09	0	0
producer	1970-01-20D21:07:07.342898000	2019-03-12D18:59:31.000000000	2975	829228	2.15E+09	1.00E+07	2.15E+09	0	1	11507	2.07E+09	2984	206529	795920	2.06E+09	0	0
producer	1970-01-20D21:07:08.343046000	2019-03-12D18:59:32.000000000	2976	829228	2.15E+09	1.00E+07	2.15E+09	0	1	11508	2.07E+09	2985	206778	795920	2.06E+09	0	0**
producer	1970-01-20D21:07:09.400558000	2019-03-12D18:59:33.000000000	0	0	0	1.00E+07	2.15E+09	0	1	84	4044	84	20985	0	0	0	0
producer	1970-01-20D21:07:10.400651000	2019-03-12D18:59:34.000000000	1	0	0	1.00E+07	2.15E+09	0	1	85	4093	85	21234	0	0	0	0
producer	1970-01-20D21:07:09.343155000	2019-03-12D18:59:33.000000000	0	822815	2.13E+09	1.00E+07	2.15E+09	0	1	11509	2.07E+09	2986	207027	795920	2.06E+09	0	0
producer	1970-01-20D21:07:10.343292000	2019-03-12D18:59:34.000000000	1	326097	8.57E+08	1.00E+07	2.15E+09	0	1	11510	2.07E+09	2987	207276	795920	2.06E+09	0	0
producer	1970-01-20D21:07:11.343382000	2019-03-12D18:59:35.000000000	0	33308	8.63E+07	1.00E+07	2.15E+09	0	1	11511	2.07E+09	2988	207525	795920	2.06E+09	0	0
producer	1970-01-20D21:07:11.400745000	2019-03-12D18:59:35.000000000	0	0	0	1.00E+07	2.15E+09	0	1	86	4142	86	21483	0	0	0	0
producer	1970-01-20D21:07:12.343489000	2019-03-12D18:59:36.000000000	0	33308	8.63E+07	1.00E+07	2.15E+09	0	1	11512	2.07E+09	2989	207774	795920	2.06E+09	0	0
producer	1970-01-20D21:07:12.400841000	2019-03-12D18:59:36.000000000	0	0	0	1.00E+07	2.15E+09	0	1	87	4191	87	21732	0	0	0	0
producer	1970-01-20D21:07:13.343595000	2019-03-12D18:59:37.000000000	0	33308	8.63E+07	1.00E+07	2.15E+09	0	1	11513	2.07E+09	2990	208023	795920	2.06E+09	0	0
producer	1970-01-20D21:07:13.400940000	2019-03-12D18:59:37.000000000	0	0	0	1.00E+07	2.15E+09	0	1	88	4240	88	21981	0	0	0	0
producer	1970-01-20D21:07:14.343690000	2019-03-12D18:59:38.000000000	0	33308	8.63E+07	1.00E+07	2.15E+09	0	1	11514	2.07E+09	2991	208272	795920	2.06E+09	0	0
producer	1970-01-20D21:07:14.401069000	2019-03-12D18:59:38.000000000	0	0	0	1.00E+07	2.15E+09	0	1	89	4289	89	22230	0	0	0	0

[edenhill]

How many partitions does the topic(s) have that you are producing to?
And how many brokers are handling those partitions?

[pratikbatua]

I am using single broker and 5 partitions

[edenhill]

Check the CPU and IO load on the broker, as well as network utilization, maybe the broker and or network cant keep up

[pratikbatua]

Hi,
I moved broker to isolated box (4 cpus) , producer to another box (2 cpus) and consumer to separate box (2 cpus).

I am using 10G network on producer/consumer/broker. Is there any limitation with broker on how much messages they can handle?

The time when the error gets generated got delayed now. I need the capacity of 150K+ messages per second.

[srinathrangaramanujam]

Exactly the issue I faced. To be frank I was complaining the client library. Actually it was the broker. See for the network throughput, if you using kubernetes in cloud, check for the IOPS for the network storage. In my case, the culprit was n/w bandwidth. I was using 1Gbps. My bad. !!!