io.druid.java.util.common.IAE: Asked to add buffers

[dutao1]

io.druid.java.util.common.IAE: Asked to add buffers

Affected Version

0.12.3

Description

 ERROR [commerce_adx_stream_flow02-2019-11-27T14:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Failed to persist merged index[commerce_adx_stream_flow02]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class io.druid.java.util.common.IAE, exceptionMessage=Asked to add buffers[2,410,316,460] larger than configured max[2,147,483,647], interval=2019-11-27T14:00:00.000Z/2019-11-27T15:00:00.000Z}
io.druid.java.util.common.IAE: Asked to add buffers[2,410,316,460] larger than configured max[2,147,483,647]
	at io.druid.java.util.common.io.smoosh.FileSmoosher.addWithSmooshedWriter(FileSmoosher.java:170) ~[java-util-0.12.3.jar:0.12.3]
	at io.druid.segment.IndexMergerV9.makeColumn(IndexMergerV9.java:425) ~[druid-processing-0.12.3.jar:0.12.3]
	at io.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:224) ~[druid-processing-0.12.3.jar:0.12.3]
	at io.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:837) ~[druid-processing-0.12.3.jar:0.12.3]
	at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:710) ~[druid-processing-0.12.3.jar:0.12.3]
	at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:688) ~[druid-processing-0.12.3.jar:0.12.3]

is one segment so big？

[dutao1]

public SmooshedWriter addWithSmooshedWriter(final String name, final long size) throws IOException
  {

    if (size > maxChunkSize) {

how can i make size Smaller？

[jon-wei]

You'll want to reduce the amount of data handled by one task, some possibilities are:

• Run parallel tasks
• Shorten the task duration

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.