Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Possible corruption: Memory capacity too small: 47136 < 12845088" exception occur in 0.14.x but not in 0.13.x #8032

Closed
quenlang opened this issue Jul 5, 2019 · 5 comments · Fixed by #8055
Closed

"Possible corruption: Memory capacity too small: 47136 < 12845088" exception occur in 0.14.x but not in 0.13.x #8032

quenlang opened this issue Jul 5, 2019 · 5 comments · Fixed by #8055

Comments

@quenlang
Copy link

quenlang commented Jul 5, 2019

Hello
An exception "Possible corruption: Memory capacity too small: 47136 < 12845088" occurs in the historical log when I performed a query with the Quantiles Sketch in druid 0.14.2 like this

...
    "aggregations": [{
        "type": "quantilesDoublesSketch",
        "name": "resp_time_his",
        "fieldName": "resp_time_his",
        "k": 256
    }],
    "postAggregations": [{
        "type": "quantilesDoublesSketchToQuantiles",
        "name": "respTimeHis",
        "field": {
            "type": "fieldAccess",
            "name": null,
            "fieldName": "resp_time_his"
        },
        "fractions": [0.25, 0.50, 0.75, 0.99]
    }],
...

Then I changed k=256 to k=128, it worked.
Also, I downgraded to druid 0.13.0 and set k=256, it worked.

The historical jvm configurations:

-server
-Xms1g
-Xmx3g
-XX:MaxDirectMemorySize=4g
-XX:NewRatio=4
-XX:+UseConcMarkSweepGC
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

The historical runtime configurations:

druid.service=druid/historical
druid.port=8083
druid.server.http.numThreads=25
druid.processing.buffer.sizeBytes=200000000
druid.processing.numThreads=4
druid.processing.numMergeBuffers=1
druid.segmentCache.locations=[{"path":"var/druid/segment-cache","maxSize":130000000000}]
druid.server.maxSize=130000000000

And the dataset which I queryed is very small, I have no idea of this exception.
@himanshug Can you give me some advices? Thank you !
Have a nice day!

The detail error messag:

2019-07-05T05:58:57,982 ERROR [qtp700171153-70[groupBy_[SVR_ACTION_DATA_MIN]_d149d0bf-6bb7-4569-bbbc-43ebb5567022]] org.apache.druid.server.QueryResource - Exception handling request: {class=org.apache.druid.server.QueryResource, exceptionType=class com.yahoo.sketches.SketchesArgumentException, exceptionMessage=Possible corruption: Memory capacity too small: 47136 < 12845088, exception=com.yahoo.sketches.SketchesArgumentException: Possible corruption: Memory capacity too small: 47136 < 12845088, query=GroupByQuery{dataSource='SVR_ACTION_DATA_MIN', querySegmentSpec=MultipleSpecificSegmentSpec{descriptors=[SegmentDescriptor{interval=2019-07-04T08:05:00.000Z/2019-07-04T08:35:00.000Z, version='2019-07-04T08:01:31.683Z', partitionNumber=0}, SegmentDescriptor{interval=2019-07-04T08:05:00.000Z/2019-07-04T08:35:00.000Z, version='2019-07-04T08:01:31.683Z', partitionNumber=1}]}, virtualColumns=[], limitSpec=DefaultLimitSpec{columns='[]', limit=2147483647}, dimFilter=(biz_system_id = 1150 && data_type = APP), granularity=AllGranularity, dimensions=[DefaultDimensionSpec{dimension='biz_system_id', outputName='bizSystemId', outputType='STRING'}], aggregatorSpecs=[DoublesSketchAggregatorFactory{name=resp_time_his, fieldName=resp_time_his, k=256}], postAggregatorSpecs=[DoublesSketchToQuantilesPostAggregator{name='respTimeHis', field=FieldAccessPostAggregator{name='null', fieldName='resp_time_his'}, fractions=[0.2199999988079071, 0.550000011920929, 0.9800000190734863, 0.9900000095367432]}], havingSpec=null}, peer=10.128.1.108}
com.yahoo.sketches.SketchesArgumentException: Possible corruption: Memory capacity too small: 47136 < 12845088
	at com.yahoo.sketches.quantiles.DirectUpdateDoublesSketchR.checkDirectMemCapacity(DirectUpdateDoublesSketchR.java:197) ~[?:?]
	at com.yahoo.sketches.quantiles.DirectUpdateDoublesSketch.wrapInstance(DirectUpdateDoublesSketch.java:108) ~[?:?]
	at com.yahoo.sketches.quantiles.DoublesUnionImpl.updateLogic(DoublesUnionImpl.java:194) ~[?:?]
	at com.yahoo.sketches.quantiles.DoublesUnionImpl.update(DoublesUnionImpl.java:104) ~[?:?]
	at org.apache.druid.query.aggregation.datasketches.quantiles.DoublesSketchMergeBufferAggregator.aggregate(DoublesSketchMergeBufferAggregator.java:70) ~[?:?]
	at org.apache.druid.query.groupby.epinephelinae.AbstractBufferHashGrouper.aggregate(AbstractBufferHashGrouper.java:165) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.aggregate(SpillingGrouper.java:167) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.aggregate(ConcurrentGrouper.java:267) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.Grouper.aggregate(Grouper.java:82) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$1.accumulate(RowBasedGrouperHelper.java:270) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$1.accumulate(RowBasedGrouperHelper.java:247) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.ConcatSequence.lambda$accumulate$0(ConcatSequence.java:43) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:43) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:82) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:168) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:45) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:148) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:244) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:232) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_60]
	at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:247) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_60]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
@gianm
Copy link
Contributor

gianm commented Jul 5, 2019

@AlexanderSaydakov / @leerho does this look like a potential issue to you? druid-0.14.2 is using sketches-core 0.13.3, and druid-0.13.0 is using sketches-core 0.10.3.

@AlexanderSaydakov
Copy link
Contributor

What is the type of the input field "fieldName": "resp_time_his"?

@quenlang
Copy link
Author

quenlang commented Jul 6, 2019

@gianm @AlexanderSaydakov Thanks for quick reply.
The type of resp_time_his is sketche object. We build outside of druid with sketchs-core 0.10.3.

@AlexanderSaydakov
Copy link
Contributor

What k do the input sketches have?
I could not reproduce this so far. I would appreciate some simple reproducible scenario.

@quenlang
Copy link
Author

k=256.

@AlexanderSaydakov AlexanderSaydakov mentioned this issue Jul 10, 2019
Merged
@AlexanderSaydakov AlexanderSaydakov mentioned this issue Jul 31, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

force native order when wrapping ByteBuffer AlexanderSaydakov/druid
3 participants
@gianm @quenlang @AlexanderSaydakov