Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

0.90.6 memory issue #4093

Closed
brackxm opened this issue Nov 5, 2013 · 10 comments · Fixed by #4104
Closed

0.90.6 memory issue #4093

brackxm opened this issue Nov 5, 2013 · 10 comments · Fixed by #4104
Assignees
@s1monw
Labels
>bug v0.90.7 v1.0.0.Beta1

Comments

@brackxm
Copy link
Contributor

brackxm commented Nov 5, 2013

0.90.6 needs a lot more direct memory compared to 0.90.5 when using an in memory store for tests.

Setting needed for 0.90.6

-XX:MaxDirectMemorySize=4608m

Setting needed for 0.90.5

-XX:MaxDirectMemorySize=512m
@clintongormley
Copy link

Hiya

Yes, this is a known issue, see #4078

We will be releasing version 0.90.7 soon, which will fix this. In the meantime, you can add this to your config:

index.warmer.enabled: false

@clintongormley
Copy link

(at least I assume this is the same thing)

Could you try the setting above and let me know if it fixed the issue?

thanks

@clintongormley clintongormley reopened this Nov 5, 2013
@brackxm
Copy link
Contributor Author

brackxm commented Nov 5, 2013

No, that setting does not fix the issue.

@clintongormley
Copy link

Could you give us some idea of what your tests do and how you configure elasticsearch?

@brackxm
Copy link
Contributor Author

brackxm commented Nov 5, 2013

stacktrace

java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:658)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
    at org.apache.lucene.store.bytebuffer.PlainByteBufferAllocator.allocate(PlainByteBufferAllocator.java:55)
    at org.apache.lucene.store.bytebuffer.CachingByteBufferAllocator.allocate(CachingByteBufferAllocator.java:52)
    at org.elasticsearch.cache.memory.ByteBufferCache.allocate(ByteBufferCache.java:101)
    at org.apache.lucene.store.bytebuffer.ByteBufferIndexOutput.switchCurrentBuffer(ByteBufferIndexOutput.java:106)
    at org.apache.lucene.store.bytebuffer.ByteBufferIndexOutput.writeBytes(ByteBufferIndexOutput.java:93)
    at org.elasticsearch.common.lucene.store.BufferedChecksumIndexOutput.flushBuffer(BufferedChecksumIndexOutput.java:69)
    at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)
    at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:102)
    at org.elasticsearch.common.lucene.store.BufferedChecksumIndexOutput.flush(BufferedChecksumIndexOutput.java:80)
    at org.apache.lucene.store.BufferedIndexOutput.close(BufferedIndexOutput.java:126)
    at org.elasticsearch.common.lucene.store.BufferedChecksumIndexOutput.close(BufferedChecksumIndexOutput.java:60)
    at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:587)
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
    at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:582)
    at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81)
    at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1082)
    at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:408)
    at org.elasticsearch.index.codec.postingsformat.ElasticSearch090PostingsFormat$1.close(ElasticSearch090PostingsFormat.java:63)
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86)
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:163)
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154)
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
    at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:102)
    at org.apache.lucene.index.TermsHash.flush(TermsHash.java:116)
    at org.apache.lucene.index.DocInverter.flush(DocInverter.java:53)
    at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81)
    at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:466)
    at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:499)
    at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:609)
    at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:367)
    at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:277)
    at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:252)
    at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:242)
    at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
    at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:118)
    at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
    at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:155)
    at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:204)
    at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:786)
    at org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:448)
    at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:228)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)

@brackxm
Copy link
Contributor Author

brackxm commented Nov 5, 2013

configuration

        final Node node = NodeBuilder.nodeBuilder()
                .local(true)
                .clusterName(clusterName)
                .settings(ImmutableSettings.settingsBuilder()
                        .put("index.store.type", "memory")
                        .put("index.number_of_shards", "1")
                        .put("index.number_of_replicas", "0")
                        .put("gateway.type", "none")
                        .put("http.enabled", false)
                        .put("index.warmer.enabled", false)
                        .put("path.data", "target/es")
                        .put("path.logs", "target/es")
                        .put("path.work", "target/es")
                        .build())
                .build();

@kimchy
Copy link
Member

kimchy commented Nov 5, 2013

can you maybe write isolate it to a test case that shows that it passes in 0.90.5, while it fails in 0.90.6 (with the index.warmer.enabled set to false)? it would be a great help in chasing it down...

@brackxm
Copy link
Contributor Author

brackxm commented Nov 5, 2013 

import org.elasticsearch.client.Client;
import org.elasticsearch.common.settings.ImmutableSettings;
import org.elasticsearch.node.Node;
import org.elasticsearch.node.NodeBuilder;

import java.util.UUID;

public class Issue {
    public static void main(final String[] args) {
        final Node node = NodeBuilder.nodeBuilder()
                .local(true)
                .clusterName("test")
                .settings(ImmutableSettings.settingsBuilder()
                        .put("index.store.type", "memory")
                        .put("index.number_of_shards", "1")
                        .put("index.number_of_replicas", "0")
                        .put("gateway.type", "none")
                        .put("http.enabled", false)
                        .put("index.warmer.enabled", false)
                        .build())
                .build();
        node.start();
        final Client client = node.client();
        final byte[] source = "{\"a\":\"a1\"}".getBytes();
        for (int i = 0; i < 1000; i++) {
            final String id = UUID.randomUUID().toString();
            client.prepareIndex("index1", "type1", id)
                    .setSource(source)
                    .setRefresh(true)
                    .execute()
                    .actionGet();
        }
    }
}

@kimchy
Copy link
Member

kimchy commented Nov 5, 2013

@s1monw and myself chased it up, seems like a regression in Lucene, opened an issue: https://issues.apache.org/jira/browse/LUCENE-5330 and already have a patch for it: https://issues.apache.org/jira/secure/attachment/12612263/LUCENE-5330.

@brackxm
Copy link
Contributor Author

brackxm commented Nov 5, 2013

thanks for looking into this

@s1monw s1monw mentioned this issue Nov 6, 2013
Merged
@s1monw s1monw closed this as completed in bb777a2 Nov 6, 2013
s1monw added a commit that referenced this issue Nov 6, 2013
@s1monw
Apply fix for LUCENE-5330 pruning the IndexWriter queue to get rid of…
14d529e 
… pending event

Closes #4093
@ghost ghost assigned s1monw Nov 6, 2013
s1monw added a commit that referenced this issue Nov 8, 2013
@s1monw
Reduce number of docs in test for #4093 - 30 docs are enough to trigg…
9c7fb86 
…er the bug
s1monw added a commit that referenced this issue Nov 8, 2013
@s1monw
Reduce number of docs in test for #4093 - 30 docs are enough to trigg…
76622f1 
…er the bug
@stuartf stuartf mentioned this issue Nov 12, 2013
Merged
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
@s1monw
Apply fix for LUCENE-5330 pruning the IndexWriter queue to get rid of…
26b85b9 
… pending event

Closes elastic#4093
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
@s1monw
Reduce number of docs in test for elastic#4093 - 30 docs are enough t…
c6233ac 
…o trigger the bug
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@s1monw s1monw

Labels
>bug v0.90.7 v1.0.0.Beta1
Projects
None yet
Milestone
No milestone
4 participants
@kimchy @clintongormley @s1monw @brackxm