Lucene wrapped in SailRepository with multithreading gives error #192

jgrzebyta · 2016-06-23T22:47:00Z

Affected: v 2.0-SNAPSHOT,
Java: openjdk-8
os: Ubuntu 16.04

I use LuceneSailRepository wrapped in SailsRepository. MemoryStore with dump file is used as a storage.

I load data in multithreads managed by Executors.newFixedThreadPoolpool. When I do .invokeAll
I got error about closed call when new request is done. It seems Lucene index in not thread safety.

The full stacktrace is:

23:29:47,677 [pool-3-thread-2] ERROR LuceneSailConnection: Rolling back org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/tmp/rdf4j-loader-s5A6SCKMe.34724407296800170144873/lucenedir/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:780)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.getIndexWriter(LuceneIndex.java:291)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.invalidateReaders(LuceneIndex.java:581)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.commit(LuceneIndex.java:651)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.addRemoveStatements(LuceneSailConnection.java:252)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.commit(LuceneSailConnection.java:217)
    at org.eclipse.rdf4j.repository.sail.SailRepositoryConnection.commit(SailRepositoryConnection.java:169)
    at triple.loader$eval939$fn__940$fn__941.invoke(loader.clj:122)
    at triple.loader$eval939$fn__940.invoke(loader.clj:98)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at triple.loader$eval935$fn__936.invoke(loader.clj:88)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at sparql$load_multidata$fn__1094$fn__1095.invoke(sparql.clj:102)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

23:29:48,694 [pool-3-thread-2] ERROR LuceneSailConnection: Committing operations in lucenesail, encountered exception org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/tmp/rdf4j-loader-s5A6SCKMe.34724407296800170144873/lucenedir/write.lock. Only some operations were stored, 1 operations are discarded. Lucene Index is now corrupt. org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/tmp/rdf4j-loader-s5A6SCKMe.34724407296800170144873/lucenedir/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:780)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.getIndexWriter(LuceneIndex.java:291)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.rollback(LuceneIndex.java:658)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.addRemoveStatements(LuceneSailConnection.java:256)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.commit(LuceneSailConnection.java:217)
    at org.eclipse.rdf4j.repository.sail.SailRepositoryConnection.commit(SailRepositoryConnection.java:169)
    at triple.loader$eval939$fn__940$fn__941.invoke(loader.clj:122)
    at triple.loader$eval939$fn__940.invoke(loader.clj:98)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at triple.loader$eval935$fn__936.invoke(loader.clj:88)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at sparql$load_multidata$fn__1094$fn__1095.invoke(sparql.clj:102)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

23:29:48,709 [clojure-agent-send-off-pool-0] ERROR sparql: There is error:  java.util.concurrent.ExecutionException
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: cannot close: prepareCommit was already called with no corresponding call to commit
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
    at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313)
    at sparql$load_multidata.invokeStatic(sparql.clj:110)
    at sparql$load_multidata.invoke(sparql.clj:90)
    at triple.multiload$fn__1114.invokeStatic(multiload.clj:15)
    at triple.multiload$fn__1114.invoke(multiload.clj:13)
    at clojure.test$test_var$fn__7983.invoke(test.clj:716)
    at clojure.test$test_var.invokeStatic(test.clj:716)
    at clojure.test$test_var.invoke(test.clj:707)
    at boot.user$eval133$fn__134.doInvoke(boot.user3412006909439801406.clj:23)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:646)
    at clojure.core$apply.invoke(core.clj:641)
    at boot.core$construct_tasks.doInvoke(core.clj:911)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:646)
    at clojure.core$apply.invoke(core.clj:641)
    at boot.core$boot$fn__933.invoke(core.clj:949)
    at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: cannot close: prepareCommit was already called with no corresponding call to commit
    at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:960)
    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1015)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.shutDown(LuceneIndex.java:322)
    at org.eclipse.rdf4j.sail.lucene.LuceneSail.shutDown(LuceneSail.java:291)
    at org.eclipse.rdf4j.repository.sail.SailRepository.shutDownInternal(SailRepository.java:169)
    at org.eclipse.rdf4j.repository.base.AbstractRepository.shutDown(AbstractRepository.java:45)
    at triple.loader$eval939$fn__940.invoke(loader.clj:98)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at triple.loader$eval935$fn__936.invoke(loader.clj:88)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at sparql$load_multidata$fn__1094$fn__1095.invoke(sparql.clj:102)
    ... 5 more

The text was updated successfully, but these errors were encountered:

pulquero · 2016-06-24T20:45:20Z

Just to confirm you are using a connection per thread?

jgrzebyta · 2016-06-28T00:27:31Z

On 24 Jun 2016 9:45 p.m., "pulquero" notifications@github.com wrote:

Just to confirm you are using a connection per thread?

Yes. Also I am opening an isolated transaction (read only or this kind) but
it seems it does not affect. Maybe the issue is I use memory storage but
the error is within the Lucent part.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

pulquero · 2016-06-29T21:12:31Z

Question for someone in the know but why is every class in rdf4j-sail-lucene marked as deprecated?

abrokenjester · 2016-06-29T21:26:24Z

It looks as if these classes were for an older Lucene version that we got rid of. The idea when transitioning to RDF4J was to remove support for Lucene 3 and 4, and only keep supporting the latest Lucene 5 (alongside solr and elasticsearch of course).

However due to the fact that the transition took so long and I had to do several large sync operations to keep the new github repo up to date with developments on the bitbucket side, something may have gone wonky. @pulquero could you doublecheck if we actually have the correct version of the lucene sail code included in the GitHub repo? If not, we should probably re-insert that part of the code from Bitbucket.

pulquero · 2016-06-30T16:38:49Z

It looks to be the correct one, which is this one https://bitbucket.org/openrdf/sesame/src/69dcca77110d/core/sail/fts/lucene/?at=master but the classes are deprecated here.

abrokenjester · 2016-06-30T19:07:41Z

They might simply be the result of a bad merge. If it's the correct code we
can just remove the deprecated notices again.

sent from my phone
On 1/07/2016 4:38 am, "pulquero" notifications@github.com wrote:

It looks to be the correct one, which is this one
https://bitbucket.org/openrdf/sesame/src/69dcca77110d/core/sail/fts/lucene/?at=master
but the classes are deprecated here.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#192 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAseyPE_T93TCiNuGJXkaTS5KgvdmRcmks5qQ_EZgaJpZM4I9UAt
.

pulquero · 2016-06-30T19:57:46Z

@jgrzebyta PR with fix now available. Please take a look and confirm if you believe the newly added test covers your use case.

jgrzebyta · 2016-07-04T15:19:13Z

@pulquero IMO The test code covers my case. Unfortunately I do not have the original code so I cannot check the solution - I went into single thread solution.

Ps. I am not sure if your pull request was added to the main project if so please could you close this issue.

Issues/#192 lucene thread fix

Issues/1337 faster spin

abrokenjester added the 🐞 bug issue is a bug label Jun 27, 2016

pulquero mentioned this issue Jun 30, 2016

Issues/#192 lucene thread fix #212

Merged

3 tasks

abrokenjester added a commit that referenced this issue Jul 5, 2016

Merge pull request #212 from pulquero/issues/#192-lucene-thread-fix

99b9dbd

Issues/#192 lucene thread fix

abrokenjester closed this as completed Jul 5, 2016

pulquero mentioned this issue Aug 30, 2016

LuceneSailConnection is not threadsafe #301

Closed

abrokenjester mentioned this issue Aug 30, 2016

Issue #293: Attempt to avoid nested NPE in BackgroundTupleResult #294

Merged

3 tasks

abrokenjester pushed a commit that referenced this issue Aug 22, 2019

Merge pull request #192 from eclipse/issues/1337_faster_spin

3c2d854

Issues/1337 faster spin

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lucene wrapped in SailRepository with multithreading gives error #192

Lucene wrapped in SailRepository with multithreading gives error #192

jgrzebyta commented Jun 23, 2016

pulquero commented Jun 24, 2016

jgrzebyta commented Jun 28, 2016

pulquero commented Jun 29, 2016

abrokenjester commented Jun 29, 2016

pulquero commented Jun 30, 2016

abrokenjester commented Jun 30, 2016

pulquero commented Jun 30, 2016

jgrzebyta commented Jul 4, 2016 •

edited

Lucene wrapped in SailRepository with multithreading gives error #192

Lucene wrapped in SailRepository with multithreading gives error #192

Comments

jgrzebyta commented Jun 23, 2016

pulquero commented Jun 24, 2016

jgrzebyta commented Jun 28, 2016

pulquero commented Jun 29, 2016

abrokenjester commented Jun 29, 2016

pulquero commented Jun 30, 2016

abrokenjester commented Jun 30, 2016

pulquero commented Jun 30, 2016

jgrzebyta commented Jul 4, 2016 • edited

jgrzebyta commented Jul 4, 2016 •

edited