If IndexWriter is interrupted on close and is using a channel (mmap/nio), it can throw a ClosedByInterruptException and prevent you from opening a new IndexWriter in the same proceses if you are using Native locks. [LUCENE-4638]

[asfimport]

The ClosedByInterruptException will prevent the index from being unlocked in close. If you try and close again, the call will hang. If you are using native locks and try to open a new IndexWriter, it will fail to get the lock. If you try IW#forceUnlock, it wont work because the not fully closed IW will still have the lock.

ideas:

• On ClosedByInterruptException, IW should continue trying to close what it can and unlock the index? Generally I have see the exception trigger in commitInternal.
• We should add a non static forceUnlock to IW that lets you remove the lock and start a new IW?
• We should make the lock protected so IW sub classes could unlock the index in advanced use cases?
• others?

---

Migrated from LUCENE-4638 by Mark Miller (@markrmiller), 2 votes, updated May 09 2016
Attachments: LUCENE-4638.patch
Linked issues:

• Fix IndexWriter.close() to not commit or wait for pending merges [LUCENE-4246] #5316

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Hmm why does the 2nd call to close hang? Do you have the original exc?

IW.rollback() should do a "better job" closing and releasing the lock, and in general on getting an exception from IW.close I think it's the only real recourse you have (ie, it's hard to know what docs you lost due to that exception).

Also, I think #5316 (IW.close should "just close", not wait for merges, commit, etc.) would improve this situation because then close would reliably release the lock.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

I should be able to find what it was hanging on, but a lot of logs to look back through. I can probably reproduce more easily instead when I get home tonight. If I remember right, it was trying to open an mmap input or something and if I remember right, it was just blocking. I'll reproduce and report the exact details.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

I have not had a chance to duplicate the hang yet - using fullmetaljenkins to work on some other bugs. I really could use a resolution to this though.

Currently, the advice for cleaning up after an IndexWriter in the javadoc is broken with native locks. You can't necessarily call close twice and you can't unlock using the static unlock method.

Here is a patch that provides a way for users to use the unlock in a finally pattern that is safe for native locks.

It adds a forceUnlock method to IndexWriter that is not static.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

Probably want to put a != null check around the writeLock.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

I don't think we should rush a fix here.

Let's see if rollback would have fixed it (and really javadocs should state that as the "recovery" if you hit exc during close), and let's understand what was hanging in the 2nd call to IW.close.

I think a new forceUnluck method in IndexWriter is too dangerous because the IndexWriter technically is still open so the app can continue to do ops after releasing the lock.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

I think a new forceUnluck method in IndexWriter is too dangerous

It's the same as the current static unlock method and javadocd the same.

I'm okay with it not being in Lucene though - I figure users would like to avoid this bug as well, but simply making the lock factory protected exposes it in an advanced enough way that it couldnt be considered dangerous. That would let me gid rid of this bug as well.

Let's see if rollback would have fixed it

I'll try that.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

simply making the lock factory protected exposes it in an advanced enough way that it couldnt be considered dangerous. That would let me gid rid of this bug as well.

I think that's a good solution for this issue?

It would still be nice to know if rollback resolves it (it's supposed to!), and why the 2nd IW.close() hangs (which is weird).

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

if thats what gets committed, please keep the issue open in that case.

This kind of behavior in close is outright buggy. its because its doing too much.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Also i would view it as a temporary solution, like until we have time to unfuck close() to not do so much.

I dont care that the issue is controversial. Its time to bring this to a head. I'm good at that.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

just a start so there is no wimpy solution committed permanently because close() does too much. I dont want 4.1 released with that solution.

[asfimport]

Commit Tag Bot (migrated from JIRA)

[trunk commit] Mark Robert Miller
http://svn.apache.org/viewvc?view=revision&revision=1425561

LUCENE-4638, SOLR-3180: try using the IW's writeLock to unlock

[asfimport]

Commit Tag Bot (migrated from JIRA)

[branch_4x commit] Mark Robert Miller
http://svn.apache.org/viewvc?view=revision&revision=1425563

LUCENE-4638, SOLR-3180: try using the IW's writeLock to unlock

[asfimport]

Commit Tag Bot (migrated from JIRA)

[trunk commit] Mark Robert Miller
http://svn.apache.org/viewvc?view=revision&revision=1425574

LUCENE-4638, SOLR-3180: revert for now (try using the IW's writeLock to unlock)

[asfimport]

Commit Tag Bot (migrated from JIRA)

[branch_4x commit] Mark Robert Miller
http://svn.apache.org/viewvc?view=revision&revision=1425576

LUCENE-4638, SOLR-3180: revert for now (try using the IW's writeLock to unlock)

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Bulk move 4.4 issues to 4.5 and 5.0