Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"SolrException: this IndexWriter is closed" preventing reindexing until Solr restarted #31

Closed
m-i-l opened this issue Jan 29, 2021 · 2 comments
Labels
bug Something isn't working

Comments

@m-i-l
Copy link
Contributor

m-i-l commented Jan 29, 2021

The indexing process sometimes gets a "this IndexWriter is closed" error from Solr when trying to submit a site to Solr. Indexing for that site stops, but indexing for other sites continues. However, given the error is Solr-side, none of the subsequent indexing processes successfully submit documents to Solr either. It requires a Solr restart to resolve.

Not entirely sure what causes the error because it isn't reproduceable, but it may be something like Solr running out of file handles.

Ideally the source of this error could be identified and remediated, or failing that some way of raising some kind of alert to make it known Solr needs to be restarted.

Not a top priority though given it has only happened twice so far.

Error on the indexing process:

2021-01-28 21:33:21 [pysolr] INFO: Finished 'http://search:8983/solr/content/update/' (post) with body '<delete><q' in 0.015 seconds, with status 500
2021-01-28 21:33:21 [pysolr] ERROR: Solr responded with an error (HTTP 500): [Reason: this IndexWriter is closed]
2021-01-28 21:33:21 [scrapy.core.engine] ERROR: Scraper close failure

Corresponding error in the Solr logs:

28/01/2021, 21:33:21
ERROR false
x:content
RequestHandlerBase
org.apache.solr.common.SolrException: this IndexWriter is closed
28/01/2021, 21:33:21
ERROR false
x:content
HttpSolrCall
null:org.apache.solr.common.SolrException: this IndexWriter is closed
28/01/2021, 21:33:26
ERROR false
x:content
UpdateLog
Error opening realtime searcher:org.apache.solr.common.SolrException: Error opening new searcher

Maybe related to https://issues.apache.org/jira/browse/SOLR-9830

@m-i-l m-i-l added the bug Something isn't working label Jan 29, 2021
@m-i-l
Copy link
Contributor Author

m-i-l commented Apr 7, 2021

Happened again on 4 Apr. I think it is a simple case of Solr running out of disk space. The Solr logs showed:

null:org.apache.solr.common.SolrException: this IndexWriter is closed
...
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
...
Caused by: java.io.IOException: No space left on device

And a df -k showed the 30Gb disk mounted at / was 100% full, with du tracing the largest consumer to be ~/data/solrdata (/var/solr within the Solr container) and most of that the actual data rather than logs or backups or anything which could be trimmed. Increasing disk to 40Gb has resolved for now, and I've set up an alert for when / gets more than 95% full.

@m-i-l
Copy link
Contributor Author

m-i-l commented Apr 16, 2021

Closing given the cause and solution have been found, and no further specific actions are required (e.g. no additional log rotation or pruning of backups).

@m-i-l m-i-l closed this as completed Apr 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant