"SolrException: this IndexWriter is closed" preventing reindexing until Solr restarted

[m-i-l]

The indexing process sometimes gets a "this IndexWriter is closed" error from Solr when trying to submit a site to Solr. Indexing for that site stops, but indexing for other sites continues. However, given the error is Solr-side, none of the subsequent indexing processes successfully submit documents to Solr either. It requires a Solr restart to resolve.

Not entirely sure what causes the error because it isn't reproduceable, but it may be something like Solr running out of file handles.

Ideally the source of this error could be identified and remediated, or failing that some way of raising some kind of alert to make it known Solr needs to be restarted.

Not a top priority though given it has only happened twice so far.

Error on the indexing process:

2021-01-28 21:33:21 [pysolr] INFO: Finished 'http://search:8983/solr/content/update/' (post) with body '<delete><q' in 0.015 seconds, with status 500
2021-01-28 21:33:21 [pysolr] ERROR: Solr responded with an error (HTTP 500): [Reason: this IndexWriter is closed]
2021-01-28 21:33:21 [scrapy.core.engine] ERROR: Scraper close failure

Corresponding error in the Solr logs:

28/01/2021, 21:33:21
ERROR false
x:content
RequestHandlerBase
org.apache.solr.common.SolrException: this IndexWriter is closed
28/01/2021, 21:33:21
ERROR false
x:content
HttpSolrCall
null:org.apache.solr.common.SolrException: this IndexWriter is closed
28/01/2021, 21:33:26
ERROR false
x:content
UpdateLog
Error opening realtime searcher:org.apache.solr.common.SolrException: Error opening new searcher

Maybe related to https://issues.apache.org/jira/browse/SOLR-9830

[m-i-l]

Happened again on 4 Apr. I think it is a simple case of Solr running out of disk space. The Solr logs showed:

null:org.apache.solr.common.SolrException: this IndexWriter is closed
...
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
...
Caused by: java.io.IOException: No space left on device

And a df -k showed the 30Gb disk mounted at / was 100% full, with du tracing the largest consumer to be ~/data/solrdata (/var/solr within the Solr container) and most of that the actual data rather than logs or backups or anything which could be trimmed. Increasing disk to 40Gb has resolved for now, and I've set up an alert for when / gets more than 95% full.

[m-i-l]

Closing given the cause and solution have been found, and no further specific actions are required (e.g. no additional log rotation or pruning of backups).