Permalink
Browse files

Commit Solr index before simulating or starting recrawl job.

This ensures up-to-date simulation query results, and recrawl
processing.
  • Loading branch information...
luccioman committed Jan 26, 2018
1 parent 5b943c0 commit 8b572b7337c79dfc76bf7c5f308ffa07f2e77125
Showing with 10 additions and 2 deletions.
  1. +3 −1 htroot/IndexReIndexMonitor_p.java
  2. +7 −1 source/net/yacy/crawler/RecrawlBusyThread.java
@@ -164,8 +164,10 @@ public static serverObjects respond(final RequestHeader header, final serverObje
}
} else if(post.containsKey("simulateRecrawl")) {
final SolrConnector solrConnector = sb.index.fulltext().getDefaultConnector();
if (!solrConnector.isClosed()) {
if (solrConnector != null && !solrConnector.isClosed()) {
try {
/* Ensure indexed data is up-to-date */
solrConnector.commit(true);
// query all or only httpstatus=200 depending on includefailed flag
final String finalQuery = RecrawlBusyThread.buildSelectionQuery(recrawlQuery, inclerrdoc);
final long count = solrConnector.getCountByQuery(finalQuery);
@@ -129,6 +129,12 @@ public RecrawlBusyThread(final Switchboard xsb, final String query, final boolea
// org.apache.solr.core.SolrCore java.lang.IllegalStateException: unexpected docvalues type NONE for field 'load_date_dt' (expected=NUMERIC). Use UninvertingReader or index with docvalues.
this.solrSortBy = null; // CollectionSchema.load_date_dt.getSolrFieldName() + " asc";
this.chunksize = sb.getConfigInt(SwitchboardConstants.CRAWLER_THREADS_ACTIVE_MAX, 200);
final SolrConnector solrConnector = this.sb.index.fulltext().getDefaultConnector();
if (solrConnector != null && !solrConnector.isClosed()) {
/* Ensure indexed data is up-to-date before running the main job */
solrConnector.commit(true);
}
}
/**
@@ -265,7 +271,7 @@ private boolean processSingleQuery() {
}
SolrDocumentList docList = null;
final SolrConnector solrConnector = sb.index.fulltext().getDefaultConnector();
if (solrConnector.isClosed()) {
if (solrConnector == null || solrConnector.isClosed()) {
this.urlsToRecrawl = 0;
this.terminatedBySolrFailure = true;
return false;

1 comment on commit 8b572b7

@Scarfmonster

This comment has been minimized.

Contributor

Scarfmonster commented on 8b572b7 Jan 28, 2018

Sorl docs warn that this is not a good idea, especially if someone is running SorlCloud. There was an issue with the config which prevented Sorl from updating the index in near-real time. I created a pull request for it here #157

Please sign in to comment.