-
Hi, first of all thanks for that great project! I'm building my own index from scratch and unfortunately I'm having some issues when starting Photon with this index. It seems that according to ElasticSearch it is in "recovering" state. However, it feels like it cannot complete recovering due to Photon crashing 10s after "ES cluster is now ready" appears, so this is basically an endless loop which I am not able to fix... Is there any recommendation about how to let ES finish recovering or - even better - how to avoid the index being in the recovering state right on startup? Below is my full stack trace. Thanks in advance for any help!
|
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 7 replies
-
photon/src/main/java/de/komoot/photon/elasticsearch/Server.java Lines 125 to 127 in 716b45f According to the docs:
|
Beta Was this translation helpful? Give feedback.
-
This looks like indexing was not finished for some reason before the database was shut down after the import was finished. Photon itself has no code for index recovery. You could try to hack something in. According to this stackoverflow any of the following should do the trick:
Best place is probably just hacking it in after If that works for you, let us know and we can think about adding a recovery functionality to photon. |
Beta Was this translation helpful? Give feedback.
-
@lonvia A follow up question: Is it okay to assume that ES cluster might never reach green status ?? I added some code in server.java file to track if it ever goes green and it never goes to green status.
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.cluster.health.ClusterHealthStatus;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
public class Server{
.....
.....
.....
// public void waitForReady() {
// esClient.admin().cluster().prepareHealth().setWaitForYellowStatus().get();
// }
public void waitForReady() {
try
{
esClient.admin().cluster().prepareHealth().setWaitForYellowStatus().get();
esClient.admin().indices().prepareFlush(PhotonIndex.NAME).get();
LOGGER.info("Index flushed successfully.");
esClient.admin().indices().prepareRefresh("photon").get();
LOGGER.info("Index refreshed successfully.");
} catch (Exception e) {
LOGGER.error("Error flushing or refreshing the index.", e);
}
try
{
// flushAndRefreshIndex();
final int checkIntervalSeconds = 6;
final int maxAttempts = 10;
int attempts = 0;
ClusterHealthResponse response = esClient.admin().cluster().prepareHealth().get();
ClusterHealthStatus currentStatus = response.getStatus();
int initialUnassignedShards = response.getUnassignedShards();
int unassignedShards = initialUnassignedShards;
while (!currentStatus.equals(ClusterHealthStatus.GREEN) && attempts < maxAttempts) {
attempts++;
int percentageLeft = (int) (((double) unassignedShards / initialUnassignedShards) * 100);
LOGGER.info("Waiting for Elasticsearch cluster to become green. Current status: {}. Attempt {}/{}. Unassigned shards: {}. Progress: {}%",
currentStatus.name(), attempts, maxAttempts, unassignedShards, 100 - percentageLeft);
Thread.sleep(checkIntervalSeconds * 1000L);
response = esClient.admin().cluster().prepareHealth().get();
currentStatus = response.getStatus();
unassignedShards = response.getUnassignedShards();
}
if (currentStatus.equals(ClusterHealthStatus.GREEN)) {
LOGGER.info("Elasticsearch cluster is ready and in a green state.");
} else {
LOGGER.warn("Elasticsearch cluster did not reach a green state after {} attempts. Final status: {}. Remaining unassigned shards: {}",
maxAttempts, currentStatus.name(), unassignedShards);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
LOGGER.error("Interrupted while waiting for Elasticsearch cluster to become green.", e);
} catch (Exception e) {
throw new RuntimeException("Exception while waiting for Elasticsearch cluster to be ready", e);
}
}
public void shutdown() {
.....
.....
.....
2024-03-31 00:23:16,303 [main] INFO de.komoot.photon.elasticsearch.Server - Started elastic search node
2024-03-31 00:23:17,668 [main] INFO de.komoot.photon.elasticsearch.Server - Started elastic search client connected to 127.0.0.1
2024-03-31 00:23:17,668 [main] INFO de.komoot.photon.App - Make sure that the ES cluster is ready, this might take some time.
2024-03-31 00:23:17,880 [main] INFO de.komoot.photon.elasticsearch.Server - Index flushed successfully.
2024-03-31 00:23:17,903 [main] INFO de.komoot.photon.elasticsearch.Server - Index refreshed successfully.
2024-03-31 00:23:17,924 [main] INFO de.komoot.photon.elasticsearch.Server - Waiting for Elasticsearch cluster to become green. Current status: YELLOW. Attempt 1/10. Unassigned shards: 1. Progress: 0%
2024-03-31 00:23:23,932 [main] INFO de.komoot.photon.elasticsearch.Server - Waiting for Elasticsearch cluster to become green. Current status: YELLOW. Attempt 2/10. Unassigned shards: 1. Progress: 0%
2024-03-31 00:23:29,938 [main] INFO de.komoot.photon.elasticsearch.Server - Waiting for Elasticsearch cluster to become green. Current status: YELLOW. Attempt 3/10. Unassigned shards: 1. Progress: 0%
2024-03-31 00:23:35,944 [main] INFO de.komoot.photon.elasticsearch.Server - Waiting for Elasticsearch cluster to become green. Current status: YELLOW. Attempt 4/10. Unassigned shards: 1. Progress: 0%
2024-03-31 00:23:41,952 [main] INFO de.komoot.photon.elasticsearch.Server - Waiting for Elasticsearch cluster to become green. Current status: YELLOW. Attempt 5/10. Unassigned shards: 1. Progress: 0%
2024-03-31 00:23:46,325 [elasticsearch[iN0keqs][management][T#2]] WARN org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor - high disk watermark [90%] exceeded on [iN0keqsVT5qAzqw6u3L_LQ][iN0keqs][/var/folders/d2/8zsvd80x06j__s7ryt_ggr5h0000gn/T/junit6757350629924376513/photon_data/elasticsearch/data/nodes/0] free: 11.5gb[4.9%], shards will be relocated away from this node
2024-03-31 00:23:47,959 [main] INFO de.komoot.photon.elasticsearch.Server - Waiting for Elasticsearch cluster to become green. Current status: YELLOW. Attempt 6/10. Unassigned shards: 1. Progress: 0%
2024-03-31 00:23:53,964 [main] INFO de.komoot.photon.elasticsearch.Server - Waiting for Elasticsearch cluster to become green. Current status: YELLOW. Attempt 7/10. Unassigned shards: 1. Progress: 0%
2024-03-31 00:23:59,967 [main] INFO de.komoot.photon.elasticsearch.Server - Waiting for Elasticsearch cluster to become green. Current status: YELLOW. Attempt 8/10. Unassigned shards: 1. Progress: 0%
2024-03-31 00:24:05,969 [main] INFO de.komoot.photon.elasticsearch.Server - Waiting for Elasticsearch cluster to become green. Current status: YELLOW. Attempt 9/10. Unassigned shards: 1. Progress: 0%
2024-03-31 00:24:11,974 [main] INFO de.komoot.photon.elasticsearch.Server - Waiting for Elasticsearch cluster to become green. Current status: YELLOW. Attempt 10/10. Unassigned shards: 1. Progress: 0%
2024-03-31 00:24:16,332 [elasticsearch[iN0keqs][management][T#2]] WARN org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor - high disk watermark [90%] exceeded on [iN0keqsVT5qAzqw6u3L_LQ][iN0keqs][/var/folders/d2/8zsvd80x06j__s7ryt_ggr5h0000gn/T/junit6757350629924376513/photon_data/elasticsearch/data/nodes/0] free: 11.5gb[4.9%], shards will be relocated away from this node
2024-03-31 00:24:17,978 [main] WARN de.komoot.photon.elasticsearch.Server - Elasticsearch cluster did not reach a green state after 10 attempts. Final status: YELLOW. Remaining unassigned shards: 1
2024-03-31 00:24:17,978 [main] INFO de.komoot.photon.App - ES cluster is now ready.
2024-03-31 00:52:36,919 [main] WARN org.elasticsearch.node.Node - version [5.6.16-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
2024-03-31 00:52:41,656 [main] INFO de.komoot.photon.elasticsearch.Server - Started elastic search node
2024-03-31 00:52:41,656 [main] INFO de.komoot.photon.App - Make sure that the ES cluster is ready, this might take some time.
2024-03-31 00:52:41,706 [main] ERROR de.komoot.photon.elasticsearch.Server - Error flushing or refreshing the index.
org.elasticsearch.index.IndexNotFoundException: no such index
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.infe(IndexNameExpressionResolver.java:678) ~[photon-0.5.0.jar:?]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:632) ~[photon-0.5.0.jar:?]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:580) ~[photon-0.5.0.jar:?]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:169) ~[photon-0.5.0.jar:?]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:145) ~[photon-0.5.0.jar:?]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:78) ~[photon-0.5.0.jar:?]
at org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.shards(TransportBroadcastReplicationAction.java:130) ~[photon-0.5.0.jar:?]
at org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:78) ~[photon-0.5.0.jar:?]
at org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:55) ~[photon-0.5.0.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[photon-0.5.0.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[photon-0.5.0.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[photon-0.5.0.jar:?]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[photon-0.5.0.jar:?]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[photon-0.5.0.jar:?]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[photon-0.5.0.jar:?]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) ~[photon-0.5.0.jar:?]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80) ~[photon-0.5.0.jar:?]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54) ~[photon-0.5.0.jar:?]
at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:62) ~[photon-0.5.0.jar:?]
at de.komoot.photon.elasticsearch.Server.waitForReady(Server.java:149) [photon-0.5.0.jar:?]
at de.komoot.photon.App.main(App.java:44) [photon-0.5.0.jar:?]
2024-03-31 00:52:41,710 [main] INFO de.komoot.photon.elasticsearch.Server - Elasticsearch cluster is ready and in a green state.
2024-03-31 00:52:41,710 [main] INFO de.komoot.photon.App - ES cluster is now ready.
Exception in thread "main" [photon] IndexNotFoundException[no such index]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:187)
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:123)
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteSingleIndex(IndexNameExpressionResolver.java:244)
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:146)
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:123)
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:95)
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:59)
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:146)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84)
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83)
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
at de.komoot.photon.elasticsearch.Server.loadFromDatabase(Server.java:318)
at de.komoot.photon.elasticsearch.Server.updateIndexSettings(Server.java:265)
at de.komoot.photon.App.main(App.java:55) |
Beta Was this translation helpful? Give feedback.
This looks like indexing was not finished for some reason before the database was shut down after the import was finished. Photon itself has no code for index recovery. You could try to hack something in. According to this stackoverflow any of the following should do the trick:
Best place is probably just hacking it in after
photon/src/main/java/de/komoot/photon/elasticsearch/Server.java
Line 115 in 716b45f
If that works for you, let us know and we can think about adding a recovery functionality to photon.