Restore shard routing. #6393
Restore shard routing. #6393
Conversation
Routing has been inadvertly changed in elastic#5562 resulting in documents going to different shards in 1.2. This is a terrible bug because an indexing request would not necessarily go to the same shard anymore, potentially leading to duplicates. Close elastic#6391
|
LGTM |
| @@ -0,0 +1,217 @@ | |||
| # Index num_shards _type _id _routing shard_id | |||
There was a problem hiding this comment.
Shouldn't this file should be in src/test/resources rather than src/main/resources since its a test file?
|
Thanks @colings86 I updated the path and will push shortly... |
|
If data is indexed with the broken 1.2 and then a fixed version is released then won't that, again, lead to data corruption. Really nasty bug :( |
|
@andrassy Agreed, indexed data with 1.2.0 will not be compatible with 1.2.1. Yet, we are working on tooling / help for the problem. I also agree that this bug is really nasty. |
|
We had 0.90.7 installed, with no routing in place. We updated to 1.2.0 and reindexed everything with custom routing, that is now enforced using aliases (we don't explicitly route documents in the queries). Does this mean that we might store (or might have stored) documents in the wrong shards, putting them in some kind of endless limbo? Any advice on how we should proceed? |
|
@micpalmia sadly roughly 50% of the documents indexed with 1.2.0 is affected by this issue - regardless of whether custom routing is used or not. We're still evaluating the options for creating a tool to help identify (and hopefully solve) the problematic documents but in your case it will likely be half of your data. Since you've reindexed once I think your quickest option will be to reindex again once you upgrade to 1.2.1. Do note that 1.2.0 is consistent with it self - so if you indexed all data with 1.2.0 you're good to access it with 1.2.0. This may help with timing the reindexing. |
|
Thank you @bleskes, should we look somewhere specific for future updates about this issue? |
|
Any updates here? This is about as ugly as it could get. It's shocking something like this made it out. Not only does it impact us, it impacts our customers and our SLAs. Is there a manual workaround at the moment? Deleting duplicates through a query and re-indexing? Re-indexing alone won't do it. |
|
@nariman-haghighi we are working on a tool that will allow to go over the data and fix it, should be out in a day or 2. |
|
@nariman-haghighi one manual workaround is to move to |
|
The workaround of re-indexing into a new index worked for 2/3 indexed but something peculiar happens with the 3rd index. It has 454 documents (all unique Ids), but the attempt to re-index with the bulk API results in 1 document in the destination index. No changes to mappings. May be a NEST issue? /cc: @Mpdreamz |
|
@nariman-haghighi Have you checked if the bulk response contained errors? Additionally, http://www.elasticsearch.org/blog/tool-help-routing-issues-elasticsearch-1-2-0/ might help get your indices back to normal. |
Routing has been inadvertly changed in #5562 resulting in documents going to
different shards in 1.2. This is a terrible bug because an indexing request
would not necessarily go to the same shard anymore, potentially leading to
duplicates.
Close #6391