Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] testRecoveryWithConcurrentIndexing timed out waiting for green state for index #40731

Closed
dliappis opened this issue Apr 2, 2019 · 3 comments · Fixed by #40800
Closed

[CI] testRecoveryWithConcurrentIndexing timed out waiting for green state for index #40731

dliappis opened this issue Apr 2, 2019 · 3 comments · Fixed by #40800
Assignees
@jasontedor
Labels
:Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. >test-failure Triaged test failures from CI

Comments

@dliappis
Copy link
Contributor

dliappis commented Apr 2, 2019

Noticed in several occasions in CI since Mar 21. Examples:

  1> [2019-03-30T10:54:41,595][INFO ][o.e.u.RecoveryIT         ] [testRecoveryWithConcurrentIndexing] after test
FAILURE 71.4s | RecoveryIT.testRecoveryWithConcurrentIndexing <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: timed out waiting for green state for index [recovery_with_concurrent_indexing] cluster state [{"cluster_name":"rolling-upgrade","cluster_uuid":"Us_C6hlTTRqBfjYADMtWrg","version":132,"state_uuid":"7nbHnX6OQKeuwxxWzd58FA","master_node":"49yrPbKOTIeapnh5WhVqrA","blocks":{"indices":{"index_mixed_7000099":{"4":{"description":"index closed","retryable":false,"levels":["read","write"]}},"index_old_7000099":{"4":{"description":"index closed","retryable":false,"levels":["read","write"]}},"closed_index_created_on_old":{"4":{"description":"index closed","retryable":false,"levels":["read","write"]}}}},"nodes":{"MvtgxXYhQT67_WcAC6dT6Q":{"name":"upgraded-node-1","ephemeral_id":"-619I8F0Q4qZcoLu6zdlTA","transport_address":"127.0.0.1:41963","attributes":{"testattr":"test"}},"49yrPbKOTIeapnh5WhVqrA":{"name":"node-2","ephemeral_id":"wBRmr3EqR_KXL_13m3snbg","transport_address":"127.0.0.1:43591","attributes":{"testattr":"test"}}}
Detailed output 1> [2019-03-30T10:54:41,595][INFO ][o.e.u.RecoveryIT ] [testRecoveryWithConcurrentIndexing] after test FAILURE 71.4s | RecoveryIT.testRecoveryWithConcurrentIndexing <<< FAILURES! > Throwable #1: java.lang.AssertionError: timed out waiting for green state for index [recovery_with_concurrent_indexing] cluster state [{"cluster_name":"rolling-upgrade","cluster_uuid":"Us_C6hlTTRqBfjYADMtWrg","version":132,"state_uuid":"7nbHnX6OQKeuwxxWzd58FA","master_node":"49yrPbKOTIeapnh5WhVqrA","blocks":{"indices":{"index_mixed_7000099":{"4":{"description":"index closed","retryable":false,"levels":["read","write"]}},"index_old_7000099":{"4":{"description":"index closed","retryable":false,"levels":["read","write"]}},"closed_index_created_on_old":{"4":{"description":"index closed","retryable":false,"levels":["read","write"]}}}},"nodes":{"MvtgxXYhQT67_WcAC6dT6Q":{"name":"upgraded-node-1","ephemeral_id":"-619I8F0Q4qZcoLu6zdlTA","transport_address":"127.0.0.1:41963","attributes":{"testattr":"test"}},"49yrPbKOTIeapnh5WhVqrA":{"name":"node-2","ephemeral_id":"wBRmr3EqR_KXL_13m3snbg","transport_address":"127.0.0.1:43591","attributes":{"testattr":"test"}}},"metadata":{"cluster_uuid":"Us_C6hlTTRqBfjYADMtWrg","cluster_coordination":{"term":1,"last_committed_config":["MvtgxXYhQT67_WcAC6dT6Q","OumhsXTwRZG95uWE2DCwfg","49yrPbKOTIeapnh5WhVqrA"],"last_accepted_config":["MvtgxXYhQT67_WcAC6dT6Q","OumhsXTwRZG95uWE2DCwfg","49yrPbKOTIeapnh5WhVqrA"],"voting_config_exclusions":[]},"templates":{},"indices":{".tasks":{"state":"open","settings":{"index":{"number_of_shards":"1","auto_expand_replicas":"0-1","provided_name":".tasks","creation_date":"1553953947887","priority":"2147483647","number_of_replicas":"1","uuid":"tTgPMYJQTfirkCPbu3hh3g","version":{"created":"7000099"}}},"mappings":{"task":{"_meta":{"version":2},"dynamic":"strict","properties":{"task":{"properties":{"headers":{"type":"object","enabled":false},"node":{"type":"keyword"},"running_time_in_nanos":{"type":"long"},"parent_id":{"type":"keyword"},"action":{"type":"keyword"},"description":{"type":"text"},"start_time_in_millis":{"type":"long"},"id":{"type":"long"},"type":{"type":"keyword"},"cancellable":{"type":"boolean"},"status":{"type":"object","enabled":false}}},"response":{"type":"object","enabled":false},"completed":{"type":"boolean"},"error":{"type":"object","enabled":false}}}},"aliases":[],"primary_terms":{"0":4},"in_sync_allocations":{"0":["pP9VapvfQcC4el6hDOsAaw","QzCX2Kg4QPSjRYG51k3qig"]}},"index_old_7000099":{"state":"close","settings":{"index":{"creation_date":"1553953951380","number_of_shards":"1","number_of_replicas":"0","uuid":"xHf037_RRbGvEJk3klWIhA","version":{"created":"7000099"},"provided_name":"index_old_7000099"}},"mappings":{},"aliases":[],"primary_terms":{"0":1},"in_sync_allocations":{"0":["cIm9u0MUS5eP6LkubkXUzw"]}},"reindexed_index_copy":{"state":"open","settings":{"index":{"creation_date":"1553953947601","number_of_shards":"1","number_of_replicas":"1","uuid":"CuLGzmo-Qaib7cOVCnvMlw","version":{"created":"7000099"},"provided_name":"reindexed_index_copy"}},"mappings":{"_doc":{"properties":{"f1":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}}}}},"aliases":[],"primary_terms":{"0":2},"in_sync_allocations":{"0":["FALKjYPyRj2M_ubfPwpNVg","t6_IuMG5SVu1AcafAiY1GQ"]}},"recover_with_soft_deletes":{"state":"open","settings":{"index":{"allocation":{"max_retries":"0"},"number_of_shards":"1","soft_deletes":{"enabled":"true"},"provided_name":"recover_with_soft_deletes","creation_date":"1553953951632","unassigned":{"node_left":{"delayed_timeout":"100ms"}},"number_of_replicas":"1","uuid":"yu0ktRbpTXmUD7FaZfdcDQ","version":{"created":"7000099"}}},"mappings":{"test":{"properties":{"test":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}}}}},"aliases":[],"primary_terms":{"0":1},"in_sync_allocations":{"0":["GuCxucX0QTyZYM5MK_Wakg","F-qDy2-vSJGtFc8qISWhxA"]}},"closed_index_created_on_old":{"state":"close","settings":{"index":{"allocation":{"max_retries":"0"},"number_of_shards":"1","provided_name":"closed_index_created_on_old","creation_date":"1553953952638","unassigned":{"node_left":{"delayed_timeout":"100ms"}},"number_of_replicas":"1","uuid":"uGzMRSvIQ5KqaKPawCBf9Q","version":{"created":"7000099"}}},"mappings":{},"aliases":[],"primary_terms":{"0":1},"in_sync_allocations":{"0":["KAVHmWPyQluuKIGpvS-zVg","L9JvFcqKSYiRd1JneKWLrQ"]}},"index_mixed_7000099":{"state":"close","settings":{"index":{"creation_date":"1553953981424","number_of_shards":"1","number_of_replicas":"0","uuid":"0ckwuCsdQf-aqgE3h5srJw","version":{"created":"7000099"},"provided_name":"index_mixed_7000099"}},"mappings":{},"aliases":[],"primary_terms":{"0":1},"in_sync_allocations":{"0":["d1B2mXMMSYyW3ScrcY3M-A"]}},"queries":{"state":"open","settings":{"index":{"creation_date":"1553953946381","number_of_shards":"1","number_of_replicas":"1","uuid":"Y-tpd2voQZ-6FO9C8d-55w","version":{"created":"7000099"},"provided_name":"queries"}},"mappings":{"_doc":{"properties":{"field1":{"type":"keyword"},"query":{"type":"percolator"},"field3":{"type":"keyword"},"field2":{"type":"keyword"}}}},"aliases":[],"primary_terms":{"0":3},"in_sync_allocations":{"0":["fRzaiUIbQT2GvE_tdvvRkw","lLZwfsfZRVmFCU4GPoIaQQ"]}},"reindexed_index":{"state":"open","settings":{"index":{"creation_date":"1553953947331","number_of_shards":"1","number_of_replicas":"0","uuid":"5Nbs5t4ORgmNVcOJuBnUJw","version":{"created":"7000099"},"provided_name":"reindexed_index"}},"mappings":{"_doc":{"properties":{"f1":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}}}}},"aliases":[],"primary_terms":{"0":4},"in_sync_allocations":{"0":["L1D_v_xgSsaNQeZAD6rgyg"]}},"test_index":{"state":"open","settings":{"index":{"creation_date":"1553953948531","number_of_shards":"5","number_of_replicas":"0","uuid":"HS4WWT_eTgiH5lS2k_6Y_Q","version":{"created":"7000099"},"provided_name":"test_index"}},"mappings":{"_doc":{"properties":{"f1":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"f2":{"type":"long"}}}},"aliases":[],"primary_terms":{"0":1,"1":4,"2":3,"3":1,"4":4},"in_sync_allocations":{"1":["Gmfxy0LvSvqlSYVD55dDBA"],"4":["8t0OsIkGSkOspQwTxeZqDg"],"3":["34g-EcenScOcHvVn3FdvIw"],"2":["N_YmfX1ATKev2wuDSQZEYg"],"0":["re1X_kDiQPyq9LzUzk49EA"]}},"recover_synced_flush_index":{"state":"open","settings":{"index":{"allocation":{"max_retries":"0"},"number_of_shards":"1","provided_name":"recover_synced_flush_index","creation_date":"1553953952375","unassigned":{"node_left":{"delayed_timeout":"100ms"}},"number_of_replicas":"1","uuid":"wtYIwMaZTdSgnY9mZCD6Lw","version":{"created":"7000099"}}},"mappings":{},"aliases":[],"primary_terms":{"0":2},"in_sync_allocations":{"0":["XxZmrc2zQnmR_QjHqnUDlQ","b-GhsyRbRtmrprwm6QCRDw"]}},"index_with_replicas":{"state":"open","settings":{"index":{"number_of_shards":"5","provided_name":"index_with_replicas","creation_date":"1553953948678","unassigned":{"node_left":{"delayed_timeout":"100ms"}},"number_of_replicas":"1","uuid":"LQBB-XMVSK-tdtxC_bl57A","version":{"created":"7000099"}}},"mappings":{"_doc":{"properties":{"f1":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"f2":{"type":"long"}}}},"aliases":[],"primary_terms":{"0":3,"1":1,"2":3,"3":2,"4":1},"in_sync_allocations":{"1":["Rvo-nhSeRWCbxTwBuU0G5Q","FDhcD3RUTequICTrPyYsUw"],"4":["8kzkU8vAS8KlbKsPmd728A","QF-tVvTlTnWO1LAvmikm0w"],"3":["8MbFPdW4RWSj_Nm-fmeK2w","daEs0CbfTnaBcK5d1cSE8w"],"2":["yCZrkNP_Qde8sS4-QPjGxg","lJfpDvRjSEud1qsnQPn3fg"],"0":["EWMCI-Y4Q_-BB9l-Oc8jAg","uD0v_xb_SEu4oduM8WzL5w"]}},"index_history_uuid":{"state":"open","settings":{"index":{"number_of_shards":"1","provided_name":"index_history_uuid","creation_date":"1553953951145","unassigned":{"node_left":{"delayed_timeout":"100ms"}},"number_of_replicas":"1","uuid":"fPCiYO_tSVSe0UpPgMsELA","version":{"created":"7000099"}}},"mappings":{},"aliases":[],"primary_terms":{"0":3},"in_sync_allocations":{"0":["TRTh0QPiSvyvJ9RE_BUmZQ","gdcblrfmTBGR3SpC0f5i1Q"]}},"test_search_template":{"state":"open","settings":{"index":{"creation_date":"1553953943988","number_of_shards":"1","number_of_replicas":"1","uuid":"Vx4DP8vmQVm_avIk16Wpqg","version":{"created":"7000099"},"provided_name":"test_search_template"}},"mappings":{"_doc":{"properties":{"f1":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}}}}},"aliases":[],"primary_terms":{"0 2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+7.x+intake/qa/rolling-upgrade/build/testrun/v7.0.0#twoThirdsUpgradedTestRunner/J0/temp/org.elasticsearch.upgrades.RecoveryIT_1C6C179920578E24-001 ":1},"in_sync_allocations":{"0":["CZg9TQWGTXuewhkxZZiQTA","usz8P375QWyzCRy8-prPUA"]}},"empty_index":{"state":"open","settings":{"index":{"number_of_shards":"5","provided_name":"empty_index","creation_date":"1553953948853","unassigned":{"node_left":{"delayed_timeout":"100ms"}},"number_of_replicas":"1","uuid":"uRa5XmgPQoK59BAij0Rg5g","version":{"created":"7000099"}}},"mappings":{},"aliases":[],"primary_terms":{"0":2,"1":2,"2":1,"3":2,"4":2},"in_sync_allocations":{"1":["_jCt9P-RSGyNJGo6W4WqOQ","hKti3OXBQMuqCi_cHj_1Xg"],"4":["WUfG6-IbQS2LNsnA0JN1HQ","NzcmykS4ToudAjtnIFpsBg"],"3":["p9ZJE_3yS2i4W9B_kUuBrQ","GdyxxzdrTcyuVwa5xlHQDA"],"2":["9llE9H4bSTWEhBgOsO3hiA","GM7xLc0cTbmHyRtztrBQlg"],"0":["rwyycFr2Stu9QgnDtOPUew","4Ff2VHi8QwePg-S0HIq9TQ"]}},"recovery_with_concurrent_indexing":{"state":"open","settings":{"index":{"allocation":{"max_retries":"0"},"number_of_shards":"1","provided_name":"recovery_with_concurrent_indexing","creation_date":"1553953950472","unassigned":{"node_left":{"delayed_timeout":"100ms"}},"number_of_replicas":"2","uuid":"GFC2LVnBSkqy9f7MgiZTcw","version":{"created":"7000099"}}},"mappings":{"test":{"properties":{"test":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}}}}},"aliases":[],"primary_terms":{"0":2},"in_sync_allocations":{"0":["5oZQVo6mT3C7El8i9h5bxg","VF1HkX1ESN629H68bdHUDg"]}}},"ingest":{"pipeline":[{"id":"my_pipeline","config":{"description":"_description","processors":[]}}]},"index-graveyard":{"tombstones":[]},"repositories":{"my_repo":{"type":"url","settings":{"url":"http://snapshot.test"}}},"stored_scripts":{"test_search_template":{"lang":"mustache","source":"{\"query\":{\"match\":{\"f1\":\"{{f1}}\"}}}","options":{"content_type":"application/json; charset=UTF-8"}}}},"routing_table":{"indices":{"index_history_uuid":{"shards":{"0":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"index_history_uuid","allocation_id":{"id":"TRTh0QPiSvyvJ9RE_BUmZQ"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"index_history_uuid","allocation_id":{"id":"gdcblrfmTBGR3SpC0f5i1Q"}}]}},"test_search_template":{"shards":{"0":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"test_search_template","allocation_id":{"id":"CZg9TQWGTXuewhkxZZiQTA"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"test_search_template","allocation_id":{"id":"usz8P375QWyzCRy8-prPUA"}}]}},"index_with_replicas":{"shards":{"1":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":1,"index":"index_with_replicas","allocation_id":{"id":"Rvo-nhSeRWCbxTwBuU0G5Q"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":1,"index":"index_with_replicas","allocation_id":{"id":"FDhcD3RUTequICTrPyYsUw"}}],"4":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":4,"index":"index_with_replicas","allocation_id":{"id":"8kzkU8vAS8KlbKsPmd728A"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":4,"index":"index_with_replicas","allocation_id":{"id":"QF-tVvTlTnWO1LAvmikm0w"}}],"3":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":3,"index":"index_with_replicas","allocation_id":{"id":"daEs0CbfTnaBcK5d1cSE8w"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":3,"index":"index_with_replicas","allocation_id":{"id":"8MbFPdW4RWSj_Nm-fmeK2w"}}],"2":[{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":2,"index":"index_with_replicas","allocation_id":{"id":"lJfpDvRjSEud1qsnQPn3fg"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":2,"index":"index_with_replicas","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}}],"0":[{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"index_with_replicas","allocation_id":{"id":"uD0v_xb_SEu4oduM8WzL5w"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":0,"index":"index_with_replicas","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}}]}},"reindexed_index":{"shards":{"0":[{"state":"UNASSIGNED","primary":true,"node":null,"relocating_node":null,"shard":0,"index":"reindexed_index","recovery_source":{"type":"EXISTING_STORE","bootstrap_new_history_uuid":false},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.959Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_valid_shard_copy"}}]}},"recovery_with_concurrent_indexing":{"shards":{"0":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"recovery_with_concurrent_indexing","allocation_id":{"id":"5oZQVo6mT3C7El8i9h5bxg"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"recovery_with_concurrent_indexing","allocation_id":{"id":"VF1HkX1ESN629H68bdHUDg"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":0,"index":"recovery_with_concurrent_indexing","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}}]}},"queries":{"shards":{"0":[{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"queries","allocation_id":{"id":"lLZwfsfZRVmFCU4GPoIaQQ"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":0,"index":"queries","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}}]}},"empty_index":{"shards":{"1":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":1,"index":"empty_index","allocation_id":{"id":"_jCt9P-RSGyNJGo6W4WqOQ"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":1,"index":"empty_index","allocation_id":{"id":"hKti3OXBQMuqCi_cHj_1Xg"}}],"4":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":4,"index":"empty_index","allocation_id":{"id":"WUfG6-IbQS2LNsnA0JN1HQ"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":4,"index":"empty_index","allocation_id":{"id":"NzcmykS4ToudAjtnIFpsBg"}}],"3":[{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":3,"index":"empty_index","allocation_id":{"id":"GdyxxzdrTcyuVwa5xlHQDA"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":3,"index":"empty_index","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}}],"2":[{"state":"STARTED","primary":false,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":2,"index":"empty_index","allocation_id":{"id":"9llE9H4bSTWEhBgOsO3hiA"}},{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":2,"index":"empty_index","allocation_id":{"id":"GM7xLc0cTbmHyRtztrBQlg"}}],"0":[{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"empty_index","allocation_id":{"id":"rwyycFr2Stu9QgnDtOPUew"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":0,"index":"empty_index","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}}]}},"recover_with_soft_deletes":{"shards":{"0":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"recover_with_soft_deletes","allocation_id":{"id":"F-qDy2-vSJGtFc8qISWhxA"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"recover_with_soft_deletes","allocation_id":{"id":"GuCxucX0QTyZYM5MK_Wakg"}}]}},"recover_synced_flush_index":{"shards":{"0":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"recover_synced_flush_index","allocation_id":{"id":"XxZmrc2zQnmR_QjHqnUDlQ"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"recover_synced_flush_index","allocation_id":{"id":"b-GhsyRbRtmrprwm6QCRDw"}}]}},"test_index":{"shards":{"1":[{"state":"UNASSIGNED","primary":true,"node":null,"relocating_node":null,"shard":1,"index":"test_index","recovery_source":{"type":"EXISTING_STORE","bootstrap_new_history_uuid":false},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_valid_shard_copy"}}],"4":[{"state":"UNASSIGNED","primary":true,"node":null,"relocating_node":null,"shard":4,"index":"test_index","recovery_source":{"type":"EXISTING_STORE","bootstrap_new_history_uuid":false},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_valid_shard_copy"}}],"3":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":3,"index":"test_index","allocation_id":{"id":"34g-EcenScOcHvVn3FdvIw"}}],"2":[{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":2,"index":"test_index","allocation_id":{"id":"N_YmfX1ATKev2wuDSQZEYg"}}],"0":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"test_index","allocation_id":{"id":"re1X_kDiQPyq9LzUzk49EA"}}]}},"reindexed_index_copy":{"shards":{"0":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"reindexed_index_copy","allocation_id":{"id":"FALKjYPyRj2M_ubfPwpNVg"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"reindexed_index_copy","allocation_id":{"id":"t6_IuMG5SVu1AcafAiY1GQ"}}]}},".tasks":{"shards":{"0":[{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":".tasks","allocation_id":{"id":"QzCX2Kg4QPSjRYG51k3qig"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":0,"index":".tasks","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}}]}}}},"routing_nodes":{"unassigned":[{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":2,"index":"index_with_replicas","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":0,"index":"index_with_replicas","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}},{"state":"UNASSIGNED","primary":true,"node":null,"relocating_node":null,"shard":0,"index":"reindexed_index","recovery_source":{"type":"EXISTING_STORE","bootstrap_new_history_uuid":false},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.959Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_valid_shard_copy"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":0,"index":"recovery_with_concurrent_indexing","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":0,"index":"queries","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":3,"index":"empty_index","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":0,"index":"empty_index","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}},{"state":"UNASSIGNED","primary":true,"node":null,"relocating_node":null,"shard":1,"index":"test_index","recovery_source":{"type":"EXISTING_STORE","bootstrap_new_history_uuid":false},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_valid_shard_copy"}},{"state":"UNASSIGNED","primary":true,"node":null,"relocating_node":null,"shard":4,"index":"test_index","recovery_source":{"type":"EXISTING_STORE","bootstrap_new_history_uuid":false},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_valid_shard_copy"}},{"state":"UNASSIGNED","primary":false,"node":null,"relocating_node":null,"shard":0,"index":".tasks","recovery_source":{"type":"PEER"},"unassigned_info":{"reason":"NODE_LEFT","at":"2019-03-30T13:53:30.960Z","delayed":false,"details":"node_left [OumhsXTwRZG95uWE2DCwfg]","allocation_status":"no_attempt"}}],"nodes":{"49yrPbKOTIeapnh5WhVqrA":[{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"index_history_uuid","allocation_id":{"id":"TRTh0QPiSvyvJ9RE_BUmZQ"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"test_search_template","allocation_id":{"id":"CZg9TQWGTXuewhkxZZiQTA"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":1,"index":"index_with_replicas","allocation_id":{"id":"Rvo-nhSeRWCbxTwBuU0G5Q"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":4,"index":"index_with_replicas","allocation_id":{"id":"8kzkU8vAS8KlbKsPmd728A"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":3,"index":"index_with_replicas","allocation_id":{"id":"daEs0CbfTnaBcK5d1cSE8w"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"recovery_with_concurrent_indexing","allocation_id":{"id":"5oZQVo6mT3C7El8i9h5bxg"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":1,"index":"empty_index","allocation_id":{"id":"_jCt9P-RSGyNJGo6W4WqOQ"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":4,"index":"empty_index","allocation_id":{"id":"WUfG6-IbQS2LNsnA0JN1HQ"}},{"state":"STARTED","primary":false,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":2,"index":"empty_index","allocation_id":{"id":"9llE9H4bSTWEhBgOsO3hiA"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"recover_with_soft_deletes","allocation_id":{"id":"F-qDy2-vSJGtFc8qISWhxA"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"recover_synced_flush_index","allocation_id":{"id":"XxZmrc2zQnmR_QjHqnUDlQ"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":3,"index":"test_index","allocation_id":{"id":"34g-EcenScOcHvVn3FdvIw"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"test_index","allocation_id":{"id":"re1X_kDiQPyq9LzUzk49EA"}},{"state":"STARTED","primary":true,"node":"49yrPbKOTIeapnh5WhVqrA","relocating_node":null,"shard":0,"index":"reindexed_index_copy","allocation_id":{"id":"FALKjYPyRj2M_ubfPwpNVg"}}],"MvtgxXYhQT67_WcAC6dT6Q":[{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"index_history_uuid","allocation_id":{"id":"gdcblrfmTBGR3SpC0f5i1Q"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"test_search_template","allocation_id":{"id":"usz8P375QWyzCRy8-prPUA"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":1,"index":"index_with_replicas","allocation_id":{"id":"FDhcD3RUTequICTrPyYsUw"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":4,"index":"index_with_replicas","allocation_id":{"id":"QF-tVvTlTnWO1LAvmikm0w"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":3,"index":"index_with_replicas","allocation_id":{"id":"8MbFPdW4RWSj_Nm-fmeK2w"}},{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":2,"index":"index_with_replicas","allocation_id":{"id":"lJfpDvRjSEud1qsnQPn3fg"}},{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"index_with_replicas","allocation_id":{"id":"uD0v_xb_SEu4oduM8WzL5w"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"recovery_with_concurrent_indexing","allocation_id":{"id":"VF1HkX1ESN629H68bdHUDg"}},{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"queries","allocation_id":{"id":"lLZwfsfZRVmFCU4GPoIaQQ"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":1,"index":"empty_index","allocation_id":{"id":"hKti3OXBQMuqCi_cHj_1Xg"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":4,"index":"empty_index","allocation_id":{"id":"NzcmykS4ToudAjtnIFpsBg"}},{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":3,"index":"empty_index","allocation_id":{"id":"GdyxxzdrTcyuVwa5xlHQDA"}},{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":2,"index":"empty_index","allocation_id":{"id":"GM7xLc0cTbmHyRtztrBQlg"}},{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"empty_index","allocation_id":{"id":"rwyycFr2Stu9QgnDtOPUew"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"recover_with_soft_deletes","allocation_id":{"id":"GuCxucX0QTyZYM5MK_Wakg"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"recover_synced_flush_index","allocation_id":{"id":"b-GhsyRbRtmrprwm6QCRDw"}},{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":2,"index":"test_index","allocation_id":{"id":"N_YmfX1ATKev2wuDSQZEYg"}},{"state":"STARTED","primary":false,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":"reindexed_index_copy","allocation_id":{"id":"t6_IuMG5SVu1AcafAiY1GQ"}},{"state":"STARTED","primary":true,"node":"MvtgxXYhQT67_WcAC6dT6Q","relocating_node":null,"shard":0,"index":".tasks","allocation_id":{"id":"QzCX2Kg4QPSjRYG51k3qig"}}]}}}] > at __randomizedtesting.SeedInfo.seed([1C6C179920578E24:9C3DD3CAA2F350EE]:0) > at org.elasticsearch.test.rest.ESRestTestCase.ensureGreen(ESRestTestCase.java:817) > at org.elasticsearch.upgrades.RecoveryIT.testRecoveryWithConcurrentIndexing(RecoveryIT.java:152) 2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1577, maxMBSortInHeap=7.29507887818673, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@32698c7c), locale=de-DE, timezone=America/Argentina/San_Luis 2> NOTE: Linux 4.15.0-1028-gcp amd64/Oracle Corporation 1.8.0_202 (64-bit)/cpus=16,threads=1,free=403109616,total=514850816 2> NOTE: All tests run in this JVM: [UpgradeClusterClientYamlTestSuiteIT, XPackIT, IndexingIT, RecoveryIT] > at java.lang.Thread.run(Thread.java:748) 1> [2019-03-30T10:54:41,622][INFO ][o.e.u.RecoveryIT ] [testHistoryUUIDIsGenerated] before test 1> [2019-03-30T10:54:41,647][INFO ][o.e.u.RecoveryIT ] [testHistoryUUIDIsGenerated] after test 1> [2019-03-30T10:54:41,658][INFO ][o.e.u.RecoveryIT ] [testCloseIndexDuringRollingUpgrade] before test 1> [2019-03-30T10:54:41,727][INFO ][o.e.u.RecoveryIT ] [testCloseIndexDuringRollingUpgrade] after test 1> [2019-03-30T10:54:41,742][INFO ][o.e.u.RecoveryIT ] [testRecoveryWithSoftDeletes] before test 1> [2019-03-30T10:54:41,774][INFO ][o.e.u.RecoveryIT ] [testRecoveryWithSoftDeletes] after test 1> [2019-03-30T10:54:41,792][INFO ][o.e.u.RecoveryIT ] [testRecoverSyncedFlushIndex] before test 1> [2019-03-30T10:54:41,821][INFO ][o.e.u.RecoveryIT ] [testRecoverSyncedFlushIndex] after test 1> [2019-03-30T10:54:41,829][INFO ][o.e.u.RecoveryIT ] [testRecoveryClosedIndex] before test 1> [2019-03-30T10:54:41,868][INFO ][o.e.u.RecoveryIT ] [testRecoveryClosedIndex] after test IGNOR/A 0.00s | RecoveryIT.testRelocationWithConcurrentIndexing

Possibly related to #34950
@dliappis dliappis added >test-failure Triaged test failures from CI :Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. labels Apr 2, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

dliappis added a commit to dliappis/elasticsearch that referenced this issue Apr 2, 2019
@dliappis
Mute RecoveryWithConcurrentIndexing test
21e0de0 
Relates elastic#40731
@dliappis dliappis mentioned this issue Apr 2, 2019
Merged
dliappis added a commit to dliappis/elasticsearch that referenced this issue Apr 2, 2019
@dliappis
Mute RecoveryWithConcurrentIndexing test
a646803 
Relates elastic#40731
@dliappis dliappis mentioned this issue Apr 2, 2019
Merged
dliappis added a commit that referenced this issue Apr 2, 2019
@dliappis
[TEST] RecoveryWithConcurrentIndexing test (#40733)
eb8a591 
Due to #40731
dliappis added a commit that referenced this issue Apr 2, 2019
@dliappis
[TEST] Mute RecoveryWithConcurrentIndexing test (#40735)
a3f53cf 
Due to #40731

Backport of #40733
@henningandersen
Copy link
Contributor

Examined this build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu8,nodes=immutable&&linux&&docker/335/consoleText

Here upgraded-node-1 seems to die due to an assertion:

2019-04-02T18:03:09,757][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [upgraded-node-1] fatal error in thread [elasticsearch[upgraded-node-1][management][T#4]], exiting
java.lang.AssertionError: shard [reindexed_index][0], node[hL16jyhTSAifGdmjplE2jg], relocating [RSLEQgbRRrSV7Ou06v1rJw], [P], s[RELOCATING], a[id=Sm6H5wWSRuKxEJ61RKWP2A, rId=jNGWG1sfQQuaiL-4LEK7Xg], expected_shard_size[3689] is not a primary shard in primary mode
        at org.elasticsearch.index.shard.IndexShard.assertPrimaryMode(IndexShard.java:1587) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.shard.IndexShard.syncRetentionLeases(IndexShard.java:2051) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.IndexService.lambda$sync$12(IndexService.java:833) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.shard.IndexShard.lambda$runUnderPrimaryPermit$16(IndexShard.java:2571) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:269) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:236) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2519) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.shard.IndexShard.runUnderPrimaryPermit(IndexShard.java:2575) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.IndexService.sync(IndexService.java:832) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.IndexService.syncRetentionLeases(IndexService.java:815) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.IndexService.access$900(IndexService.java:99) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.index.IndexService$AsyncRetentionLeaseSyncTask.runInternal(IndexService.java:1002) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:141) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:677) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

Thanks @DaveCTurner for helping me find this! Seems like a problem maybe with relocation hand-off related to retention lease syncing?

@dnhatn dnhatn assigned jasontedor and unassigned dnhatn Apr 3, 2019
@dnhatn
Copy link
Member

dnhatn commented Apr 3, 2019

Relates to #40386 (review)

@ywelsch ywelsch mentioned this issue Apr 3, 2019
Merged
ywelsch added a commit that referenced this issue Apr 3, 2019
@ywelsch
Avoid background sync on relocated primary (#40800)
b1c6f8e 
There were some test failures caused by the background retention lease sync running on a relocated
primary. This commit fixes the situation that triggered the assertion and reactivates the failing test.

Closes #40731
ywelsch added a commit that referenced this issue Apr 3, 2019
@ywelsch
Avoid background sync on relocated primary (#40800)
6ae7d59 
There were some test failures caused by the background retention lease sync running on a relocated
primary. This commit fixes the situation that triggered the assertion and reactivates the failing test.

Closes #40731
ywelsch added a commit that referenced this issue Apr 3, 2019
@ywelsch
Avoid background sync on relocated primary (#40800)
e85496d 
There were some test failures caused by the background retention lease sync running on a relocated
primary. This commit fixes the situation that triggered the assertion and reactivates the failing test.

Closes #40731
ywelsch added a commit that referenced this issue Apr 3, 2019
@ywelsch
Avoid background sync on relocated primary (#40800)
7113ded 
There were some test failures caused by the background retention lease sync running on a relocated
primary. This commit fixes the situation that triggered the assertion and reactivates the failing test.

Closes #40731
gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this issue May 27, 2019
@dliappis
[TEST] RecoveryWithConcurrentIndexing test (elastic#40733)
3e1cef1 
Due to elastic#40731
gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this issue May 27, 2019
@ywelsch
Avoid background sync on relocated primary (elastic#40800)
8cd5457 
There were some test failures caused by the background retention lease sync running on a relocated
primary. This commit fixes the situation that triggered the assertion and reactivates the failing test.

Closes elastic#40731
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jasontedor jasontedor

Labels
:Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Avoid background sync on relocated primary ywelsch/elasticsearch
5 participants
@dliappis @jasontedor @dnhatn @elasticmachine @henningandersen