Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shard has not been created, mark shard as failed #3589

Closed
lukapor opened this issue Aug 28, 2013 · 1 comment
Closed

Shard has not been created, mark shard as failed #3589

lukapor opened this issue Aug 28, 2013 · 1 comment

Comments

@lukapor
Copy link

lukapor commented Aug 28, 2013

Hi,

We have some issue to report.
Our cluster has next configuration
Cloud: amazon
Number nodes: 3 (master and data)
Number shard: 5
Number replicas: 1
zen discover: ec2
min master nodes: 2
es version: 0.90.3

Twice a day one node disappears from cluster (you can see from screenshots).

And error log from 1a node:

[2013-08-27 07:37:31,047][INFO ][cluster.metadata ] [EU West 1A] updating number_of_replicas to [2] for indices [production_compositedata_0]
[2013-08-27 07:37:33,418][INFO ][cluster.metadata ] [EU West 1A] updating number_of_replicas to [2] for indices [production_global_person]
[2013-08-27 07:56:12,661][INFO ][cluster.metadata ] [EU West 1A] updating number_of_replicas to [1] for indices [production_compositedata_0]
[2013-08-27 07:56:15,781][INFO ][cluster.metadata ] [EU West 1A] updating number_of_replicas to [1] for indices [production_global_person]
[2013-08-27 11:04:45,806][WARN ][discovery.ec2 ] [EU West 1A] received a join request for an existing node [[EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c}]
[2013-08-27 11:04:46,024][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_compositedata_0][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,024][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,024][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_compositedata_0][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,040][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_compositedata_0][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,040][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,040][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,211][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_compositedata_0][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,211][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,211][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,211][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_compositedata_0][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,211][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,540][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,540][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,541][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_compositedata_0][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,541][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_compositedata_0][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,735][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,736][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,737][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_compositedata_0][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,856][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,856][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,913][WARN ][cluster.action.shard ] [EU West 1A] received shard failed for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 14:30:57,803][INFO ][node ] [EU West 1A] stopping ...
[2013-08-27 14:30:58,162][INFO ][node ] [EU West 1A] stopped
[2013-08-27 14:30:58,162][INFO ][node ] [EU West 1A] closing ...
[2013-08-27 14:30:58,177][INFO ][node ] [EU West 1A] closed
[2013-08-27 14:31:05,384][INFO ][node ] [EU West 1A] version[0.90.3], pid[4852], build[5c38d60/2013-08-06T13:18:31Z]
[2013-08-27 14:31:05,384][INFO ][node ] [EU West 1A] initializing ...
[2013-08-27 14:31:05,416][INFO ][plugins ] [EU West 1A] loaded [transport-thrift, bcsocial-similarity, cloud-aws], sites [bigdesk]
[2013-08-27 14:31:10,111][INFO ][node ] [EU West 1A] initialized
[2013-08-27 14:31:10,111][INFO ][node ] [EU West 1A] starting ...
[2013-08-27 14:31:10,142][INFO ][thrift ] [EU West 1A] bound on port [9500]
[2013-08-27 14:31:10,439][INFO ][transport ] [EU West 1A] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/192.168.24.26:9300]}
[2013-08-27 14:31:17,973][INFO ][cluster.service ] [EU West 1A] detected_master [EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}, added {[EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c},[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b},}, reason: zen-disco-receive(from master [[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}])
[2013-08-27 14:31:19,658][INFO ][discovery ] [EU West 1A] ProductionSearch/3GzUTWxzS6eHbrCenZaHgw
[2013-08-27 14:31:19,736][INFO ][http ] [EU West 1A] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/192.168.24.26:9200]}
[2013-08-27 14:31:19,736][INFO ][node ] [EU West 1A] started
[2013-08-27 23:16:22,453][INFO ][discovery.ec2 ] [EU West 1A] master_left [[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}], reason [do not exists on master, act as master failure]
[2013-08-27 23:16:22,453][INFO ][cluster.service ] [EU West 1A] master {new [EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}, previous [EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}}, removed {[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b},}, reason: zen-disco-master_failed ([EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b})
[2013-08-28 05:46:25,046][INFO ][cluster.service ] [EU West 1A] added {[EU West 1B][wP1cdN2ESZeBMIoVXe5zZQ][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b},}, reason: zen-disco-receive(join from node[[EU West 1B][wP1cdN2ESZeBMIoVXe5zZQ][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}])
[2013-08-28 09:07:33,374][INFO ][node ] [EU West 1A] stopping ...
[2013-08-28 09:07:33,514][INFO ][node ] [EU West 1A] stopped
[2013-08-28 09:07:33,514][INFO ][node ] [EU West 1A] closing ...
[2013-08-28 09:07:33,530][INFO ][node ] [EU West 1A] closed
[2013-08-28 09:07:41,158][INFO ][node ] [EU West 1A] version[0.90.3], pid[2532], build[5c38d60/2013-08-06T13:18:31Z]
[2013-08-28 09:07:41,158][INFO ][node ] [EU West 1A] initializing ...
[2013-08-28 09:07:41,189][INFO ][plugins ] [EU West 1A] loaded [transport-thrift, bcsocial-similarity, cloud-aws], sites [bigdesk]
[2013-08-28 09:07:46,088][INFO ][node ] [EU West 1A] initialized
[2013-08-28 09:07:46,088][INFO ][node ] [EU West 1A] starting ...
[2013-08-28 09:07:46,134][INFO ][thrift ] [EU West 1A] bound on port [9500]
[2013-08-28 09:07:46,415][INFO ][transport ] [EU West 1A] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/192.168.24.26:9300]}
[2013-08-28 09:07:53,934][INFO ][cluster.service ] [EU West 1A] detected_master [EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c}, added {[EU West 1B][wP1cdN2ESZeBMIoVXe5zZQ][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b},[EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c},}, reason: zen-disco-receive(from master [[EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c}])
[2013-08-28 09:07:54,511][INFO ][discovery ] [EU West 1A] ProductionSearch/O3bUWXlkR2OIXD0tY2LKmQ
[2013-08-28 09:07:54,589][INFO ][http ] [EU West 1A] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/192.168.24.26:9200]}
[2013-08-28 09:07:54,589][INFO ][node ] [EU West 1A] started

...1b node:

[2013-08-27 11:07:16,139][INFO ][discovery.ec2 ] [EU West 1B] master_left [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]
[2013-08-27 11:07:16,139][INFO ][cluster.service ] [EU West 1B] master {new [EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}, previous [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}}, removed {[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-master_failed ([EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a})
[2013-08-27 14:31:17,737][INFO ][cluster.service ] [EU West 1B] added {[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-receive(join from node[[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}])
[2013-08-27 23:16:22,097][INFO ][cluster.service ] [EU West 1B] removed {[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-node_failed([EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2013-08-28 05:45:59,420][INFO ][node ] [EU West 1B] stopping ...
[2013-08-28 05:45:59,576][INFO ][node ] [EU West 1B] stopped
[2013-08-28 05:45:59,576][INFO ][node ] [EU West 1B] closing ...
[2013-08-28 05:45:59,638][INFO ][node ] [EU West 1B] closed
[2013-08-28 05:46:10,605][INFO ][node ] [EU West 1B] version[0.90.3], pid[2572], build[5c38d60/2013-08-06T13:18:31Z]
[2013-08-28 05:46:10,621][INFO ][node ] [EU West 1B] initializing ...
[2013-08-28 05:46:10,683][INFO ][plugins ] [EU West 1B] loaded [transport-thrift, bcsocial-similarity, cloud-aws], sites [bigdesk]
[2013-08-28 05:46:17,126][INFO ][node ] [EU West 1B] initialized
[2013-08-28 05:46:17,126][INFO ][node ] [EU West 1B] starting ...
[2013-08-28 05:46:17,157][INFO ][thrift ] [EU West 1B] bound on port [9500]
[2013-08-28 05:46:17,500][INFO ][transport ] [EU West 1B] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/192.168.32.27:9300]}
[2013-08-28 05:46:25,253][INFO ][cluster.service ] [EU West 1B] detected_master [EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}, added {[EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c},[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-receive(from master [[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}])
[2013-08-28 05:46:26,439][INFO ][discovery ] [EU West 1B] ProductionSearch/wP1cdN2ESZeBMIoVXe5zZQ
[2013-08-28 05:46:27,110][INFO ][http ] [EU West 1B] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/192.168.32.27:9200]}
[2013-08-28 05:46:27,125][INFO ][node ] [EU West 1B] started
[2013-08-28 08:50:37,742][INFO ][cluster.service ] [EU West 1B] master {new [EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c}, previous [EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}}, removed {[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-receive(from master [[EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c}])
[2013-08-28 09:07:33,514][INFO ][discovery.ec2 ] [EU West 1B] master_left [[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}], reason [shut_down]
[2013-08-28 09:07:53,684][INFO ][cluster.service ] [EU West 1B] added {[EU West 1A][O3bUWXlkR2OIXD0tY2LKmQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-receive(from master [[EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c}])

..and 1c node:

[2013-08-27 11:04:38,711][INFO ][discovery.ec2 ] [EU West 1C] master_left [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]
[2013-08-27 11:04:38,711][INFO ][cluster.service ] [EU West 1C] master {new [EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}, previous [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}}, removed {[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-master_failed ([EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a})
[2013-08-27 11:04:39,756][INFO ][discovery.ec2 ] [EU West 1C] master_left [[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}], reason [no longer master]
[2013-08-27 11:04:39,756][WARN ][discovery.ec2 ] [EU West 1C] not enough master nodes after master left (reason = no longer master), current nodes: {[EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c},}
[2013-08-27 11:04:39,772][INFO ][cluster.service ] [EU West 1C] removed {[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b},}, reason: zen-disco-master_failed ([EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b})
[2013-08-27 11:04:45,918][INFO ][cluster.service ] [EU West 1C] detected_master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}, added {[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b},[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-receive(from master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}])
[2013-08-27 11:04:46,121][WARN ][indices.cluster ] [EU West 1C] [production_global_person][0] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,121][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,121][WARN ][indices.cluster ] [EU West 1C] [production_global_person][2] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,121][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,121][WARN ][indices.cluster ] [EU West 1C] [production_global_person][3] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,121][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,121][WARN ][indices.cluster ] [EU West 1C] [production_compositedata_0][0] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,121][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_compositedata_0][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,121][WARN ][indices.cluster ] [EU West 1C] [production_compositedata_0][2] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,121][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_compositedata_0][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,121][WARN ][indices.cluster ] [EU West 1C] [production_compositedata_0][3] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,121][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_compositedata_0][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,308][WARN ][indices.cluster ] [EU West 1C] [production_global_person][0] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,308][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,308][WARN ][indices.cluster ] [EU West 1C] [production_global_person][2] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,308][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,308][WARN ][indices.cluster ] [EU West 1C] [production_global_person][3] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,308][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,308][WARN ][indices.cluster ] [EU West 1C] [production_compositedata_0][2] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,324][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_compositedata_0][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,324][WARN ][indices.cluster ] [EU West 1C] [production_compositedata_0][3] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,324][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_compositedata_0][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,636][WARN ][indices.cluster ] [EU West 1C] [production_global_person][0] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,636][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,636][WARN ][indices.cluster ] [EU West 1C] [production_global_person][3] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,636][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,636][WARN ][indices.cluster ] [EU West 1C] [production_compositedata_0][2] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,636][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_compositedata_0][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,636][WARN ][indices.cluster ] [EU West 1C] [production_compositedata_0][3] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,636][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_compositedata_0][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,839][WARN ][indices.cluster ] [EU West 1C] [production_global_person][0] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,839][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,839][WARN ][indices.cluster ] [EU West 1C] [production_global_person][3] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,839][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,839][WARN ][indices.cluster ] [EU West 1C] [production_compositedata_0][2] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,839][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_compositedata_0][2], node[kIu8-vq4T8KSB83cgq8qIQ], [R], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,963][WARN ][indices.cluster ] [EU West 1C] [production_global_person][0] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,963][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][0], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:46,963][WARN ][indices.cluster ] [EU West 1C] [production_global_person][3] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:46,963][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:04:47,026][WARN ][indices.cluster ] [EU West 1C] [production_global_person][3] master [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-27 11:04:47,026][WARN ][cluster.action.shard ] [EU West 1C] sending failed shard for [production_global_person][3], node[kIu8-vq4T8KSB83cgq8qIQ], [P], s[STARTED], reason [master [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-27 11:07:16,392][INFO ][cluster.service ] [EU West 1C] master {new [EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}, previous [EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}}, removed {[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-receive(from master [[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}])
[2013-08-27 14:30:58,040][INFO ][discovery.ec2 ] [EU West 1C] master_left [[EU West 1A][FS6eNXw1R7iQGmN6YHfarQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}], reason [shut_down]
[2013-08-27 14:31:17,602][INFO ][cluster.service ] [EU West 1C] added {[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-receive(from master [[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}])
[2013-08-27 23:16:22,171][INFO ][cluster.service ] [EU West 1C] removed {[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-receive(from master [[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}])
[2013-08-27 23:16:22,420][INFO ][cluster.service ] [EU West 1C] master {new [EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}, previous [EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}}, removed {[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b},}, added {[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-receive(from master [[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}])
[2013-08-28 05:45:59,638][INFO ][discovery.ec2 ] [EU West 1C] master_left [[EU West 1B][Y5G-CYdzSP2FiZ0Miqshxg][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b}], reason [shut_down]
[2013-08-28 05:46:25,097][INFO ][cluster.service ] [EU West 1C] added {[EU West 1B][wP1cdN2ESZeBMIoVXe5zZQ][inet[/192.168.32.27:9300]]{aws_availability_zone=eu-west-1b},}, reason: zen-disco-receive(from master [[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}])
[2013-08-28 08:50:37,594][INFO ][discovery.ec2 ] [EU West 1C] master_left [[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]
[2013-08-28 08:50:37,594][INFO ][cluster.service ] [EU West 1C] master {new [EU West 1C][kIu8-vq4T8KSB83cgq8qIQ][inet[/192.168.48.28:9300]]{aws_availability_zone=eu-west-1c}, previous [EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}}, removed {[EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-master_failed ([EU West 1A][3GzUTWxzS6eHbrCenZaHgw][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a})
[2013-08-28 09:07:53,533][INFO ][cluster.service ] [EU West 1C] added {[EU West 1A][O3bUWXlkR2OIXD0tY2LKmQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a},}, reason: zen-disco-receive(join from node[[EU West 1A][O3bUWXlkR2OIXD0tY2LKmQ][inet[/192.168.24.26:9300]]{aws_availability_zone=eu-west-1a}])

Thank you for your help.

1a
1c
1b
@clintongormley
Copy link

Hi @lukapor

Sorry it has taken a while to look at this. It looks like your nodes are being shut down by some other process. If you're still seeing this issue on a more recent version, please could you open another ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@clintongormley @lukapor