New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

timeInQueue for pending_tasks growing too fast? #15988

Closed
nellicus opened this Issue Jan 14, 2016 · 3 comments

Comments

Projects
None yet
3 participants
@nellicus
Member

nellicus commented Jan 14, 2016

Hello, on 2.1.1 3 nodes cluster seeing timeInQueue growing faster than it should?

  1. start three node cluster whose pending_tasks queue grows right after starting
  2. call GET _cat/pending_tasks every second
abonuccelli@w530 ~ $ while true;do date;curl -XGET -u admin:r1ng3r -k 'https://w530:9200/_cat/pending_tasks';sleep 1;done
jue ene 14 15:36:23 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:24 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:25 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:26 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:28 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:29 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:30 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:31 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:32 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:33 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:34 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:35 CET 2016
curl: (7) Failed to connect to w530 port 9200: Connection refused
jue ene 14 15:36:36 CET 2016
5 11.3m NORMAL local-gateway-elected-state                                                                                          
7    6m URGENT zen-disco-join(join from node[{node3}{zPm--nb4Q1GyDfUgPRki0w}{192.168.1.105}{w530/192.168.1.105:9302}{master=true}]) 
6 10.7m HIGH   cluster_reroute(async_shard_fetch)                                                                                   
jue ene 14 15:36:37 CET 2016
8 14.2s HIGH cluster_reroute(async_shard_fetch) 
9  1.4s HIGH cluster_reroute(async_shard_fetch) 
jue ene 14 15:36:38 CET 2016
 9 19.4m HIGH   cluster_reroute(async_shard_fetch) 
10  9.3m URGENT shard-started ([logstash-unparsed-2016.01.14][0], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[23], s[INITIALIZING], a[id=6bPUppNYR96GpmRzRA2lVA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
11  9.1m URGENT shard-started ([logstash-unparsed-2016.01.14][2], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[26], s[INITIALIZING], a[id=tKqikFUkTqmHtr4SwQKmzQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
12  1.4m URGENT shard-started ([logstash-auth-2016.01.14][2], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[22], s[INITIALIZING], a[id=0uNZSK5bTjav1G5U462xPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store]     
13  1.1m URGENT shard-started ([logstash-auth-2016.01.14][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[21], s[INITIALIZING], a[id=-FSh28JvTw6CzPlT-rKslQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store]     
14    1m URGENT shard-started ([logstash-unparsed-2016.01.14][1], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[26], s[INITIALIZING], a[id=A6yx_FkqRb6BRmNyn4v9dA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
15   47s URGENT shard-started ([logstash-auth-2016.01.14][1], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[22], s[INITIALIZING], a[id=7LEi3mQMRlOebCBTVITYqQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store]     
jue ene 14 15:36:39 CET 2016
17  5.1m URGENT shard-started ([logstash-auth-2016.01.12][0], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[40], s[INITIALIZING], a[id=oRhDl4tWQjyXrHjYtyyFfw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store]     
18  3.9m URGENT shard-started ([logstash-auth-2016.01.13][2], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[32], s[INITIALIZING], a[id=WM9zpWx5T4ujXxPdskbuOA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store]     
19  3.9m URGENT shard-started ([logstash-unparsed-2016.01.13][1], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[36], s[INITIALIZING], a[id=NZmkrLKfTMKT4a4iSfGtBQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
20  3.1m URGENT shard-started ([logstash-auth-2016.01.13][1], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[36], s[INITIALIZING], a[id=7_fdmALYRYCE6Hs1Ai0Lfg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store]     
22  1.2m URGENT shard-started ([.marvel-es-data][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[26], s[INITIALIZING], a[id=2E_4eG24S8m1mUDSg1NLCQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store]              
21  1.2m URGENT shard-started ([.watch_history-2016.01.05][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[29], s[INITIALIZING], a[id=tUm4Rt7XQhK7zJTRMqawFg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store]    
16 20.8m HIGH   _add_listener_ 
jue ene 14 15:36:40 CET 2016
29 16.5m URGENT shard-started ([logstash-syslog-2016.01.12][1], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[38], s[INITIALIZING], a[id=Cz2-XhQJSBCkUtLTU_XoYg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
30 16.2m URGENT shard-started ([logstash-syslog-2016.01.12][1], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[38], s[INITIALIZING], a[id=Cz2-XhQJSBCkUtLTU_XoYg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [master {node2}{I_iHcd03Q3aX2KXBVsjMhA}{192.168.1.105}{192.168.1.105:9301}{master=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started] 
31 14.5m URGENT shard-started ([logstash-syslog-2016.01.12][2], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[34], s[INITIALIZING], a[id=W3HDLC-nTWm33uJt9mCvyQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
33  8.2m URGENT shard-started ([logstash-unparsed-2016.01.12][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[37], s[INITIALIZING], a[id=70JqKeAaTgO9HGqX0UYaWg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
36 47.8s URGENT shard-started ([.watch_history-2016.01.12][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[34], s[INITIALIZING], a[id=L9Wpm4MlQtGuhq2Ps7-3Gg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
32   12m URGENT shard-started ([logstash-auth-2016.01.12][2], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[33], s[INITIALIZING], a[id=jAEM0xJMSFCAw7xAp3Zh5g], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
34    8m URGENT shard-started ([logstash-auth-2016.01.12][1], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[33], s[INITIALIZING], a[id=aQYnVPBWTx24BR5L-H9egQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
35    8m URGENT shard-started ([logstash-unparsed-2016.01.12][1], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[36], s[INITIALIZING], a[id=nFYleWmMQumM9tnc-58TrA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
16 44.9m HIGH   _add_listener_                                                                                                                               
jue ene 14 15:36:42 CET 2016
46  9.3m URGENT shard-started ([logstash-unparsed-2016.01.10][1], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[36], s[INITIALIZING], a[id=8US-Te0YSQCym42iTKKAjw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
47  3.9m URGENT shard-started ([logstash-unparsed-2016.01.10][1], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[36], s[INITIALIZING], a[id=8US-Te0YSQCym42iTKKAjw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [master {node2}{I_iHcd03Q3aX2KXBVsjMhA}{192.168.1.105}{192.168.1.105:9301}{master=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started] 
49  2.4m URGENT shard-started ([logstash-syslog-2016.01.10][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[36], s[INITIALIZING], a[id=3NzmZFu6S8yYVgkir8_Hfg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
48  2.6m URGENT shard-started ([logstash-syslog-2016.01.10][2], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[36], s[INITIALIZING], a[id=VGbLKgt7RYu0WmSuVg3ZDQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
53 25.6s URGENT shard-started ([.watch_history-2016.01.10][0], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[34], s[INITIALIZING], a[id=hykDFlrTQnOVJjY6WPSR4g], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [master {node2}{I_iHcd03Q3aX2KXBVsjMhA}{192.168.1.105}{192.168.1.105:9301}{master=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]    
50  1.7m URGENT shard-started ([logstash-unparsed-2016.01.10][2], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[34], s[INITIALIZING], a[id=rn21EPXSQbOw2YKx_bTilg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
51  1.6m URGENT shard-started ([logstash-unparsed-2016.01.10][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[34], s[INITIALIZING], a[id=yrciN2zZQPaY0XFWkpwUGQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
52  1.3m URGENT shard-started ([.watch_history-2016.01.10][0], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[34], s[INITIALIZING], a[id=hykDFlrTQnOVJjY6WPSR4g], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
16    1h HIGH   _add_listener_ 
54 25.6s URGENT shard-started ([logstash-unparsed-2016.01.10][2], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[34], s[INITIALIZING], a[id=rn21EPXSQbOw2YKx_bTilg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [master {node2}{I_iHcd03Q3aX2KXBVsjMhA}{192.168.1.105}{192.168.1.105:9301}{master=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started] 
jue ene 14 15:36:43 CET 2016
64  8.4m URGENT shard-started ([logstash-syslog-2016.01.09][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[36], s[INITIALIZING], a[id=RnkOosAyRpiaOGvX0hBpTQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
65    6m URGENT shard-started ([logstash-unparsed-2016.01.08][1], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[40], s[INITIALIZING], a[id=I5Q3OFfpSZ-1inpSGLiwZQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
66  5.6m URGENT shard-started ([logstash-unparsed-2016.01.08][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[44], s[INITIALIZING], a[id=AyqMhVCHRpeSdGGZblgKig], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
69  1.2m URGENT shard-started ([.watch_history-2016.01.09][0], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[34], s[INITIALIZING], a[id=4yoPNJg8RjSSSpcMNwolHA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
67  5.5m URGENT shard-started ([logstash-syslog-2016.01.09][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[36], s[INITIALIZING], a[id=RnkOosAyRpiaOGvX0hBpTQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [master {node2}{I_iHcd03Q3aX2KXBVsjMhA}{192.168.1.105}{192.168.1.105:9301}{master=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started] 
68  5.1m URGENT shard-started ([logstash-syslog-2016.01.08][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[34], s[INITIALIZING], a[id=gWL8rqL5RICFBlOPbXXVlg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
16  1.4h HIGH   _add_listener_                                                                                                                               
70 44.1s URGENT shard-started ([logstash-syslog-2016.01.09][1], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[34], s[INITIALIZING], a[id=z21y4FyJTeGab0hhunzPFg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
jue ene 14 15:36:44 CET 2016
78 12.7m URGENT shard-started ([.watch_history-2016.01.07][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[38], s[INITIALIZING], a[id=HnIbZsKmRw2F811HF42Hbw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
79 11.7m URGENT shard-started ([.watch_history-2016.01.07][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[38], s[INITIALIZING], a[id=HnIbZsKmRw2F811HF42Hbw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [master {node2}{I_iHcd03Q3aX2KXBVsjMhA}{192.168.1.105}{192.168.1.105:9301}{master=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started] 
81  8.8m URGENT shard-started ([logstash-syslog-2016.01.07][1], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[36], s[INITIALIZING], a[id=3qO53WStT9CXgbkFnUksiA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
80  9.3m URGENT shard-started ([logstash-syslog-2016.01.07][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[34], s[INITIALIZING], a[id=axCQvobnTLC9PvJPjWAMGw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
83  6.2m URGENT shard-started ([.watch_history-2016.01.08][0], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[34], s[INITIALIZING], a[id=hXmEuKK6Sy2HWm-rHEDAew], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
87   14s URGENT shard-started ([test-geo][1], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[85], s[INITIALIZING], a[id=AGzIixN_RROIhjgaVgQ0GA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery from store] 
82  6.3m URGENT shard-started ([logstash-syslog-2016.01.08][2], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[34], s[INITIALIZING], a[id=tz9YgDHwT7yvXKEXQOX66A], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
84  2.8m URGENT shard-started ([.watches][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[47], s[INITIALIZING], a[id=snXyoQs_QeaA06FhtMKQrg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
85 40.5s URGENT shard-started ([logstash-unparsed-2016.01.07][0], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[35], s[INITIALIZING], a[id=nzKhYoHqSa6v5mjLWQ-kyg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery from store] 
86 38.6s URGENT shard-started ([.watches][0], node[zPm--nb4Q1GyDfUgPRki0w], [P], v[47], s[INITIALIZING], a[id=snXyoQs_QeaA06FhtMKQrg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [master {node2}{I_iHcd03Q3aX2KXBVsjMhA}{192.168.1.105}{192.168.1.105:9301}{master=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]                  
16  1.7h HIGH   _add_listener_                                                                                                                              
jue ene 14 15:36:45 CET 2016
101  2.9m URGENT shard-started ([test-idx][1], node[IjwKfD1UQMqtdKXPsLR8Cw], [P], v[90], s[INITIALIZING], a[id=0BHaiJdMQcSt2NeZJM09pA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.661Z]]), reason [after recovery from store]                                                                                                                     
102  1.8m URGENT shard-started ([logstash-unparsed-2016.01.13][1], node[zPm--nb4Q1GyDfUgPRki0w], [R], v[38], s[INITIALIZING], a[id=j9ubCTjyS8qIamqigOblLQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.664Z]]), reason [after recovery (replica) from node [{node1}{IjwKfD1UQMqtdKXPsLR8Cw}{192.168.1.105}{w530/192.168.1.105:9300}{master=true}]] 
103 35.3s URGENT shard-started ([logstash-unparsed-2016.01.14][2], node[zPm--nb4Q1GyDfUgPRki0w], [R], v[28], s[INITIALIZING], a[id=12I5tGEJRzC6y65FTq4hdw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-01-14T14:36:35.665Z]]), reason [after recovery (replica) from node [{node1}{IjwKfD1UQMqtdKXPsLR8Cw}{192.168.1.105}{w530/192.168.1.105:9300}{master=true}]] 
 16  2.1h HIGH   _add_listener_                                                                            
 97 12.7m HIGH   cluster_reroute(async_shard_fetch)                                                        
jue ene 14 15:36:47 CET 2016
jue ene 14 15:36:48 CET 2016
jue ene 14 15:36:49 CET 2016
jue ene 14 15:36:51 CET 2016
jue ene 14 15:36:52 CET 2016
jue ene 14 15:36:53 CET 2016
jue ene 14 15:36:54 CET 2016
jue ene 14 15:36:55 CET 2016
jue ene 14 15:36:56 CET 2016
jue ene 14 15:36:58 CET 2016
jue ene 14 15:36:59 CET 2016
jue ene 14 15:37:00 CET 2016
jue ene 14 15:37:01 CET 2016
jue ene 14 15:37:02 CET 2016
jue ene 14 15:37:03 CET 2016
jue ene 14 15:37:04 CET 2016
jue ene 14 15:37:06 CET 2016
jue ene 14 15:37:07 CET 2016
jue ene 14 15:37:08 CET 2016
jue ene 14 15:37:09 CET 2016
jue ene 14 15:37:10 CET 2016
jue ene 14 15:37:11 CET 2016
jue ene 14 15:37:12 CET 2016
jue ene 14 15:37:14 CET 2016
jue ene 14 15:37:15 CET 2016
jue ene 14 15:37:16 CET 2016
jue ene 14 15:37:17 CET 2016
jue ene 14 15:37:18 CET 2016
jue ene 14 15:37:19 CET 2016
jue ene 14 15:37:20 CET 2016
jue ene 14 15:37:21 CET 2016
jue ene 14 15:37:22 CET 2016
jue ene 14 15:37:23 CET 2016
jue ene 14 15:37:25 CET 2016
jue ene 14 15:37:26 CET 2016
jue ene 14 15:37:27 CET 2016
jue ene 14 15:37:28 CET 2016
jue ene 14 15:37:29 CET 2016
jue ene 14 15:37:30 CET 2016
jue ene 14 15:37:31 CET 2016
jue ene 14 15:37:32 CET 2016
jue ene 14 15:37:33 CET 2016
jue ene 14 15:37:35 CET 2016
jue ene 14 15:37:36 CET 2016
108 12.9m URGENT delete-index [.shield_audit_log-2016.01.14] 
109 12.5m HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:37 CET 2016
108 33.5m URGENT delete-index [.shield_audit_log-2016.01.14] 
109 33.2m HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:38 CET 2016
108 52.8m URGENT delete-index [.shield_audit_log-2016.01.14] 
109 52.5m HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:39 CET 2016
108 1.1h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 1.1h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:40 CET 2016
108 1.4h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 1.4h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:41 CET 2016
108 1.7h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 1.7h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:42 CET 2016
108 2.1h URGENT delete-index [.shield_audit_log-2016.01.14] 
109   2h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:44 CET 2016
108 2.4h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 2.3h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:45 CET 2016
108 2.7h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 2.7h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:46 CET 2016
108 3h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 3h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:47 CET 2016
108 3.3h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 3.3h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:48 CET 2016
108 3.6h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 3.6h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:49 CET 2016
108 3.9h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 3.9h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:50 CET 2016
108 4.2h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 4.2h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:51 CET 2016
108 4.5h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 4.5h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:52 CET 2016
108 4.8h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 4.8h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:53 CET 2016
108 5.1h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 5.1h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:55 CET 2016
108 5.4h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 5.4h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:56 CET 2016
108 5.7h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 5.7h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:57 CET 2016
108 6.1h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 6.1h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:58 CET 2016
108 6.4h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 6.4h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:37:59 CET 2016
108 6.7h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 6.7h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:38:00 CET 2016
108 7h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 7h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:38:01 CET 2016
108 7.3h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 7.3h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:38:02 CET 2016
108 7.6h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 7.6h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:38:04 CET 2016
108 7.9h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 7.9h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:38:05 CET 2016
108 8.2h URGENT delete-index [.shield_audit_log-2016.01.14] 
109 8.2h HIGH   cluster_reroute(async_shard_fetch)          
jue ene 14 15:38:06 CET 2016
108  8.5h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 10.6m URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109  8.5h HIGH   cluster_reroute(async_shard_fetch)                                  
jue ene 14 15:38:07 CET 2016
108  8.8h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 29.7m URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109  8.8h HIGH   cluster_reroute(async_shard_fetch)                                  
111  7.8m HIGH   _add_listener_                                                      
jue ene 14 15:38:08 CET 2016
108  9.2h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 48.4m URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109  9.1h HIGH   cluster_reroute(async_shard_fetch)                                  
111 26.5m HIGH   _add_listener_                                                      
jue ene 14 15:38:09 CET 2016
108  9.5h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  1.1h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109  9.5h HIGH   cluster_reroute(async_shard_fetch)                                  
111 44.9m HIGH   _add_listener_                                                      
jue ene 14 15:38:10 CET 2016
108 9.8h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 1.4h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 9.8h HIGH   cluster_reroute(async_shard_fetch)                                  
111   1h HIGH   _add_listener_                                                      
jue ene 14 15:38:11 CET 2016
108 10.1h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  1.7h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 10.1h HIGH   cluster_reroute(async_shard_fetch)                                  
111  1.3h HIGH   _add_listener_                                                      
jue ene 14 15:38:12 CET 2016
108 10.4h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110    2h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 10.4h HIGH   cluster_reroute(async_shard_fetch)                                  
111  1.6h HIGH   _add_listener_                                                      
jue ene 14 15:38:14 CET 2016
108 10.7h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  2.3h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 10.7h HIGH   cluster_reroute(async_shard_fetch)                                  
111  1.9h HIGH   _add_listener_                                                      
jue ene 14 15:38:15 CET 2016
108  11h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 2.6h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109  11h HIGH   cluster_reroute(async_shard_fetch)                                  
111 2.2h HIGH   _add_listener_                                                      
jue ene 14 15:38:16 CET 2016
108 11.3h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  2.9h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 11.3h HIGH   cluster_reroute(async_shard_fetch)                                  
111  2.6h HIGH   _add_listener_                                                      
jue ene 14 15:38:17 CET 2016
108 11.6h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  3.2h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 11.6h HIGH   cluster_reroute(async_shard_fetch)                                  
111  2.9h HIGH   _add_listener_                                                      
jue ene 14 15:38:18 CET 2016
108 11.9h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  3.5h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 11.9h HIGH   cluster_reroute(async_shard_fetch)                                  
111  3.2h HIGH   _add_listener_                                                      
jue ene 14 15:38:19 CET 2016
108 12.2h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  3.9h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 12.2h HIGH   cluster_reroute(async_shard_fetch)                                  
111  3.5h HIGH   _add_listener_                                                      
jue ene 14 15:38:20 CET 2016
108 12.6h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  4.2h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 12.6h HIGH   cluster_reroute(async_shard_fetch)                                  
111  3.8h HIGH   _add_listener_                                                      
jue ene 14 15:38:21 CET 2016
108 12.9h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  4.5h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 12.9h HIGH   cluster_reroute(async_shard_fetch)                                  
111  4.1h HIGH   _add_listener_                                                      
jue ene 14 15:38:22 CET 2016
108 13.2h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  4.8h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 13.2h HIGH   cluster_reroute(async_shard_fetch)                                  
111  4.4h HIGH   _add_listener_                                                      
jue ene 14 15:38:24 CET 2016
108 13.5h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  5.1h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 13.5h HIGH   cluster_reroute(async_shard_fetch)                                  
111  4.7h HIGH   _add_listener_                                                      
jue ene 14 15:38:25 CET 2016
108 13.8h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  5.4h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 13.8h HIGH   cluster_reroute(async_shard_fetch)                                  
111    5h HIGH   _add_listener_                                                      
jue ene 14 15:38:26 CET 2016
108 14.1h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  5.7h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 14.1h HIGH   cluster_reroute(async_shard_fetch)                                  
111  5.4h HIGH   _add_listener_                                                      
jue ene 14 15:38:27 CET 2016
108 14.4h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110    6h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 14.4h HIGH   cluster_reroute(async_shard_fetch)                                  
111  5.7h HIGH   _add_listener_                                                      
jue ene 14 15:38:28 CET 2016
108 14.8h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  6.4h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 14.8h HIGH   cluster_reroute(async_shard_fetch)                                  
111    6h HIGH   _add_listener_                                                      
jue ene 14 15:38:29 CET 2016
108 15.1h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  6.7h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 15.1h HIGH   cluster_reroute(async_shard_fetch)                                  
111  6.3h HIGH   _add_listener_                                                      
jue ene 14 15:38:30 CET 2016
108 15.4h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110    7h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 15.4h HIGH   cluster_reroute(async_shard_fetch)                                  
111  6.6h HIGH   _add_listener_                                                      
jue ene 14 15:38:31 CET 2016
108 15.7h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  7.3h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 15.7h HIGH   cluster_reroute(async_shard_fetch)                                  
111  6.9h HIGH   _add_listener_                                                      
jue ene 14 15:38:33 CET 2016
108  16h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 7.6h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109  16h HIGH   cluster_reroute(async_shard_fetch)                                  
111 7.2h HIGH   _add_listener_                                                      
jue ene 14 15:38:34 CET 2016
108 16.3h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  7.9h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 16.3h HIGH   cluster_reroute(async_shard_fetch)                                  
111  7.5h HIGH   _add_listener_                                                      
jue ene 14 15:38:35 CET 2016
108 16.6h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  8.2h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 16.6h HIGH   cluster_reroute(async_shard_fetch)                                  
111  7.8h HIGH   _add_listener_                                                      
jue ene 14 15:38:36 CET 2016
108 16.9h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  8.5h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 16.9h HIGH   cluster_reroute(async_shard_fetch)                                  
111  8.2h HIGH   _add_listener_                                                      
jue ene 14 15:38:37 CET 2016
108 17.2h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  8.8h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 17.2h HIGH   cluster_reroute(async_shard_fetch)                                  
111  8.5h HIGH   _add_listener_                                                      
jue ene 14 15:38:38 CET 2016
108 17.5h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  9.1h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 17.5h HIGH   cluster_reroute(async_shard_fetch)                                  
111  8.8h HIGH   _add_listener_                                                      
jue ene 14 15:38:39 CET 2016
108 17.8h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  9.4h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 17.8h HIGH   cluster_reroute(async_shard_fetch)                                  
111  9.1h HIGH   _add_listener_                                                      
112    4m HIGH   _add_listener_                                                      
jue ene 14 15:38:40 CET 2016
108 18.1h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110  9.7h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 18.1h HIGH   cluster_reroute(async_shard_fetch)                                  
111  9.4h HIGH   _add_listener_                                                      
112 22.2m HIGH   _add_listener_                                                      
jue ene 14 15:38:41 CET 2016
108 18.4h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 10.1h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 18.4h HIGH   cluster_reroute(async_shard_fetch)                                  
111  9.7h HIGH   _add_listener_                                                      
112 40.4m HIGH   _add_listener_                                                      
jue ene 14 15:38:43 CET 2016
108 18.8h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 10.4h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 18.7h HIGH   cluster_reroute(async_shard_fetch)                                  
111   10h HIGH   _add_listener_                                                      
112 58.6m HIGH   _add_listener_                                                      
jue ene 14 15:38:44 CET 2016
108 19.1h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 10.7h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 19.1h HIGH   cluster_reroute(async_shard_fetch)                                  
111 10.3h HIGH   _add_listener_                                                      
112  1.2h HIGH   _add_listener_                                                      
jue ene 14 15:38:45 CET 2016
108 19.4h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110   11h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 19.4h HIGH   cluster_reroute(async_shard_fetch)                                  
111 10.6h HIGH   _add_listener_                                                      
112  1.6h HIGH   _add_listener_                                                      
jue ene 14 15:38:46 CET 2016
108 19.7h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 11.3h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 19.7h HIGH   cluster_reroute(async_shard_fetch)                                  
111 10.9h HIGH   _add_listener_                                                      
112  1.9h HIGH   _add_listener_                                                      
jue ene 14 15:38:47 CET 2016
108   20h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 11.6h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109   20h HIGH   cluster_reroute(async_shard_fetch)                                  
111 11.3h HIGH   _add_listener_                                                      
112  2.2h HIGH   _add_listener_                                                      
jue ene 14 15:38:48 CET 2016
108 20.3h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 11.9h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 20.3h HIGH   cluster_reroute(async_shard_fetch)                                  
111 11.6h HIGH   _add_listener_                                                      
112  2.5h HIGH   _add_listener_                                                      
jue ene 14 15:38:49 CET 2016
108 20.6h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 12.2h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 20.6h HIGH   cluster_reroute(async_shard_fetch)                                  
111 11.9h HIGH   _add_listener_                                                      
112  2.8h HIGH   _add_listener_                                                      
jue ene 14 15:38:50 CET 2016
108 20.9h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 12.5h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 20.9h HIGH   cluster_reroute(async_shard_fetch)                                  
111 12.2h HIGH   _add_listener_                                                      
112  3.1h HIGH   _add_listener_                                                      
jue ene 14 15:38:51 CET 2016
108 21.2h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 12.8h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 21.2h HIGH   cluster_reroute(async_shard_fetch)                                  
111 12.5h HIGH   _add_listener_                                                      
112  3.4h HIGH   _add_listener_                                                      
jue ene 14 15:38:53 CET 2016
108 21.5h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 13.1h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 21.5h HIGH   cluster_reroute(async_shard_fetch)                                  
111 12.8h HIGH   _add_listener_                                                      
112  3.7h HIGH   _add_listener_                                                      
jue ene 14 15:38:54 CET 2016
108 21.8h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 13.4h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 21.8h HIGH   cluster_reroute(async_shard_fetch)                                  
111 13.1h HIGH   _add_listener_                                                      
112    4h HIGH   _add_listener_                                                      
jue ene 14 15:38:55 CET 2016
108 22.1h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 13.7h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 22.1h HIGH   cluster_reroute(async_shard_fetch)                                  
111 13.4h HIGH   _add_listener_                                                      
112  4.3h HIGH   _add_listener_                                                      
jue ene 14 15:38:56 CET 2016
108 22.5h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 14.1h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 22.5h HIGH   cluster_reroute(async_shard_fetch)                                  
111 13.7h HIGH   _add_listener_                                                      
112  4.7h HIGH   _add_listener_                                                      
jue ene 14 15:38:57 CET 2016
108 22.8h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 14.4h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 22.8h HIGH   cluster_reroute(async_shard_fetch)                                  
111   14h HIGH   _add_listener_                                                      
112    5h HIGH   _add_listener_                                                      
jue ene 14 15:38:58 CET 2016
108 23.1h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 14.7h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 23.1h HIGH   cluster_reroute(async_shard_fetch)                                  
111 14.3h HIGH   _add_listener_                                                      
112  5.3h HIGH   _add_listener_                                                      
jue ene 14 15:38:59 CET 2016
108 23.4h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110   15h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 23.4h HIGH   cluster_reroute(async_shard_fetch)                                  
111 14.6h HIGH   _add_listener_                                                      
112  5.6h HIGH   _add_listener_                                                      
jue ene 14 15:39:00 CET 2016
108 23.7h URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 15.3h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109 23.7h HIGH   cluster_reroute(async_shard_fetch)                                  
111 14.9h HIGH   _add_listener_                                                      
112  5.9h HIGH   _add_listener_                                                      
jue ene 14 15:39:01 CET 2016
108    1d URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 15.6h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109    1d HIGH   cluster_reroute(async_shard_fetch)                                  
111 15.3h HIGH   _add_listener_                                                      
112  6.2h HIGH   _add_listener_                                                      
jue ene 14 15:39:03 CET 2016
108    1d URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 15.9h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109    1d HIGH   cluster_reroute(async_shard_fetch)                                  
111 15.6h HIGH   _add_listener_                                                      
112  6.5h HIGH   _add_listener_                                                      
jue ene 14 15:39:04 CET 2016
108    1d URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 16.2h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109    1d HIGH   cluster_reroute(async_shard_fetch)                                  
111 15.9h HIGH   _add_listener_                                                      
112  6.8h HIGH   _add_listener_                                                      
jue ene 14 15:39:05 CET 2016
108    1d URGENT delete-index [.shield_audit_log-2016.01.14]                         
110 16.5h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109    1d HIGH   cluster_reroute(async_shard_fetch)                                  
111 16.2h HIGH   _add_listener_                                                      
112  7.1h HIGH   _add_listener_                                                      
jue ene 14 15:39:06 CET 2016
108    1d URGENT delete-index [.shield_audit_log-2016.01.14]                         
113 13.8m URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109    1d HIGH   cluster_reroute(async_shard_fetch)                                  
111 16.5h HIGH   _add_listener_                                                      
112  7.4h HIGH   _add_listener_                                                      
jue ene 14 15:39:07 CET 2016
108    1d URGENT delete-index [.shield_audit_log-2016.01.14]                         
113 32.6m URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109    1d HIGH   cluster_reroute(async_shard_fetch)                                  
111 16.8h HIGH   _add_listener_                                                      
112  7.7h HIGH   _add_listener_                                                      
jue ene 14 15:39:08 CET 2016
108    1d URGENT delete-index [.shield_audit_log-2016.01.14]                         
113 50.8m URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109    1d HIGH   cluster_reroute(async_shard_fetch)                                  
111 17.1h HIGH   _add_listener_                                                      
112    8h HIGH   _add_listener_                                                      
jue ene 14 15:39:09 CET 2016
108    1d URGENT delete-index [.shield_audit_log-2016.01.14]                         
113  1.1h URGENT create-index [.shield_audit_log-2016.01.14], cause [auto(bulk api)] 
109    1d HIGH   cluster_reroute(async_shard_fetch)                                  
111 17.4h HIGH   _add_listener_                                                      
112  8.3h HIGH   _add_listener_                                                      
^C
abonuccelli@w530 ~ $ 

@nellicus nellicus changed the title from timeInQueue calculation for pending_tasks growing too fast? to timeInQueue for pending_tasks growing too fast? Jan 14, 2016

@clintongormley

This comment has been minimized.

Member

clintongormley commented Jan 14, 2016

Confirmed on 2.2.0-SNAPSHOT - within a few seconds it is reporting timeInQueue of several hours!

@jasontedor jasontedor self-assigned this Jan 14, 2016

@jasontedor

This comment has been minimized.

Member

jasontedor commented Jan 14, 2016

@clintongormley Should be dividing by 1000000 instead of 1000 (so the value being returned is microseconds, not milliseconds). Or even better, just use built-in methods to convert nanoseconds to milliseconds. I'll throw up a quick pull request.

@jasontedor

This comment has been minimized.

Member

jasontedor commented Jan 14, 2016

jasontedor added a commit that referenced this issue Jan 15, 2016

Merge pull request #15995 from jasontedor/prioritized-runnable-age
Fix calculation of age of pending tasks

Closes #15988
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment