Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High disk watermark exceeded on one or more nodes, rerouting shards #15919

Closed
najamss opened this issue Jan 12, 2016 · 8 comments
Closed

High disk watermark exceeded on one or more nodes, rerouting shards #15919

najamss opened this issue Jan 12, 2016 · 8 comments
Labels
discuss :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes)

Comments

@najamss
Copy link

najamss commented Jan 12, 2016

I am running Elasticsearch, and Kibana on Windows and using Synology NAS as storage for Elasticsearch. For few days, Elasticsearch started behaving weird; therefore, I checked elasticsearch.log and found the following errors:

[WARN ][cluster.routing.allocation.decider] [Desmond Pitt] high disk watermark [0b] exceeded on [O2-Ef7fET9S_MJNAL-q_yA][Desmond Pitt] free: -1b[100%], shards will be relocated away from this node
[WARN ][cluster.routing.allocation.decider] [Desmond Pitt] high disk watermark [0b] exceeded on [O2-Ef7fET9S_MJNAL-q_yA][Desmond Pitt] free: -1b[100%], shards will be relocated away from this node

[INFO ][cluster.routing.allocation.decider] [Desmond Pitt] high disk watermark exceeded on one or more nodes, rerouting shards
DEBUG][action.bulk ] [Desmond Pitt] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
DEBUG][action.bulk ] [Desmond Pitt] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]

I have already run this command GET _node/stats and found the following issue:

fs":{"timestamp":1452402078885,"total":{},"data":[{"path":"\abc-synology1.xyz.edu\syslog\elasticsearch\nodes\0"}]}

Which I guess, shows the file system fails to report disk usage, which confuses the high water mark check.

If I am right, how can I resolve it or what else could be the issue?

By the way, I am referred here by Elastic Search forums: https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/8
@bleskes
Copy link
Contributor

bleskes commented Jan 12, 2016

Thanks @najamss . I think the question here is what we want to do when a node doesn't report the file disk usage.

  • Do we see it as a requirement? In which case, should we test at node start and refuse to start if the stats are not there? Also, if a node doesn't report usage during operation, do we relocate all shards away as we seem to be doing now (but with better logging)?
  • Do we accept this as acceptable hickups ? if so, we should adapt the disk threshold allocator to accept this.

@dakrone thoughts?

@dakrone
Copy link
Member

dakrone commented Jan 12, 2016

Do we see it as a requirement? In which case, should we test at node start and
refuse to start if the stats are not there?

I don't think we should view it as a hard requirement, I think it should
probably warn about it and self-disable the AllocationDecider if there are no
disk usages reported, what do you think?

Also, if a node doesn't report usage during operation, do we relocate all
shards away as we seem to be doing now (but with better logging)?

Are you talking about if it previously reported the disk statistics but suddenly
stopped?

Regardless, if a node doesn't report usage we should not relocate all shards
away. In that case we should warn heavily that a user is entering unknown
territory and we cannot protect them against disk getting full.

Do we accept this as acceptable hickups ? if so, we should adapt the disk
threshold allocator to accept this.

I don't think we should try to plan for a case where disk stats are reported
correctly, and the magically fail to report (I think that should be handled
better by the OS!). It sounds like here that this is an all-or-nothing case, we
either have the fs numbers or we don't at all.

I'm open to suggestions though, what do you think @bleskes?

@najamss
Copy link
Author

najamss commented Jan 13, 2016

Hello Folks,

Honestly speaking I am not getting either of y'all. I am a beginner to ELK stack and found that maybe it's easy to deploy but very difficult to maintain, it requires a full time employee to manage it which is true for any other commercial product. Solution was running fine before but suddenly it started behaving like this. Now it's throwing weird errors like:

][cluster.action.shard ] [Aireo] [logstash-2016.01.08][1] received shard failed for [logstash-2016.01.08][1], node[79PYnn4uROy7KOZjakxDrQ], [P], s[INITIALIZING], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-01-13T05:23:28.862Z], details[shard failure [failed recovery][IndexShardGatewayRecoveryException[[logstash-2016.01.08][1] failed recovery]; nested: ElasticsearchException[failed to obtain write log pre translog recovery]; nested: LockObtainFailedException[Lock obtain timed out: NativeFSLock@\synology1.abc.xyz.edu\syslog\elasticsearch\nodes\0\indices\logstash-2016.01.08\1\index\write.lock]; ]]], indexUUID [sI1jLxDERgmVtx2aqGhT3w], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[logstash-2016.01.08][1] failed recovery]; nested: ElasticsearchException[failed to obtain write log pre translog recovery]; nested: LockObtainFailedException[Lock obtain timed out: NativeFSLock@\synology1.abc.xyz.edu\syslog\elasticsearch\nodes\0\indices\logstash-2016.01.08\1\index\write.lock]; ]]

@bleskes
Copy link
Contributor

bleskes commented Jan 14, 2016

@najamss your issue is different - something prevents the node from acquiring the write lock (something is using that folder, or the NAS is acting up in more ways than just failing to report stats) if you didn't solve the issue by now can you open a new issue so we can discuss it there. Please also let us know what version of ES you are using and what happened before.

@dakrone re

Regardless, if a node doesn't report usage we should not relocate all shards
away. In that case we should warn heavily that a user is entering unknown
territory and we cannot protect them against disk getting full.

Agreed. That is my instinct as well.

It sounds like here that this is an all-or-nothing case, we
either have the fs numbers or we don't at all.

I'm not sure. @najamss reports it used to be OK and suddenly stopped.

In any case this:

[WARN ][cluster.routing.allocation.decider] [Desmond Pitt] high disk watermark [0b] exceeded on [O2-Ef7fET9S_MJNAL-q_yA][Desmond Pitt] free: -1b[100%], shards will be relocated away from this node

feels like the wrong response to missing FS stats from a node: "fs":{"timestamp":1452402078885,"total":{},"data":[{"path":"\abc-synology1.xyz.edu\syslog\elasticsearch\nodes\0"}]}

@dakrone
Copy link
Member

dakrone commented Jan 14, 2016

In any case this:

[WARN ][cluster.routing.allocation.decider] [Desmond Pitt] high disk watermark [0b] exceeded on [O2-Ef7fET9S_MJNAL-q_yA][Desmond Pitt] free: -1b[100%], shards will be relocated away from this node

feels like the wrong response to missing FS stats from a node:
"fs":{"timestamp":1452402078885,"total":{},"data":[{"path":"\abc-synology1.xyz.edu\syslog\elasticsearch\nodes\0"}]}

So part of the problem is that FsInfo.Path defines these:

long total = -1;
long free = -1;
long available = -1;

However, because filesystems are crazy, some of these are actually valid values!
I have seen systems with filesystems that report negative amounts for the
free/available values (I don't think I've seen them for the total, so that's
something we can check).

I think something we can do is check at least the total < 0 and just ignore
updating the stats in the event that it's a transient error.

dakrone added a commit to dakrone/elasticsearch that referenced this issue Jan 14, 2016
@dakrone
Skip capturing least/most FS info for an FS with no total
927e61d 
If an operating system reports -1 for the total bytes of a filesystem
path, we should ignore it when capturing the least and most available
statistics.

Relates to elastic#15919
@dakrone dakrone mentioned this issue Jan 14, 2016
Closed
dakrone added a commit that referenced this issue Jan 14, 2016
@dakrone
Skip capturing least/most FS info for an FS with no total
03087cf 
If an operating system reports -1 for the total bytes of a filesystem
path, we should ignore it when capturing the least and most available
statistics.

Relates to #15919

Squashed commit of the following:

commit 5d2258f
Author: Lee Hinman <lee@writequit.org>
Date:   Thu Jan 14 14:14:27 2016 -0700

    Change test to test positive total with negative 'free' value

commit 927e61d
Author: Lee Hinman <lee@writequit.org>
Date:   Thu Jan 14 13:09:28 2016 -0700

    Skip capturing least/most FS info for an FS with no total

    If an operating system reports -1 for the total bytes of a filesystem
    path, we should ignore it when capturing the least and most available
    statistics.

    Relates to #15919
dakrone added a commit that referenced this issue Jan 14, 2016
@dakrone
Skip capturing least/most FS info for an FS with no total
8384f73 
If an operating system reports -1 for the total bytes of a filesystem
path, we should ignore it when capturing the least and most available
statistics.

Relates to #15919

Squashed commit of the following:

commit 5d2258f
Author: Lee Hinman <lee@writequit.org>
Date:   Thu Jan 14 14:14:27 2016 -0700

    Change test to test positive total with negative 'free' value

commit 927e61d
Author: Lee Hinman <lee@writequit.org>
Date:   Thu Jan 14 13:09:28 2016 -0700

    Skip capturing least/most FS info for an FS with no total

    If an operating system reports -1 for the total bytes of a filesystem
    path, we should ignore it when capturing the least and most available
    statistics.

    Relates to #15919
@bleskes
Copy link
Contributor

bleskes commented Jan 17, 2016

Thanks lee. Does #16001 mean we can close this?

@dakrone
Copy link
Member

dakrone commented Jan 17, 2016

@bleskes yeah I think so, closing this

@dakrone dakrone closed this as completed Jan 17, 2016
@najamss najamss mentioned this issue Jan 19, 2016
Closed
@SalahAdDin
Copy link

SalahAdDin commented Sep 21, 2016 •
edited

I have the same problem with the official docker image:

elasticsearch_1  | [2016-09-20 05:09:32,905][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:09:32,905][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:10:02,905][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:10:32,905][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:10:32,905][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:11:02,906][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:11:32,906][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:11:32,906][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:12:02,907][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:12:32,907][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:12:32,907][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:13:02,908][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:13:32,908][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:13:32,908][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
web_1            | Performing system checks...
web_1            | 
web_1            | System check identified no issues (0 silenced).
web_1            | September 19, 2016 - 21:39:09
web_1            | Django version 1.10.1, using settings '''
web_1            | Starting development server at http://0.0.0.0:8000/
web_1            | Quit the server with CONTROL-C.
elasticsearch_1  | [2016-09-20 05:14:02,908][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:14:32,909][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:14:32,909][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:15:02,909][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:15:32,910][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:15:32,910][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:16:02,910][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:16:32,911][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:16:32,911][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:17:02,911][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:17:32,912][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:17:32,912][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:18:03,036][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:18:33,037][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:18:33,037][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:19:03,037][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:19:33,038][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:19:33,038][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:20:03,038][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:20:33,038][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:20:33,038][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:21:03,039][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:21:33,039][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:21:33,039][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:22:03,040][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:22:33,040][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:22:33,040][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:23:03,040][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:23:33,041][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:23:33,041][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:24:03,041][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:24:33,042][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:24:33,042][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:25:03,042][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:25:33,043][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:25:33,043][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:26:03,043][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:26:33,043][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:26:33,043][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:27:03,044][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:27:33,044][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:27:33,044][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:28:03,045][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:28:33,045][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:28:33,045][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:29:03,046][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:29:33,046][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:29:33,046][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:30:03,046][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:30:33,047][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:30:33,047][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:31:03,047][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:31:33,048][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:31:33,048][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:32:03,048][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:32:33,049][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:32:33,049][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:33:03,049][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:33:33,049][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:33:33,049][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:34:03,050][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:34:33,050][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:34:33,050][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:35:03,051][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:35:33,051][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:35:33,051][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:36:03,052][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:36:33,052][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:36:33,052][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:37:03,053][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:37:33,053][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:37:33,053][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:38:03,053][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:38:33,054][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:38:33,054][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:39:03,054][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:39:33,055][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:39:33,055][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:40:03,055][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:40:33,056][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:40:33,056][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:41:03,056][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:41:33,056][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:41:33,056][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:42:03,057][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:42:33,057][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:42:33,057][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:43:03,058][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:43:33,058][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:43:33,058][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:44:03,058][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:44:33,059][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:44:33,059][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:45:03,059][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:45:33,060][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:45:33,060][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:46:03,060][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:46:33,061][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:46:33,061][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:47:03,061][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:47:33,062][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:47:33,062][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:48:03,062][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:48:33,063][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:48:33,063][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:49:03,063][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:49:33,063][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:49:33,063][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:50:03,064][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:50:33,064][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:50:33,064][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:51:03,065][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:51:33,065][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:51:33,065][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:52:03,065][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:52:33,066][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:52:33,066][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:53:03,066][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:53:33,067][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:53:33,067][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:54:03,067][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:54:33,067][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:54:33,068][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:55:03,068][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:55:33,068][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:55:33,068][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:56:03,069][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:56:33,069][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:56:33,069][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:57:03,070][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:57:33,070][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:57:33,070][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:58:03,070][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:58:33,071][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:58:33,071][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 05:59:03,071][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:59:33,072][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 05:59:33,072][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:00:03,072][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:00:33,072][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:00:33,072][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:01:03,073][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:01:33,073][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:01:33,073][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:02:03,074][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:02:33,074][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:02:33,074][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:03:03,075][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:03:33,075][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:03:33,075][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:04:03,076][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:04:33,076][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:04:33,076][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:05:03,076][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:05:33,077][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:05:33,077][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:06:03,077][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:06:33,078][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:06:33,078][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:07:03,078][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:07:33,078][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:07:33,078][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:08:03,079][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:08:33,079][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:08:33,079][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:09:03,080][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:09:33,080][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:09:33,080][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:10:03,080][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:10:33,081][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:10:33,081][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:11:03,081][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:11:33,082][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:11:33,082][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:12:03,082][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:12:33,082][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:12:33,083][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:13:03,083][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:13:33,083][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:13:33,083][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:14:03,084][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:14:33,084][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:14:33,084][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:15:03,085][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:15:33,085][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:15:33,085][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:16:03,085][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:16:33,086][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:16:33,086][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:17:03,086][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:17:33,087][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:17:33,087][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:18:03,087][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:18:33,088][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:18:33,088][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:19:03,088][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:19:33,088][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:19:33,088][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:20:03,089][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:20:33,089][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:20:33,089][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:21:03,090][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:21:33,090][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:21:33,090][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:22:03,090][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:22:33,091][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:22:33,091][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:23:03,091][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:23:33,092][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:23:33,092][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:24:03,092][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:24:33,092][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:24:33,092][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:25:03,093][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:25:33,093][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:25:33,093][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:26:03,094][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:26:33,094][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:26:33,094][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:27:03,094][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:27:33,095][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:27:33,095][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:28:03,095][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:28:33,096][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:28:33,096][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:29:03,096][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:29:33,096][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:29:33,097][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:30:03,097][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:30:33,097][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:30:33,097][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:31:03,098][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:31:33,098][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:31:33,098][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:32:03,099][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:32:33,099][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:32:33,099][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:33:03,100][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:33:33,100][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:33:33,100][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:34:03,101][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:34:33,101][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:34:33,101][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:35:03,102][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:35:33,102][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:35:33,102][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:36:03,102][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:36:33,103][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:36:33,103][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:37:03,103][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:37:33,104][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:37:33,104][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:38:03,104][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:38:33,104][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:38:33,105][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:39:03,105][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:39:33,105][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:39:33,105][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:40:03,106][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:40:33,106][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:40:33,106][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:41:03,137][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:41:33,137][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:41:33,137][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:42:03,138][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:42:33,138][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:42:33,138][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:43:03,139][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:43:33,139][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:43:33,139][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:44:03,139][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:44:33,244][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:44:33,244][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:45:03,245][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:45:33,245][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:45:33,245][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:46:03,246][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:46:33,246][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:46:33,246][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:47:03,247][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:47:33,247][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:47:33,247][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:48:03,247][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:48:33,248][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:48:33,248][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:49:03,248][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:49:33,249][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:49:33,249][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:50:03,249][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:50:33,250][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:50:33,250][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:51:03,251][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:51:33,251][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:51:33,251][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:52:03,252][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:52:33,252][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:52:33,252][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:53:03,253][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:53:33,253][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:53:33,253][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:54:03,253][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:54:33,254][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:54:33,254][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:55:03,254][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:55:33,255][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:55:33,255][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:56:03,255][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:56:33,255][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:56:33,255][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:57:03,256][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:57:33,256][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:57:33,256][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:58:03,257][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:58:33,257][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:58:33,257][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 06:59:03,258][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:59:33,258][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 06:59:33,258][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 07:00:03,258][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 07:00:33,259][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 07:00:33,259][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 07:01:03,259][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 07:01:33,260][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 07:01:33,260][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 07:02:03,260][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 07:02:33,260][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 07:02:33,260][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 22:30:09,169][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 22:30:39,170][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node
elasticsearch_1  | [2016-09-20 22:30:39,170][INFO ][cluster.routing.allocation.decider] [Carlos Lobo] rerouting shards: [high disk watermark exceeded on one or more nodes]
elasticsearch_1  | [2016-09-20 22:31:09,170][WARN ][cluster.routing.allocation.decider] [Carlos Lobo] high disk watermark [90%] exceeded on [v1-xs1R8TbaUh3RKm1EznQ][Carlos Lobo][/usr/share/elasticsearch/data/elasticsearch/nodes/0] free: 4.7gb[5%], shards will be relocated away from this node

@lcawl lcawl added :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. and removed :Allocation labels Feb 13, 2018
@clintongormley clintongormley added :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) and removed :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. labels Feb 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
discuss :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@dakrone @clintongormley @bleskes @SalahAdDin @najamss @lcawl