Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Low Disk Watermark Logging Oddness #14194

Closed
skearns64 opened this issue Oct 19, 2015 · 3 comments

Comments

@skearns64
Copy link
Contributor

commented Oct 19, 2015

Using a 2.0.0-snapshot build, I see odd logging behavior for the low disk watermark.

The free-disk calculation is working fine, but the logging for this is quite odd. After starting a single-node cluster with 5 shards (each with 1 shard, and 1 unassigned replica), I see several log messages about low disk watermark all at once. 30 seconds later, I see another group of low disk watermark messages, but each time there are more messages than previously. First time was 4 identical log lines, then 7, then 10, then 13.

I was able to reproduce several times after restarting the nodes. I inserted newlines into the excerpted logs below. This could become a real problem if the trend continues.

[2015-10-19 09:21:12,202][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:21:12,202][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:21:12,203][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:21:12,203][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node


[2015-10-19 09:21:42,201][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:21:42,201][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:21:42,201][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:21:42,201][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:21:42,202][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:21:42,202][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:21:42,202][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node


[2015-10-19 09:22:12,206][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:12,208][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:12,208][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:12,208][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node


[2015-10-19 09:22:42,211][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,211][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,211][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:22:42,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node


[2015-10-19 09:23:12,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node


[2015-10-19 09:23:42,217][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,217][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,217][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,217][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,217][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
[2015-10-19 09:23:42,220][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node
@clintongormley

This comment has been minimized.

Copy link
Member

commented Oct 20, 2015

@dakrone could you take a look at this please?

@dakrone

This comment has been minimized.

Copy link
Member

commented Oct 20, 2015

@skearns64 When I try to reproduce this, I am getting only a single logging message (extra spacing added by me):

~/scratch/elasticsearch-2.0.0-SNAPSHOT λ bin/elasticsearch
[2015-10-20 06:33:54,293][INFO ][node                     ] [Aftershock] version[2.0.0-SNAPSHOT], pid[4608], build[e1a7cc2/2015-10-20T12:29:09Z]
[2015-10-20 06:33:54,293][INFO ][node                     ] [Aftershock] initializing ...
[2015-10-20 06:33:54,355][INFO ][plugins                  ] [Aftershock] loaded [], sites []
[2015-10-20 06:33:54,441][INFO ][env                      ] [Aftershock] using [1] data paths, mounts [[/home (/dev/mapper/fedora_thulcandra-home)]], net usable_space [270.9gb], net total_space [396.1gb], spins? [no], types [ext4]
[2015-10-20 06:33:55,850][INFO ][node                     ] [Aftershock] initialized
[2015-10-20 06:33:55,851][INFO ][node                     ] [Aftershock] starting ...
[2015-10-20 06:33:55,976][INFO ][transport                ] [Aftershock] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2015-10-20 06:33:55,985][INFO ][discovery                ] [Aftershock] elasticsearch/4jyUZlbuQJik6bNqb7bxuA
[2015-10-20 06:33:59,021][INFO ][cluster.service          ] [Aftershock] new_master {Aftershock}{4jyUZlbuQJik6bNqb7bxuA}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2015-10-20 06:33:59,052][INFO ][http                     ] [Aftershock] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2015-10-20 06:33:59,053][INFO ][node                     ] [Aftershock] started
[2015-10-20 06:33:59,072][INFO ][gateway                  ] [Aftershock] recovered [0] indices into cluster_state
[2015-10-20 06:34:02,803][INFO ][cluster.metadata         ] [Aftershock] [test] creating index, cause [api], templates [], shards [5]/[1], mappings []
[2015-10-20 06:34:29,033][INFO ][cluster.routing.allocation.decider] [Aftershock] low disk watermark [25%] exceeded on [4jyUZlbuQJik6bNqb7bxuA][Aftershock][/home/hinmanm/scratch/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 270.9gb[68.3%], replicas will not be assigned to this node

[2015-10-20 06:34:59,028][INFO ][cluster.routing.allocation.decider] [Aftershock] low disk watermark [25%] exceeded on [4jyUZlbuQJik6bNqb7bxuA][Aftershock][/home/hinmanm/scratch/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 270.9gb[68.3%], replicas will not be assigned to this node

[2015-10-20 06:35:29,028][INFO ][cluster.routing.allocation.decider] [Aftershock] low disk watermark [25%] exceeded on [4jyUZlbuQJik6bNqb7bxuA][Aftershock][/home/hinmanm/scratch/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 270.9gb[68.3%], replicas will not be assigned to this node

[2015-10-20 06:35:59,027][INFO ][cluster.routing.allocation.decider] [Aftershock] low disk watermark [25%] exceeded on [4jyUZlbuQJik6bNqb7bxuA][Aftershock][/home/hinmanm/scratch/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 270.9gb[68.3%], replicas will not be assigned to this node

You mentioned using a 2.0.0-snapshot build, what SHA are you building? I am building from the 2.0 branch, e1a7cc2.

jaymode added a commit to jaymode/elasticsearch that referenced this issue Oct 21, 2015
The ExtensionPoint.ClassSet binds adds the extension classes to a a Multibinder and binds
the classes and calls the asEagerSingleton method on the multibinder. This does not actually
create a singleton. Instead we first bind the class as a singleton and add then add the class
to the multibinder.

Closes elastic#14194
@dakrone

This comment has been minimized.

Copy link
Member

commented Oct 21, 2015

More info on this. The multiple logging lines is caused by DiskThresholdDecider not being bound as a singleton. Usually this wouldn't cause any side effects for Elasticsearch, however, Marvel injects the DTD into its NodesStatsCollecter with a Provider<...>, which was creating a new instance of DiskThresholdDecider every time it collected the stats. The additional instances were then logging, which is why the logging increased in number every time it ran.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.