1.4 disk allocation decider logging and error handling

[ppearcy]

Moved to 1.4, all looked well, but things started failing in our docker CI environment during a call to bootstrap the cluster to create some search templates.

Basically, would just see the following error flow:

application_1   | [info] application - Creating search template: testTemplate, file: test.json, type: search
elasticsearch_1 | [2014-11-14 02:53:55,763][INFO ][cluster.metadata         ] [Visimajoris] [.scripts] creating index, cause [auto(index api)], shards [1]/[0], mappings []
application_1   | [.scripts][0] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: index {[.scripts][mustache][testTemplate], source[{

This was a huge mystery until the logging on the search cluster was bumped up to DEBUG:

elasticsearch_1 | [2014-11-14 19:27:33,523][INFO ][cluster.metadata         ] [XXX] [.scripts] creating index, cause [auto(index api)], shards [1]/[0], mappings []
elasticsearch_1 | [2014-11-14 19:27:33,539][DEBUG][cluster.routing.allocation.decider] [XXX] Node [IdXCumBrSYOTpgesR2QQ3Q] has 9.009641449850488% free disk
elasticsearch_1 | [2014-11-14 19:27:33,539][DEBUG][cluster.routing.allocation.decider] [XXX] Less than the required 10.0 free bytes threshold (9.009641449850488 bytes free) on node IdXCumBrSYOTpgesR2QQ3Q, preventing allocation even though primary has never been allocated
elasticsearch_1 | [2014-11-14 19:27:33,542][DEBUG][cluster.routing.allocation.decider] [XXX] Node [IdXCumBrSYOTpgesR2QQ3Q] has 9.009641449850488% free disk
elasticsearch_1 | [2014-11-14 19:27:33,542][DEBUG][cluster.routing.allocation.decider] [XXX] Less than the required 10.0 free bytes threshold (9.009641449850488 bytes free) on node IdXCumBrSYOTpgesR2QQ3Q, preventing allocation even though primary has never been allocated
elasticsearch_1 | [2014-11-14 19:27:33,543][DEBUG][indices                  ] [XXX] [.scripts] closing ... (reason [failed to create index])

I think there are a few issues going on here:

• The current log message is confusing since one line says 9.009641449850488% free disk and next line says 10.0 free bytes threshold (9.009641449850488 bytes free)
• This message is only logged at DEBUG level. If you can't create an index because of low disk, that is an error.
• It would be nice if the error returned to the client indicated the reason the index was unassigned.

[dakrone]

Hi @ppearcy, we've addressed this in #8270 and #8382, which should be part of 1.4.1

[ppearcy]

Ah, great thanks! Will close out and open a new one if there are still issues.