ilm_enabled: auto does not work on custom index values

[tomsommer]

The documentation does not mention that ilm_enabled: auto does not work if a custom index-value is set, this will confuse a lot of people, since this setting is pretty standard.

logstash-output-elasticsearch/lib/logstash/outputs/elasticsearch/ilm.rb

Line 19 in c6872b9

default_index?(@index) || !default_rollover_alias?(@ilm_rollover_alias)

It would be nice if ilm_enabled: auto supported custom index names

[delfuego]

I'm not sure if what I was seeing today was related to this, but might've been — I spent all day chasing down why I was seeing a forever-growing logstash index that wasn't under ILM management, and why the index template didn't reflect any ILM settings (index.lifecycle.name and index.lifecycle.rollover_alias). Of note, though, my elasticsearch output config explicitly had ilm_enabled => true (not auto). But it also explicitly had index => "logstash" set as well... and with this config, at startup I would see a true index named logstash get created, without any ILM management applied to it.

Only after I deleted the logstash index and removed the index => "logstash" entry in my elasticsearch output config did I see ILM truly get enabled — at logstash startup, instead of creating a logstash index, it created a logstash-2019.06.19-00001 index and created the logstash alias, and the ILM API shows that the logstash-2019.06.19-00001 index is indeed now under ILM management.

Of note, during previous testing yesterday, if my elasticsearch output config had index => "log-ilm" (rather than index => "logstash"), everything worked as expected -- so if this is a bug, it's with having index => "logstash" rather than having any index set.

[delfuego]

Oh, I'm running ES 7.1.1 (on a basic license) and Logstash 7.1.1.

[delfuego]

Confirmed the above; this feels like a bug (or a series of bugs), and not just a documentation issue.

If I start up a new, clean Elasticsearch 7.1.1 docker container, and then start up a new, clean Logstash 7.1.1 container with the following pipeline config:

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
		index => "logstash"
	}
}

then Logstash starts up, creates an index template policy logstash as follows (showing only settings for brevity) where there aren't any ILM-related settings, and creates an actual ES index named logstash:

{
	"logstash": {
		"order": 0,
		"version": 60001,
		"index_patterns": ["logstash-*"],
		"settings": {
			"index": {
				"number_of_shards": "1",
				"refresh_interval": "5s"
			}
		}
}

If I start up another new, clean ES container and another new, clean Logstash container omitting the index => "logstash" line from the pipeline config:

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
	}
}

then Logstash starts up, creates an index template policy logstash as follows, creates an index logstash-2019.06.19-000001, and creates an alias logstash pointing to it:

{
	"logstash": {
		"order": 0,
		"version": 60001,
		"index_patterns": ["logstash-*"],
		"settings": {
			"index": {
				"lifecycle": {
					"name": "logstash-policy",
					"rollover_alias": "logstash"
				},
				"number_of_shards": "1",
				"refresh_interval": "5s"
			}
		}
}

Three other final notes:

1. If I specify a non-logstash index name in the ES output pipeline config (e.g., index => "testing"), then ILM still doesn't get configured properly — at startup, Logstash creates an index template policy that doesn't contain the ILM settings, and creates a real index with the name that I specify.
2. BUT if I also specify the ilm_enabled, ilm_pattern, and ilm_rollover_alias options (e.g., ilm_enabled => true, ilm_pattern => ""{now/d}-000001", and ilm_rollover_alias => "testing"), then ILM does get configured correctly... at startup, Logstash creates an index template policy that contains the ILM settings, creates an index with the proper pattern name, and creates an alias to it.
3. BUT if I include these three options but still explicitly call my index logstash — index => "logstash", ilm_enabled => true, ilm_pattern => ""{now/d}-000001", and ilm_rollover_alias => "logstash" — then at startup, Logstash reverts to the no-ILM-configured setup.

So, I'd say there's a few bugs in here.

[serkodev]

@delfuego
The three final notes is nice, specially for the point 2 and it fix my problem, thanks a lot!
I have been waste about half day on solving this.

[stefnestor]

For paper-trail, confirming this doc disagrees on outcome compared to above point#3

If the index property is supplied in the output definition, it will be overwritten by the rollover alias.