logstash netflow template broken for elasticsearch 6.0+

[xplorn]

Getting a lot of deprecated field errors trying to load the logstash netflow template. Begins with

elasticsearch.log-
[2017-11-17T09:47:17,034][DEBUG][o.e.a.a.i.t.p.TransportPutIndexTemplateAction] [LMhYXro] failed to put template [netflow
]
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping [default]: [include_in_all] is not allow
ed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on m
apping fields to create your own catch all field.

removing include_in_all fields gives-
[2017-11-17T09:59:46,885][DEBUG][o.e.a.a.i.t.p.TransportPutIndexTemplateAction] [LMhYXro] failed to put template [netflow]
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping [default]: Enabling [_all] is disabled in 6.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.

then getting it to load gives (elasticsearch_deprecation.log)-

[2017-11-16T11:02:41,957][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2017-11-16T11:24:10,728][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2017-11-16T11:30:28,972][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2017-11-16T14:23:03,886][WARN ][o.e.d.i.m.UidFieldMapper ] Fielddata access on the _uid field is deprecated, use _id instead
[2017-11-16T14:23:13,221][WARN ][o.e.d.i.m.UidFieldMapper ] Fielddata access on the _uid field is deprecated, use _id instead
[2017-11-17T09:38:29,039][WARN ][o.e.d.i.m.UidFieldMapper ] Fielddata access on the _uid field is deprecated, use _id instead
[2017-11-17T09:47:16,836][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2017-11-17T09:59:46,826][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2017-11-17T09:59:46,882][WARN ][o.e.d.i.m.AllFieldMapper ] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
[2017-11-17T10:01:29,260][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2017-11-17T10:01:29,291][WARN ][o.e.d.i.m.MapperService ] [default] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
[2017-11-17T10:05:38,520][WARN ][o.e.d.i.m.UidFieldMapper ] Fielddata access on the _uid field is deprecated, use _id instead

After more cleaning up, elasticsearch still isn't showing a netflow-* index. I suspect it's the last entry about _uid field but I've not written logstash templates and am more likely breaking than fixing at this point.

[robcowart]

You will notice that there is an es-6.0.0-pre branch. I will be completing a new release for 6.0.0 now that it is GA. In the meantime the index template from the es-6.0.0-pre branch should work.

[dhermans]

hi -thanks for developing this tool - looks amazing.

i tried your es-6.0.0-pre branch ( had other issues trying to get 5.x working ) and attempted to push some captures into logstash..

i'm not sure i'm pushing the captures in correctly, is
cat my.pcap | nc -4 -u 10.1.1.1 2055
enough?

none get indexed - i just get ( i think i got same result with live data ):

[2017-12-02T13:24:43,194][WARN ][logstash.codecs.netflow ] Ignoring Netflow version v0
[2017-12-02T13:24:43,194][WARN ][logstash.codecs.netflow ] Ignoring Netflow version v54467
[2017-12-02T13:24:43,195][WARN ][logstash.codecs.netflow ] Ignoring Netflow version v1780
[2017-12-02T13:24:43,196][WARN ][logstash.codecs.netflow ] Ignoring Netflow version v36618
[2017-12-02T13:24:43,196][WARN ][logstash.codecs.netflow ] Ignoring Netflow version v28416
[2017-12-02T13:24:43,197][WARN ][logstash.codecs.netflow ] Ignoring Netflow version v34054
[2017-12-02T13:24:43,197][WARN ][logstash.codecs.netflow ] Ignoring Netflow version v1860
[2017-12-02T13:24:43,197][WARN ][logstash.codecs.netflow ] Ignoring Netflow version v42071
[2017-12-02T13:24:43,198][WARN ][logstash.codecs.netflow ] Ignoring Netflow version v36618

every now and then:
[2017-12-02T13:24:43,343][WARN ][logstash.codecs.netflow ] Invalid netflow packet received (value '56330' not as expected for obj.flow_records)

[2017-12-02T13:24:43,632][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 257 from source id 1, because no template to decode it with has been received. This message will usually go away after 1 minute.

obviously no indexes are created...

any help appreciated...

[robcowart]

Modified index template for compatibility with Elasticsearch versions 5.4.x thru 6.1.x. 638f958