Logstsah reporting "IPv6 address must be 16 bytes"

[nin9s]

from time to time getting a logstash crash saying:

[2018-02-25T18:21:36,851][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: IPv6 address must be 16 bytes>, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:2502:in `initialize'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:2242:in `decode_rdata'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1649:in `block in get_rr'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1547:in `get_length16'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1649:in `get_rr'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1524:in `block in decode'", "org/jruby/RubyRange.java:485:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1523:in `block in decode'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1536:in `initialize'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1500:in `decode'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:710:in `request'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:536:in `block in fetch_resource'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1108:in `block in resolv'", "org/jruby/RubyArray.java:1734:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1106:in `block in resolv'", "org/jruby/RubyArray.java:1734:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1105:in `block in resolv'", "org/jruby/RubyArray.java:1734:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1103:in `resolv'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:527:in `fetch_resource'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:517:in `each_resource'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:471:in `each_name'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:154:in `block in each_name'", "org/jruby/RubyArray.java:1734:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:153:in `each_name'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:135:in `getname'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.7/lib/logstash/filters/dns.rb:279:in `getname'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.7/lib/logstash/filters/dns.rb:266:in `block in retriable_getname'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.7/lib/logstash/filters/dns.rb:251:in `block in retriable_request'", "org/jruby/ext/timeout/Timeout.java:117:in `timeout'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.7/lib/logstash/filters/dns.rb:250:in `retriable_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.7/lib/logstash/filters/dns.rb:265:in `retriable_getname'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.7/lib/logstash/filters/dns.rb:211:in `block in reverse'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.7/lib/logstash/filters/dns.rb:183:in `reverse'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.7/lib/logstash/filters/dns.rb:99:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `block in multi_filter'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47:in `multi_filter'", "(eval):316506:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):316503:in `block in initialize'", "(eval):316521:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):316517:in `block in initialize'", "(eval):13125:in `block in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:447:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385:in `block in start_workers'"]}

Unfortunately I can't have a clue where to start searching except the "message saying: <ArgumentError: IPv6 address must be 16 bytes>

Thanks

[robcowart]

It looks like the DNS filter is complaining about the address being passed to it. Can you look at a packet capture of the flows in question to make sure the address is correctly decoded. You need to find out if the flow is being sent correctly and the codec is at fault, or whether the flow itself is malformed.

[nin9s]

I could if I would have any clue about what exact flow logstash is complaining about ...
Can I narrow down this somehow?

[nin9s]

I've managed to find out that it must have somthing to do with the IPFIX field "IPv6 flow label" beeing sent by my miktotik router. As soon as I disable this filed logstash isnt crashing anymore.

What exactly is that filed for? Do I need it and more importantly how can I see the content of the field during the crash?

[robcowart]

IPv6 Flow Labels are related to a feature of IPv6 which enables a flow to request specific handling by the network devices carrying the traffic. There is no where in ElastiFlow where I do any specific handling of flow labels, so I am wondering if disabling this field is also causing other fields to not be included in the flow.

[nin9s]

sorry for reopening the issue - just for clarification. I've managed to find out that the crash is related to a malfunctioning dnssec server. As soon as I changed the external DNS the crashes never happened again.

Just wanted to let you know the problem was NOT related to elastiflow in any kind.

issue can be closed again :)

[robcowart]

Thanks for the follow up. Much appreciated!