Crashes on "too old resource version" messages

[szymonpk]

What happened:

From time to time we have the following message in the pod/kubelet logs:

W0504 17:03:59.230317    4644 reflector.go:289] object-"splunk"/"splunk-sandbox-splunk-kubernetes-logging": watch of *v1.ConfigMap ended with: too old resource version: 56153661 (56154652) 

It is crashing fluentd-hec:

#<Thread:0x00005614d0fd6f70@/usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/filter_kubernetes_metadata.rb:277 run> terminated with exception (report_on_exception is true):
/usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:43:in `rescue in set_up_namespace_thread': undefined method `<' for nil:NilClass (NoMethodError)
        from /usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:38:in `set_up_namespace_thread'
        from /usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/filter_kubernetes_metadata.rb:277:in `block in configure'
2020-05-04 14:37:51 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Could not parse jq filter: .record.source = \"/var/log/journal/\" + .record.source | .record.sourcetype = (.tag | ltrimstr(\"journald.\")) | .record.cluster_name = \"sandbox\" | .record.index = \"sandbox\" |.record, error: undefined method `<' for nil:NilClass"

It is related to the upstream issue with the kubernetes metadata plugin. It work when you downgrade the plugin to 2.4.1.

What you expected to happen:

Fluentd shouldn't crash.

How to reproduce it (as minimally and precisely as possible):

I am not sure, I think too old resource version messages are expected but I do not know how to force them to appear.

Anything else we need to know?:

It happens with fluentd-hec 1.2.x image.

Environment:

• Kubernetes version (use kubectl version): EKS 1.14
• Ruby version (use ruby --version): 2.5
• OS (e.g: cat /etc/os-release): Amazon Linux 2
• Splunk version: 8.x
• Others: n/a

[hvaghani221]

@szymonpk can you please test with latest SCK release (v1.4.10)?
We have upgraded lots of dependencies. If you are still facing the issue, please let us know.

[hvaghani221]

Closing the issue for now. If still facing the issue, feel free to reopen or crate a new issue