Fluentd keeps crashing on GKE 1.16

[lechen26]

looks like related to version 2.4.5 of fluent-plugin-kubernetes_metadata_filter

the error is as follows:

#<Thread:0x0000000002e2bc40@/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/filter_kubernetes_metadata.rb:277 run> terminated with exception (report_on_exception is true):
/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:43:in `rescue in set_up_namespace_thread': undefined method `<' for nil:NilClass (NoMethodError)
        from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:38:in `set_up_namespace_thread'
        from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/filter_kubernetes_metadata.rb:277:in `block in configure'
/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:127:in `block in process_namespace_watcher_notices': Error while watching namespaces: too old resource version: 73764561 (73772817) (RuntimeError)
        from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/watch_stream.rb:28:in `block in each'
        from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/http-4.4.0/lib/http/response/body.rb:37:in `each'
        from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/watch_stream.rb:25:in `each'
        from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:107:in `process_namespace_watcher_notices'
        from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `set_up_namespace_thread'
        from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/filter_kubernetes_metadata.rb:277:in `block in configure'
2020-06-23 12:03:17 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `<' for nil:NilClass"
  2020-06-23 12:03:17 +0000 [error]: #0 /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:43:in `rescue in set_up_namespace_thread'
  2020-06-23 12:03:17 +0000 [error]: #0 /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:38:in `set_up_namespace_thread'
  2020-06-23 12:03:17 +0000 [error]: #0 /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/filter_kubernetes_metadata.rb:277:in `block in configure'
2020-06-23 12:03:18 +0000 [info]: Worker 0 finished unexpectedly with status 1

its seems as a known issue here fabric8io/fluent-plugin-kubernetes_metadata_filter#226 and they have reverted to release 2.4.1

thanks

[igorpeshansky]

This was fixed in release 1.6.36, but there's no ETA for when it'll be rolled out to GKE.