worker-0 is getting SIGKILLed instead of nicely reloading upon issuing gracefulReload

[alex-vmw]

Describe the bug
We were performing tests that were issuing multiple /api/config.gracefulReload calls to fluentd. Sometimes after issuing a /api/config.gracefulReload we see that worker-0 is getting SIGKILLed instead of nicely reloading.

When the issue happens we see the below message in the logs:

2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: Worker 0 finished unexpectedly with signal SIGKILL

When the issue does NOT happen we see the below message in the logs instead:

2021-04-21 02:29:03 +0000 [info]: #0 fluent/log.rb:329:info: restart fluentd worker worker=0

To Reproduce
Reload fluentd config multiple times via /api/config.gracefulReload call and look for below message in the log:

2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: Worker 0 finished unexpectedly with signal SIGKILL

Expected behavior
Fluentd worker-0 should always nicely reload when /api/config.gracefulReload is issued.

Your Environment
Fluentd or td-agent version: fluentd 1.12.2 we also tested with 1.9.1 and 1.11.2 and saw same issue in those as well.
Operating system: Ubuntu 20.04.1 LTS
Kernel version: 5.4.0-42-generic

Your Configuration

<ROOT>
  <system>
    log_level info
    rpc_endpoint "127.0.0.1:24444"
  </system>
  <source>
    @type systemd
    @id in_systemd_docker
    path "/var/log/journal"
    tag "systemd.unit"
    read_from_head false
    <storage>
      @type "local"
      persistent true
      path "/var/log/-kube-fluentd-operator--fluentd-journald-cursor.json"
    </storage>
    <entry>
      field_map {"_SYSTEMD_UNIT":"unit","MESSAGE":"log","_PID":["pid"],"_PRIORITY":"priority","_COMM":"cmd","_HOSTNAME":"hostname"}
      field_map_strict true
      fields_lowercase true
    </entry>
  </source>
  <match systemd.unit>
    @type rewrite_tag_filter
    <rule>
      key "unit"
      pattern ^(.+)$
      tag "systemd.$1"
    </rule>
  </match>
  <filter systemd.kubelet.service>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/*.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-containers.log.pos"
    tag "kubernetes.*"
    read_from_head true
    <parse>
      @type "multiline"
      format1 /^(?<partials>([^\n]+ (stdout|stderr) P [^\n]+\n)*)/
      format2 /(?<time>[^\n]+) (?<stream>stdout|stderr) F (?<log>[^\n]*)/
      format3 /|(?<json>{.*})/
      time_format "%Y-%m-%dT%H:%M:%S.%N%:z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type record_transformer
    @id filter_crio_container_logs
    enable_ruby true
    remove_keys partials
    <record>
      log ${record["partials"]&.gsub(/.+ (stdout|stderr) P (.+)\n/, '\\2')}${record["log"]}
    </record>
  </filter>
  <filter kubernetes.**>
    @type parser
    @id filter_docker_container_logs
    key_name "json"
    remove_key_name_field true
    reserve_data true
    emit_invalid_record_to_error false
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      time_type string
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_startupscript
    path "/var/log/startupscript.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-startupscript.log.pos"
    tag "startupscript"
    <parse>
      @type "syslog"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_docker
    path "/var/log/docker.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-docker.log.pos"
    tag "docker"
    <parse>
      @type "regexp"
      expression /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_etcd
    path "/var/log/etcd.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-etcd.log.pos"
    tag "k8s.etcd"
    <parse>
      @type "none"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kubelet
    multiline_flush_interval 5s
    path "/var/log/kubelet.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kubelet.log.pos"
    tag "k8s.kubelet"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_proxy
    multiline_flush_interval 5s
    path "/var/log/kube-proxy.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-proxy.log.pos"
    tag "k8s.kube-proxy"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver
    multiline_flush_interval 5s
    path "/var/log/kube-apiserver.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-apiserver.log.pos"
    tag "k8s.kube-apiserver"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_controller_manager
    multiline_flush_interval 5s
    path "/var/log/kube-controller-manager.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-controller-manager.log.pos"
    tag "k8s.kube-controller-manager"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_scheduler
    multiline_flush_interval 5s
    path "/var/log/kube-scheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-scheduler.log.pos"
    tag "k8s.kube-scheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_rescheduler
    multiline_flush_interval 5s
    path "/var/log/rescheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-rescheduler.log.pos"
    tag "k8s.rescheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_glbc
    multiline_flush_interval 5s
    path "/var/log/glbc.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-glbc.log.pos"
    tag "k8s.glbc"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_cluster_autoscaler
    multiline_flush_interval 5s
    path "/var/log/cluster-autoscaler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-cluster-autoscaler.log.pos"
    tag "k8s.cluster-autoscaler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver_audit
    multiline_flush_interval 5s
    path "/var/log/kubernetes/kube-apiserver-audit.log"
    pos_file "/var/log/-kube-fluentd-operator--kube-apiserver-audit.log.pos"
    tag "k8s.kube-apiserver-audit"
    <parse>
      @type "multiline"
      format_firstline "/^\\S+\\s+AUDIT:/"
      format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
      time_format "%Y-%m-%dT%T.%L%Z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
    @id filter_kube_metadata
  </filter>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <filter kubernetes.**>
    @type record_transformer
    enable_ruby true
    <record>
      kubernetes_namespace_container_name ${record["kubernetes"]["namespace_name"]}.${record["kubernetes"]["pod_name"]}.${record["kubernetes"]["container_name"]}
      container_info ${record["docker"]["container_id"]}-${record["stream"]}
    </record>
  </filter>
  <match kubernetes.**>
    @type rewrite_tag_filter
    <rule>
      key "kubernetes_namespace_container_name"
      pattern ^(.+)$
      tag "kube.$1"
    </rule>
  </match>
  <filter kube.*.*.*>
    @type record_modifier
    remove_keys "dummy_"
    <record>
      dummy_ ${record["kubernetes"]&.delete("master_url"); record["kubernetes"]&.delete("namespace_id"); if record["kubernetes"]&.has_key?("labels"); record["kubernetes"]["labels"].delete("pod-template-generation"); record["kubernetes"]["labels"].delete("controller-revision-hash");  record["kubernetes"]["labels"].delete("pod-template-hash"); end; nil}
    </record>
  </filter>
  <filter kube.*.*.*>
    @type record_transformer
    remove_keys kubernetes_namespace_container_name
  </filter>
  <filter kube.kube-system.**>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type prometheus
  </source>
  <source>
    @type prometheus_monitor
  </source>
  <source>
    @type prometheus_output_monitor
  </source>
  <match systemd.** kube.kube-system.**>
    @type copy
    <store>
      @id X
      @type "vmware_loginsight"
      host "X.X.X.X"
      include_tag_key true
      log_text_keys ["log","msg","message"]
      port 9000
      scheme "http"
      serializer "json"
      ssl_verify false
      tag_key "tag"
    </store>
  </match>
  <match **>
    @type null
  </match>
</ROOT>

Your Error Log

2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: fluentd RPC got /api/config.gracefulReload request
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: fluentd supervisor process got SIGUSR2
2021-04-21 02:09:57 +0000 [info]: fluent/log.rb:329:info: Reloading new config
2021-04-21 02:09:57 +0000 [info]: fluent/log.rb:329:info: adding rewrite_tag_filter rule: unit [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x0000000004474560 @keys="unit">, /^(.+)$/, "", "systemd.$1"]
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Kubernetes URL is not set - inspecting environ
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Kubernetes URL is now 'https://172.21.0.1:443/api'
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Found directory with secrets: /var/run/secrets/kubernetes.io/serviceaccount
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Found CA certificate: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Found pod token: /var/run/secrets/kubernetes.io/serviceaccount/token
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Creating K8S client
2021-04-21 02:09:57 +0000 [info]: fluent/log.rb:329:info: adding rewrite_tag_filter rule: kubernetes_namespace_container_name [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00007f4ef49266a8 @keys="kubernetes_namespace_container_name">, /^(.+)$/, "", "kube.$1"]
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: adding store type="vmware_loginsight"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: using configuration file: <ROOT>
  <system>
    log_level info
    rpc_endpoint 127.0.0.1:24444
  </system>
  <source>
    @type systemd
    @id in_systemd_docker
    path "/var/log/journal"
    tag "systemd.unit"
    read_from_head false
    <storage>
      @type "local"
      persistent true
      path "/var/log/-kube-fluentd-operator--fluentd-journald-cursor.json"
    </storage>
    <entry>
      field_map {"_SYSTEMD_UNIT":"unit","MESSAGE":"log","_PID":["pid"],"_PRIORITY":"priority","_COMM":"cmd","_HOSTNAME":"hostname"}
      field_map_strict true
      fields_lowercase true
    </entry>
  </source>
  <match systemd.unit>
    @type rewrite_tag_filter
    <rule>
      key "unit"
      pattern ^(.+)$
      tag "systemd.$1"
    </rule>
  </match>
  <filter systemd.kubelet.service>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/*.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-containers.log.pos"
    tag "kubernetes.*"
    read_from_head true
    <parse>
      @type "multiline"
      format1 /^(?<partials>([^\n]+ (stdout|stderr) P [^\n]+\n)*)/
      format2 /(?<time>[^\n]+) (?<stream>stdout|stderr) F (?<log>[^\n]*)/
      format3 /|(?<json>{.*})/
      time_format "%Y-%m-%dT%H:%M:%S.%N%:z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type record_transformer
    @id filter_crio_container_logs
    enable_ruby true
    remove_keys partials
    <record>
      log ${record["partials"]&.gsub(/.+ (stdout|stderr) P (.+)\n/, '\\2')}${record["log"]}
    </record>
  </filter>
  <filter kubernetes.**>
    @type parser
    @id filter_docker_container_logs
    key_name "json"
    remove_key_name_field true
    reserve_data true
    emit_invalid_record_to_error false
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      time_type string
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_startupscript
    path "/var/log/startupscript.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-startupscript.log.pos"
    tag "startupscript"
    <parse>
      @type "syslog"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_docker
    path "/var/log/docker.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-docker.log.pos"
    tag "docker"
    <parse>
      @type "regexp"
      expression /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_etcd
    path "/var/log/etcd.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-etcd.log.pos"
    tag "k8s.etcd"
    <parse>
      @type "none"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kubelet
    multiline_flush_interval 5s
    path "/var/log/kubelet.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kubelet.log.pos"
    tag "k8s.kubelet"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_proxy
    multiline_flush_interval 5s
    path "/var/log/kube-proxy.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-proxy.log.pos"
    tag "k8s.kube-proxy"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver
    multiline_flush_interval 5s
    path "/var/log/kube-apiserver.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-apiserver.log.pos"
    tag "k8s.kube-apiserver"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_controller_manager
    multiline_flush_interval 5s
    path "/var/log/kube-controller-manager.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-controller-manager.log.pos"
    tag "k8s.kube-controller-manager"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_scheduler
    multiline_flush_interval 5s
    path "/var/log/kube-scheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-scheduler.log.pos"
    tag "k8s.kube-scheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_rescheduler
    multiline_flush_interval 5s
    path "/var/log/rescheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-rescheduler.log.pos"
    tag "k8s.rescheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_glbc
    multiline_flush_interval 5s
    path "/var/log/glbc.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-glbc.log.pos"
    tag "k8s.glbc"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_cluster_autoscaler
    multiline_flush_interval 5s
    path "/var/log/cluster-autoscaler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-cluster-autoscaler.log.pos"
    tag "k8s.cluster-autoscaler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver_audit
    multiline_flush_interval 5s
    path "/var/log/kubernetes/kube-apiserver-audit.log"
    pos_file "/var/log/-kube-fluentd-operator--kube-apiserver-audit.log.pos"
    tag "k8s.kube-apiserver-audit"
    <parse>
      @type "multiline"
      format_firstline "/^\\S+\\s+AUDIT:/"
      format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
      time_format "%Y-%m-%dT%T.%L%Z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <filter kubernetes.**>
    @type record_transformer
    enable_ruby true
    <record>
      kubernetes_namespace_container_name ${record["kubernetes"]["namespace_name"]}.${record["kubernetes"]["pod_name"]}.${record["kubernetes"]["container_name"]}
      container_info ${record["docker"]["container_id"]}-${record["stream"]}
    </record>
  </filter>
  <match kubernetes.**>
    @type rewrite_tag_filter
    <rule>
      key "kubernetes_namespace_container_name"
      pattern ^(.+)$
      tag "kube.$1"
    </rule>
  </match>
  <filter kube.*.*.*>
    @type record_modifier
    remove_keys "dummy_"
    <record>
      dummy_ ${record["kubernetes"]&.delete("master_url"); record["kubernetes"]&.delete("namespace_id"); if record["kubernetes"]&.has_key?("labels"); record["kubernetes"]["labels"].delete("pod-template-generation"); record["kubernetes"]["labels"].delete("controller-revision-hash");  record["kubernetes"]["labels"].delete("pod-template-hash"); end; nil}
    </record>
  </filter>
  <filter kube.*.*.*>
    @type record_transformer
    remove_keys kubernetes_namespace_container_name
  </filter>
  <filter kube.kube-system.**>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type prometheus
  </source>
  <source>
    @type prometheus_monitor
  </source>
  <source>
    @type prometheus_output_monitor
  </source>
  <match systemd.** kube.kube-system.** kube.avi-system.** kube.cluster-services.** kube.decc-tests.** kube.default.** kube.gulel-core.** kube.ingress.**>
    @type copy
    <store>
      @id out_vmware_loginsight_kubecluster_logs
      @type "vmware_loginsight"
      host "10.182.107.52"
      include_tag_key true
      log_text_keys ["log","msg","message"]
      port 9000
      scheme "http"
      serializer "json"
      ssl_verify false
      tag_key "tag"
    </store>
  </match>
  <match **>
    @type null
  </match>
</ROOT>
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: Worker 0 finished unexpectedly with signal SIGKILL
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding match pattern="systemd.unit" type="rewrite_tag_filter"
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered output plugin 'rewrite_tag_filter'
2021-04-21 02:09:59 +0000 [info]: #0 fluent/log.rb:329:info: adding rewrite_tag_filter rule: unit [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x0000000002915468 @keys="unit">, /^(.+)$/, "", "systemd.$1"]
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="systemd.kubelet.service" type="parser"
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered filter plugin 'parser'
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'regexp'
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'kubernetes'
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kubernetes.**" type="record_transformer"
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered filter plugin 'record_transformer'
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kubernetes.**" type="parser"
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'json'
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered filter plugin 'kubernetes_metadata'
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2021-04-21 02:09:59 +0000 [debug]: #0 fluent/log.rb:308:debug: Kubernetes URL is not set - inspecting environ
2021-04-21 02:09:59 +0000 [debug]: #0 fluent/log.rb:308:debug: Kubernetes URL is now 'https://172.21.0.1:443/api'
2021-04-21 02:09:59 +0000 [debug]: #0 fluent/log.rb:308:debug: Found directory with secrets: /var/run/secrets/kubernetes.io/serviceaccount
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: Found CA certificate: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: Found pod token: /var/run/secrets/kubernetes.io/serviceaccount/token
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: Creating K8S client
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kubernetes.**" type="record_transformer"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding match pattern="kubernetes.**" type="rewrite_tag_filter"
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: adding rewrite_tag_filter rule: kubernetes_namespace_container_name [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00000000032771a0 @keys="kubernetes_namespace_container_name">, /^(.+)$/, "", "kube.$1"]
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kube.*.*.*" type="record_modifier"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered filter plugin 'record_modifier'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kube.*.*.*" type="record_transformer"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kube.kube-system.**" type="parser"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding match pattern="systemd.** kube.kube-system.** kube.avi-system.** kube.cluster-services.** kube.decc-tests.** kube.default.** kube.gulel-core.** kube.ingress.**" type="copy"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered output plugin 'copy'
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: adding store type="vmware_loginsight"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered output plugin 'vmware_loginsight'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding match pattern="**" type="null"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered output plugin 'null'
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered buffer plugin 'memory'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="systemd"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'systemd'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'multiline'
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'tail'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'syslog'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'none'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="prometheus"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'prometheus'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="prometheus_monitor"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'monitor_agent'
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'prometheus_monitor'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="prometheus_output_monitor"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'prometheus_output_monitor'
2021-04-21 02:10:00 +0000 [warn]: #0 fluent/log.rb:350:warn: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: starting fluentd worker pid=21827 ppid=1 worker=0
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: listening prometheus http server on http:://0.0.0.0:24231//metrics for worker0
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: Start async HTTP server listening http://0.0.0.0:24231
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: Async::IO::SocketBinding to #<Addrinfo: 0.0.0.0:24231 TCP>
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-x4n6n_gulel-nimbus_gulel-container-716c1123-c4d94247f088e7c728dde61b09fd97b2614529931d81441fb1c7122150539c15.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-3bbf17ba-p58cb_gulel-worker_gulel-container-3bbf17ba-ea2da9f9fccb45a90613321fd2b9b827acec313f07437e7e25e043e5d8a4f2e9.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-cb51080e-dldz8_gulel-worker_gulel-container-cb51080e-1822bea7688a3b0867749954b5aa678c336c814ba2f0c6ece24bad5023b55771.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-bxbsd_gulel-nimbus_gulel-container-aa5df3e5-fc443f8ae7db9a102e77ed305d42e6e6e415b6f83ab4bfd479c6714ec48b8643.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/node-exporter-fjzmp_cluster-services_node-exporter-2ba2ebd7e22ec4b9bb65ec97fe3f9428367cb7ea9e6d84cb073fe56e74b7aeea.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/node-exporter-fjzmp_cluster-services_kube-rbac-proxy-54672a541018e7b8ac5cfc48ce0f5341f1ed2d04fd358c31718498d5cd1cc76c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-9c8cb82a-9p5b6_gulel-worker_gulel-container-9c8cb82a-1bb2d9f7bea86b48bdcd1f48a010f81618e7790429d575381ffba75aed35fd31.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-2nd4v_gulel-nimbus_gulel-container-da3d64e9-819568e95c2ad0bece1fc089b2a01f96a31c8c9f124962cd78457ddf9fdb86a8.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/redis-master-0_gulel-core_redis-d3ab86f045d201a8d1f403151109d78760800069dd32ee44bb8e050b8fbdb579.log
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: gulel-core/redis-master-0
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: gulel-core
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-df090692-x6nxp_gulel-worker_gulel-container-df090692-1371a6b4317343c90c6f7f65c3f3f882f46c8c517e7d7090e9e8a21f3905a530.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/vsphere-csi-node-7xc6j_kube-system_node-driver-registrar-c934e748c655e1edbafaf56ca502a9b431aa85132602bd50526bb98689745354.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-6c10db90-hwgmm_gulel-worker_gulel-container-6c10db90-35dc4109de3c71f2865f1aafb602211e8d681fbea2674681e2ac512355d06ccd.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-0aa5d677-2lnz7_gulel-worker_gulel-container-0aa5d677-c0e73223a6471e29c56390017ee21b208a326e8b8887d85b95e7ed102716a615.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-080de4f4-tqcd2_gulel-worker_gulel-container-080de4f4-10fd99598c9e9a924459086e8b1b98cc1ff1b3886f5131e811673e9acec611bd.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-v6x2f_gulel-nimbus_gulel-container-3dc10de7-8f794664e6665e59169a854d5fdcfb062fba122c2fac907fc92c8a8afde7c41d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-8cjfc_gulel-nimbus_gulel-container-0f97a1bf-bfa0abdddba2086312f12f15cb97f67a88fdf0854b22d40b54fa4ec2dc4c0c92.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-fdhbt_gulel-nimbus_gulel-container-ec76f9d7-7cacdd4b2f764e6c38238af51b68d9e7b39ab2e01a4efb87480ca3c7d5eb32cc.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-lpmr5_gulel-nimbus_gulel-container-8a91aaeb-6bdcacbaf527308bfe4dea87a51530ea2ecde1c6a0375f9c8a7ca8eb858d547c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/dns-analyser-226c2_decc-tests_dns-analyser-68a16c77ccc01e1d0b5a47399150ff986db84b946f68b485d97eec6ea182eac3.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-launcher-64b8bf6c87-b2qhr_gulel-core_launcher-382073d7c99978eae2dd708d8b91cbbbc6c4f3b5e22c8f5cc43e6b66809fce94.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-n8g4n_gulel-nimbus_gulel-container-7090d280-97465abf4251ba9386a62c0f24efdd5094bb6c800642e51615f31412c9fa05cc.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-cd72696c-gn44q_gulel-worker_gulel-container-cd72696c-bbcee55395c3d1095092580eefa11d787aa74773802cacaf0407bbfe4eda8a5c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-a1e6b5e2-tpbjm_gulel-worker_gulel-container-a1e6b5e2-8425322a761b74bcb27ecebee962513f86d00646374aefe4543e862ec3743029.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/amko-0_avi-system_amko-5136ddda74ea4fa6828578552820803eca47346c495f73b254ffc146d16ffbb5.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-tdpg9_gulel-nimbus_gulel-container-d4b5a145-3245c17dbf33b0e30f2365515a3071ef1bb31e1e95fff08c6a07824bb135c8e0.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-b8be93c2-mxx57_gulel-worker_gulel-container-b8be93c2-acd729f3c849474875e90253cf8b2da7112ed88f22e71ef9b4887c793a36b11d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-frblc_gulel-nimbus_gulel-container-efdbd678-372d99f5759d04d793085be83b08364f6ccf8bc1c963bb136c0b194e003f6629.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-dffe3e2e-kklhq_gulel-worker_gulel-container-dffe3e2e-bf5c05c3ce1472d91c41fdda5b5180937fa37e01d4b847abb0df20a2d11e47fe.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-797f2472-mjlnq_gulel-worker_gulel-container-797f2472-bd07de4657d88b81e634a8d62c5c488909ad49431cc0226436c7a82de8503ed8.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-rbr49_gulel-nimbus_gulel-container-b941b2db-dbb6dc75696ff7ca878bcf77157e7104df230028d91ceda90d3d763b3624454b.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-56850107-gtx5t_gulel-worker_gulel-container-56850107-067020efc500cc1dce418c597c1527e07e5d8169b004bc72b2a3754b3cf633cb.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-fqhh8_gulel-nimbus_gulel-container-e2f13708-fbd5023b4f490d89013798758cb5a4913954f25d5fd97e966d55fe529b781d73.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f3f24cec-v92fx_gulel-worker_gulel-container-f3f24cec-0ef41dac511e7aee97dba18ab2114ad416c9696b452e8ad15bd4e95db8474061.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/wavefront-proxy-64bb97d567-pbn7k_cluster-services_wavefront-proxy-7b7b4aab699225347581254e075bef52193377b5acf53e29c1eed614b70ced2d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-dm7kd_gulel-nimbus_gulel-container-7769ba08-a5431afae7c3cad26c983d19736078adc052935b71bd5cbd8da98d59b5c176d3.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-lxnmt_gulel-nimbus_gulel-container-dfd642a9-9f2ed9479ef1fc72bbec1597b3a20ac1e05935fd05961a882ed47b9d3d4f181c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-2a45c77d-8mshq_gulel-worker_gulel-container-2a45c77d-41cdf801493d868ae40dbab8906564f996fbd71c12520e7d36ba411a9482a85b.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-254e0c58-7sdkp_gulel-worker_gulel-container-254e0c58-98b8026683720bc84ebe510feeb58c1053865edb52213b417d108735de63a59c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-mk7wf_gulel-nimbus_gulel-container-7a5f9eb2-7a7ade713c7f4a195e776430cbcec98171243fbae4ad848be40ad1bc66dabe03.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-72c55315-jzmpz_gulel-worker_gulel-container-72c55315-c7c61657a8579226dd5196afb6121add96315e0308efef0a6668c0c0c2066aa8.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-brhxm_gulel-nimbus_gulel-container-5f016956-2305be0ed34c6ae3d95c7f348329c17a8c9be9c78f3187b4b8809e167c13567c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-ed48df94-9dmmn_gulel-worker_gulel-container-ed48df94-182f7f39b774ac1c3735ff1b4c157638ca157872d09d24b675302d26e28464e5.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-828b1ce3-mxmgr_gulel-worker_gulel-container-828b1ce3-615bbb0f48bb746e2b89c749d5b88cc3d9396c5dc4302d569690f12ae0692c3d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-0a4b4b51-fvrj5_gulel-worker_gulel-container-0a4b4b51-bd885bd0885d87de41b0884a95a912f57d78f7c153e85fd8c9a440c6a88f1ed7.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-690680ba-blmb8_gulel-worker_gulel-container-690680ba-5420cbac19b39ae9621434880b3d33202afc6ce8bbbeccbc6ce6d4b01025ab3f.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-gd8kx_gulel-nimbus_gulel-container-518a193f-5e9e305e69d9697cab96ce50d758fd222024770cd0e77c27c3a320c5c123a981.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/simple-flask-app-1-5cf57d9858-hx4tz_default_avi-ingress-simple-flask-app-1-581425470ec14950a4c9bcabdf62509c4e7150e2a8a4861183a4c6c36cd4ba53.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/devtools-test9-579d46559c-czsmq_default_devtools-test9-e3da3c06a7ff7b55c9dac0b5ce8eaf65b38f9b512a12366099aded1fef62bd5e.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-ndkds_gulel-nimbus_gulel-container-8638022a-d775090cf5c60d2eeabd9b5771a52c402ae153e83f24c74a331b84087d4ec035.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f7v4p_gulel-nimbus_gulel-container-d299f2ff-f5c7a95b68b0db8bee73ff9d5345dc25532b905491ed46b4bec378b3c8742e5a.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-4sd49_gulel-nimbus_gulel-container-a37d80d3-601b8b7d39401db489d383b07c469e1357e6359491b1b8ee285b8bba0fbbaafa.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f3d6cfef-f29fh_gulel-worker_gulel-container-f3d6cfef-b3ca02cdff0d97d0794465846efb3ba60188fcf0fd69b7b446308a137142b03c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-knq4b_gulel-nimbus_gulel-container-4306ed6b-d6c7b314b89fd1b7f986987dfb55040ae04c5cb0d0c3359ec9e2f905809db639.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/ako-0_avi-system_ako-4ac2dd331841699e850a18a6d08fa4c54a534e9076d3e5e8f9b3217595933636.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/antrea-agent-ng6zk_kube-system_install-cni-30e2407edd47e568de992bf495d75c90e11281caec8d9b31dce29e4805554f5d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/redis-master-0_gulel-core_redis-bcf428fce103c52a6f1cbbe5f67d4bff6bcd5b4361afd6209832d08012bd4d1e.log
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: gulel-core/redis-master-0
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: gulel-core
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f5e95df9-9s72x_gulel-worker_gulel-container-f5e95df9-2224550a49008851d50a0a3c1a84fdf88d59a98f399e1594dabeb30d1ec8b16d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-znzvv_gulel-nimbus_gulel-container-92ed7b7a-9a0593622b4572e04bf1b68797201a41972c4a40502e161088a86a1f12744a47.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/antrea-agent-ng6zk_kube-system_antrea-ovs-a3c32e86d0dcca43a28caff94732a518038d568ffd48bf14f589d10a910f38f1.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/vsphere-csi-node-7xc6j_kube-system_liveness-probe-5b358341424c5dbfbb1a431e6bb0bdba43f07f9f76e224e9546b2f172b81055a.log
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: kube-system/vsphere-csi-node-7xc6j
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: kube-system
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-l42k8_gulel-nimbus_gulel-container-e940ce85-d49cce8d1c7610fa26af60939c82dd151d9d5991adf11c1142655046534c0431.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-9ddf16a2-xfdsn_gulel-worker_gulel-container-9ddf16a2-076ebdf890a2cd87c57fff349212214108e962d4f89b27ffd3edf0c75cb45170.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-cx4sg_gulel-nimbus_gulel-container-53637411-efef1eed3628b9d3d1e2fbe7e760ee4c6cf56c5a9749207ac71d146ac911664c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-fluentd-operator-bk4bz_cluster-services_reloader-b6389c82549a504b3801926ecd63a2161cad81eeda7c1fcad91232b6acfc501e.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-238f72e4-6z45s_gulel-worker_gulel-container-238f72e4-e1223fe0e897eec6d1ce2dfaaa36348baae90d529f6f74efd18afe448045b6dc.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-m6x7g_gulel-nimbus_gulel-container-7405325f-b517943c19fd639f59c2d5a9ddf0b37af81c08409134538c7eb7dfc10bf8ee20.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-0debfe48-xpgvh_gulel-worker_gulel-container-0debfe48-f3ac8b3fce1ce260f3a224bbd3faa8b3316b7d597ba32ca417a2103e3cc34824.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/avi-ingress-test-vmohite5-78ddc68d75-7jx4j_default_avi-ingress-test-vmohite5-e9363ad309ae4154370397de6602d474b54517e3c7d677f372bd6b14b5088985.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-4st2v_gulel-nimbus_gulel-container-3706cf80-96844bedc92478b277633cf63f2e8e7e534f911e453a47a4f7839f5d1b0907c9.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-b5d392d7-5fd7b_gulel-worker_gulel-container-b5d392d7-d451c086e89963d46a6804e6198d7991880a6fdd4d3c60f1adb11ca2841b4b89.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-6349c60a-4rtd7_gulel-worker_gulel-container-6349c60a-f902f37a7400af7ae6172f3a7ed34e9f65ddf7f147c66e36bae7daad96d1056e.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/antrea-agent-ng6zk_kube-system_antrea-agent-2302c55456870a3c6f4f8efa6231037e12fc71c31254f62a184d4ffe45ea1cb0.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/avi-ingress-test-vmohite1-698474f8f9-9qzzh_default_avi-ingress-test-vmohite1-da9c901f762723a9c3297ed46a849a46d2a050c818f06756ea636a16efed7138.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-5dc1d02f-8lqbs_gulel-worker_gulel-container-5dc1d02f-423b4e99f08fbbafa25434da42be31187618bf4f4593c16499f0f099087592ab.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f7c68c84-wr5zh_gulel-worker_gulel-container-f7c68c84-baf968a68f4e1e89bf29a213e43e8390c2e64522e5d3b5436a518d9d554b1713.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-gzb67_gulel-nimbus_gulel-container-10b4b02e-0049df16d6ddc1fb4b8673dbe541e7bf822c8e5d636f03710d0f0ea652172de8.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-0f96f412-jmxrb_gulel-worker_gulel-container-0f96f412-fec4182b02e2724381459248dd01c4b42f56a9fb6f2f3f9eea2985850fde7958.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-4fbece99-6s6mn_gulel-worker_gulel-container-4fbece99-fccea3a1f19377ebb397a7ee44dfe1a690b4dd9a5f37b00fe79f6fca75d81d30.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/avi-ingress-test-vmohite4-596f6bc5f7-c4k7n_default_avi-ingress-test-vmohite4-497e177ee338171a2a2df3e1fd42c029435d902eedcfef182b27879126ef94f0.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-1153a8f6-qnq5k_gulel-worker_gulel-container-1153a8f6-ed75e93583aae4535ed24dbea566368f0bc6d81a37da1877b01ed04a1c7e3003.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-fjn5n_gulel-nimbus_gulel-container-03739a5c-e54d8d413f45f486c0fbf03d139c7b01cbcf5e4954c3229284f61cf9e99299c5.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-xxm8j_gulel-nimbus_gulel-container-0bbae7cf-b9a460fd1a53fdad4bc81301338148e02e42b63f0552997966ed89c4213fd5ad.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-69545c86-n8hhl_gulel-worker_gulel-container-69545c86-feb9649c2afe1ee7042d2a32a11a597fdb7d45a6e47039f189794ebc0fa1f887.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-409c7a3d-2tfcj_gulel-worker_gulel-container-409c7a3d-ced352bb8aee2dd36953f06187ee3c45fff333a4c51e273beb885eebb3bc63d5.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-2srt6_gulel-nimbus_gulel-container-f7aa5330-b275b62f0c3a3351c44d67971aa56a7b69edf29de4e391e05367c98ffa15d4ca.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-fvn8d_gulel-nimbus_gulel-container-3661369e-e0cfd787602cb87c865287bc4ec07403ea515ffa882fd538e30799f5ae9fcc13.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/avi-ingress-test-vmohite6-5ddb66cc69-rx2f2_default_avi-ingress-test-vmohite6-6b62779368cba4a248b9b2ae0c52c918d3f20cb59d1c5012c4000813314ef654.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-99eb2b1a-fp2jw_gulel-worker_gulel-container-99eb2b1a-de76edb6ba47731784cd71810471b281cf6da2f5f830b9a89b319f08139d4ae8.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-86d8g_gulel-nimbus_gulel-container-cfe4f336-b14d30b88ff6006b26d3f3f22c82c05eaedad92baa4aed7cfc2f494be49d8710.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-wdxwh_gulel-nimbus_gulel-container-5f6912db-094819918a37776c8f920379b4f22d83a41aa922afb18abaf03aede46eb3aafc.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-l87hq_gulel-nimbus_gulel-container-221f2d18-918ce4c82b27afdee4bf3c516a6d75d62ef92487f41a72b595667c70b823d09e.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-g66z6_gulel-nimbus_gulel-container-156dd45a-cc89ce70c567b409260378bd88d1006a7dc79ebedd9e5d7d82df16c09a6d1ac7.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/vsphere-csi-node-7xc6j_kube-system_vsphere-csi-node-8d57d6c883155b50a1eed4ec8c32d1d80ffd26b5bdf3f9e865eceb6d915715ac.log
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: kube-system/vsphere-csi-node-7xc6j
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: kube-system
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/avi-ingress-test-vmohite-776758d87c-rfp85_default_avi-ingress-test-vmohite-1d8c5a555ececa09e5e32efd51687f302f07cb25a4521a2ef63521781ea09e05.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/coredns-78dfc56b6f-dbg6w_kube-system_coredns-f0e56eb26fce56c32ea5168326852f62737ad5f7f5ece77c7a66398aeadc402c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-frslx_gulel-nimbus_gulel-container-84cc7b41-e9832af3b8da3ec3457b16514beb81cb9de304eaa6ec542bef23dc5feea8825e.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-9ef05fd1-nmf57_gulel-worker_gulel-container-9ef05fd1-d92ee5e7f57ee15da77faa85dc4e2468d5f2e18762165d91ae20857fbdd2aaf2.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-j2gqn_gulel-nimbus_gulel-container-41ea5a71-093009cc896beb4274a2e0fb1261e24599ea42fe62014c6b06c19ff8e4ea4ae9.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-lmvrk_gulel-nimbus_gulel-container-84c884f6-5b59112cfb66bc897083db45bff5cd3c23df3ef29821b122ec1f38f7c780ac1f.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-jrdzr_gulel-nimbus_gulel-container-d56fadeb-869cefed088ddf77affa59620244bbc07f51761229a538e83c6941a18ca58f21.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-7a1886c6-2pwl7_gulel-worker_gulel-container-7a1886c6-7f8695a354ec3c0e938ee058a9579f33247d8e51193d6f9359449eeafcce8350.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-vsmvc_gulel-nimbus_gulel-container-faeb472d-2a3541f8af38d2c11837bb353fd8ed9f2d5f186925ca80aa60f1c98eb0e42b09.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-d24dedfe-mlg48_gulel-worker_gulel-container-d24dedfe-ee319ae2282cedfb5e1320f94a8b08ac4ca263b348c898bc3f546079897f69fe.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-18b1f346-b45qt_gulel-worker_gulel-container-18b1f346-616251a0583ed3b2913582086320bb6224525d184ee9cc52cb2442caa02f616f.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-4xzvn_gulel-nimbus_gulel-container-74d686e8-953f477249f430caef0c2965a6121a39c311a1ff7b18fd789015160e483cef67.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-6bwm2_gulel-nimbus_gulel-container-56475349-f43b4a5882a9351093a45fbbec7afe1c5c66b05aa5c815b78a11246959d8f9fa.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-9cjq2_gulel-nimbus_gulel-container-1e39095f-80e991b0cea02475274391afb2c081de5439b5d8bdaacb681426897dcf613a8d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f4thl_gulel-nimbus_gulel-container-1ffce882-d6d6728621d280fdc6f92f0d21b0ed7ada6707acf78aa9171af6870dab6d1b08.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/metrics-server-67d7bb8965-l95gt_cluster-services_metrics-server-e9c9b95d8809a9e9eccea15c58ea3ba0a47ce6ef62dfc5a8dac93faa443b4598.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-proxy-27jt4_kube-system_kube-proxy-deaa9fa41007a33167c1044c296731b0e0f6249d23190857521933abfb6a03fa.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-53b9a040-qrh6n_gulel-worker_gulel-container-53b9a040-36c4ca15bd2a0085095274744cf498662f2e38971b1e43b6051edba455b5625c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/nfs-proxy-54bb9c4458-qm25f_cluster-services_nfs-proxy-063928113ca392cd3f06708cca1ce5b9394a84677fa221f2d7c0daa53e912e37.log
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: cluster-services/nfs-proxy-54bb9c4458-qm25f
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: cluster-services
2021-04-21 02:10:01 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:08 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:08 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: cluster-services/wavefront-proxy-64bb97d567-pbn7k
2021-04-21 02:10:08 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: cluster-services
2021-04-21 02:10:08 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:21 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:21 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: cluster-services/metrics-server-67d7bb8965-l95gt
2021-04-21 02:10:21 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: cluster-services
2021-04-21 02:10:21 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:29 +0000 [info]: #0 fluent/log.rb:329:info: stats - namespace_cache_size: 13, pod_cache_size: 131, namespace_cache_api_updates: 7, pod_cache_api_updates: 7, id_cache_miss: 7, pod_cache_host_updates: 131, namespace_cache_host_updates: 13
2021-04-21 02:10:42 +0000 [debug]: #0 fluent/log.rb:308:debug: #<Async::HTTP::Server:0x00000000042e21c0>Incoming connnection from #<Addrinfo: 172.24.12.100:58052 TCP> to Async::HTTP::Protocol::HTTP1
2021-04-21 02:10:59 +0000 [info]: #0 fluent/log.rb:329:info: stats - namespace_cache_size: 13, pod_cache_size: 131, namespace_cache_api_updates: 7, pod_cache_api_updates: 7, id_cache_miss: 7, pod_cache_host_updates: 131, namespace_cache_host_updates: 13
2021-04-21 02:11:01 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:11:01 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: gulel-core/gulel-launcher-64b8bf6c87-b2qhr
2021-04-21 02:11:01 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: gulel-core
2021-04-21 02:11:01 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:11:29 +0000 [info]: #0 fluent/log.rb:329:info: stats - namespace_cache_size: 13, pod_cache_size: 131, namespace_cache_api_updates: 8, pod_cache_api_updates: 8, id_cache_miss: 8, pod_cache_host_updates: 131, namespace_cache_host_updates: 13

Additional context

[kenhys]

NOTE: It seems that Worker 0 finished unexpectedly with signal SIGKILL is derived from:
https://github.com/treasure-data/serverengine/blob/master/lib/serverengine/multi_process_server.rb#L145

[kenhys]

It may be better to guard access to gracefulReload until reload_config finish.
https://github.com/fluent/fluentd/blob/master/lib/fluent/supervisor.rb#L135-L144

[kenhys]

mount_proc need to modify supporting custom response?
https://github.com/fluent/fluentd/blob/master/lib/fluent/rpc.rb#L41

[ashie]

AFAIK there is only one case that fluentd itself sends SIGKILL to workers (another case exists on Windows though).
When graceful stop takes 10 minutes, fluentd's supervisor process will send SIGKILL to workers.

https://github.com/treasure-data/serverengine/blob/master/lib/serverengine/process_manager.rb#L80

      graceful_kill_timeout: 600,

https://github.com/treasure-data/serverengine/blob/08193f12e8a36518a35244b94f2865d974538450/lib/serverengine/process_manager.rb#L406-L412

               # escalation
               @graceful_kill_start_time &&
               @graceful_kill_timeout >= 0 &&
               @graceful_kill_start_time < now - @graceful_kill_timeout

But it wont' be triggered in this case, because this case is graceful reload, not graceful stop.
In this case, fluentd tries to reload config without replacing a worker process.

In addition, since fluentd doesn't enable serverengine's heartbeat feature, it won't be triggered too:
https://github.com/treasure-data/serverengine/blob/08193f12e8a36518a35244b94f2865d974538450/lib/serverengine/process_manager.rb#L404-L406
https://github.com/treasure-data/serverengine/blob/08193f12e8a36518a35244b94f2865d974538450/lib/serverengine/multi_spawn_server.rb#L38

Because I don't know other cases that fluentd triggers SIGKILL, I'm suspecting that something other sends SIGKILL.
OOM killer?

[cosmo0920]

Because I don't know other cases that fluentd triggers SIGKILL, I'm suspecting that something other sends SIGKILL.
OOM killer?

And also the possibility is a bit of smaller, workers could be SIGKILLed by SIGSEGV. (I think that this case is not caused by SIGSEGV....)

[ashie]

And also the possibility is a bit of smaller, workers could be SIGKILLed by SIGSEGV. (I think that this case is not caused by SIGSEGV....)

I got the following output by causing intentional SEGV:

/home/aho/Projects/Fluentd/fluentd/lib/fluent/engine.rb:152: [BUG] Segmentation fault at 0x000003e80025bb0b
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux-gnu]

-- Control frame information -----------------------------------------------
c:0011 p:---- s:0069 e:000068 CFUNC  :sleep
c:0010 p:0070 s:0064 e:000063 METHOD /home/aho/Projects/Fluentd/fluentd/lib/fluent/engine.rb:152
...
2021-06-09 11:19:30 +0900 [info]: Worker 0 finished unexpectedly with signal SIGABRT
...

So that we can exculde this case.

[ashie]

AFAIK there is only one case that fluentd itself sends SIGKILL to workers (another case exists on Windows though).
When graceful stop takes 10 minutes, fluentd's supervisor process will send SIGKILL to workers.

I checked the implementation again but I can't find a case that it sends KILL signal.
So I'm suspecting OOM killer.

As I mentioned at #3413 (comment), gracefulReload needs more memory, it might be the cause of this issue.

On gracefulReload, fluentd tries to create a new Fluent::Engine with a new config without replacing the process. The old Fluent::Engine instance is also running while this, so that the process holds two engines while gracefulReload. (when multiple gracefulReload is attempted in short time, it might have three or more engines, I don't yet confirm it though).

[cosmo0920]

In k8s environment, users sometimes set up CPU and memory restrictions for preventing eating up CPU and memory resources:
https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-resource-requests-and-limits

When users set up CPU and memory restrictions for Pods/Daemonsets, they get frequently OOM killer issues via SIGSEGV.

[ashie]

As I mentioned before, we think it's triggered by OOM killer, the cause is high memory usage.
For high memory usage or memory leak, we have several issues already. e.g.) #1657
For reducing memory usage on multiple gracefulReload, we filed #3415.
So I close this issue.