Bug 1722898: Logging data from all projects are stored to .orphaned i…

…ndexes with Elasticsearch

https://bugzilla.redhat.com/show_bug.cgi?id=1722898
Cause: Fluentd is unable to correctly determine the docker log
driver.  It thinks the log driver is journald when it is json-file.
Fluentd then looks for the `CONTAINER_NAME` field in the record to
hold the kubernetes metadata and it is not present.

Consequence: Fluentd is not able to add kubernetes metadata to
records.  Records go to the .orphaned index.  Fluentd spews lots
of errors like this:
[error]: record cannot use elasticsearch index na     me type project_full: record is missing kubernetes field

Fix: Fluentd should not rely on reading the docker configuration file
to determine if the record contains kubernetes metadata.  It should
look at both the record tag and the record data and use whatever
kubernetes metadata it finds there.

Result: Fluentd can correctly add kubernetes metadata and assign
records to the correct indices no matter which log driver docker
is using.

Records read from files under /var/log/containers/*.log will have
a fluentd tag like kubernetes.var.log.containers.**.  This applies
both to CRI-O and docker file logs.  Kubernetes records read from
journald with CONTAINER_NAME will have a tag like
journal.kubernetes.**.  There is no CRI-O journald log driver yet,
and it is not clear how those records will be represented, but
hopefully they will follow the same CONTAINER_NAME convention, in
which case they will Just Work.

fluentd/configs.d/openshift/filter-k8s-meta.conf

@@ -7,9 +7,16 @@
   ssl_partial_chain "#{ENV['SSL_PARTIAL_CHAIN'] || 'true'}"
 </filter>
 
-<filter kubernetes.**>
+<filter kubernetes.journal.**>
+  @type parse_json_field
+  merge_json_log "#{ENV['MERGE_JSON_LOG'] || 'false'}"
+  preserve_json_log "#{ENV['PRESERVE_JSON_LOG'] || 'true'}"
+  json_fields "#{ENV['JSON_FIELDS'] || 'MESSAGE,log'}"
+</filter>
+
+<filter kubernetes.var.log.containers.**>
   @type parse_json_field
   merge_json_log "#{ENV['MERGE_JSON_LOG'] || 'false'}"
   preserve_json_log "#{ENV['PRESERVE_JSON_LOG'] || 'true'}"
-  json_fields "#{ENV['JSON_FIELDS'] || (ENV['USE_JOURNAL'] == 'true' ? 'MESSAGE,log' : 'log,MESSAGE')}"
+  json_fields "#{ENV['JSON_FIELDS'] || 'log,MESSAGE'}"
 </filter>

fluentd/run.sh

@@ -60,36 +60,13 @@ issue_deprecation_warnings() {
     : # none at the moment
 }
 
-docker_uses_journal() {
-    # need to be able to handle cases like
-    # OPTIONS='--log-driver=json-file ....' # or use --log-driver=journald
-    # if "log-driver" is set in /etc/docker/daemon.json, assume that it is
-    # authoritative
-    # otherwise, look for /etc/sysconfig/docker
-    # also note the unintuitive logic - in this case, a 0 return means true, and a 1
-    # return means false
-    if grep -q '^[^#].*"log-driver":' /etc/docker/daemon.json 2> /dev/null ; then
-        if grep -q '^[^#].*"log-driver":.*journald' /etc/docker/daemon.json 2> /dev/null ; then
-            return 0
-        fi
-    elif grep -q "^OPTIONS='[^']*--log-driver[   =][     ]*journald" /etc/sysconfig/docker 2> /dev/null ; then
-        return 0
-    fi
-    return 1
-}
-
 if [ -z "${JOURNAL_SOURCE:-}" ] ; then
     if [ -d /var/log/journal ] ; then
         export JOURNAL_SOURCE=/var/log/journal
     else
         export JOURNAL_SOURCE=/run/log/journal
     fi
 fi
-if docker_uses_journal ; then
-    export USE_JOURNAL=true
-else
-    export USE_JOURNAL=false
-fi
 
 IPADDR4=`/usr/sbin/ip -4 addr show dev eth0 | grep inet | sed -e "s/[ \t]*inet \([0-9.]*\).*/\1/"`
 IPADDR6=`/usr/sbin/ip -6 addr show dev eth0 | grep inet6 | sed "s/[ \t]*inet6 \([a-f0-9:]*\).*/\1/"`