Bug 1740263: enable the parsing of docker log-driver=json-file continuation lines using fluent concat plugin #1723

richm · 2019-08-14T15:14:23Z

This commit allows fluentd to reconstruct partial lines written
by the docker json-file and journald log drivers when logs
exceed the 16K byte limit.

There are two new environment variables:

USE_MULTILINE_JSON - by default this is false - if you do
oc set env ds/logging-fluentd USE_MULTILINE_JSON=true
then fluentd will be able to reconstruct docker json-file
partial logs.
USE_MULTILINE_JOURNAL - by default this is false - if you do
oc set env ds/logging-fluentd USE_MULTILINE_JOURNAL=true
then fluentd will be able to reconstruct docker journald
partial logs.

For json-file logs, the "log" field ends in \n for the final
part of the log, and does not end in \n for starting and
continuation lines. For journald logs, the field
CONTAINER_PARTIAL_MESSAGE=true is present for starting and
continuation lines, but is omitted for final lines.

fluent-plugin-concat 2.4.0 was backported to work with ruby 2.0
and fluentd 0.12. The main feature was the ability to have
only multiline_end_regexp without multiline_start_regexp
which is required for docker json-file log support. The
partial_key support for journald was already there for cri-o.
The wrinkle with journald is that all records to be
reconstructed must have the CONTAINER_PARTIAL_MESSAGE field,
so a filter was added to set CONTAINER_PARTIAL_MESSAGE=false
for container log records which did not already have the
CONTAINER_PARTIAL_MESSAGE field, in order to make the concat
filter work for partial_key.

If you want to try this out without building the image, you can
follow these steps:

Hack fluent.conf like this:

    #@include configs.d/dynamic/input-docker-*.conf
    <source>
      @type tail
      @id docker-input
      @label @INGRESS
      path "/var/log/containers/*.log"
      pos_file "/var/log/es-containers.log.pos"
      time_format %Y-%m-%dT%H:%M:%S.%N%Z
      tag kubernetes.*
      format json
      keep_time_key true
      read_from_head "true"
      exclude_path []
      @label @CONCAT
    </source>
    <label @CONCAT>
      <filter kubernetes.**>
        @type concat
        key log
        multiline_end_regexp /\n$/
      </filter>
      <match kubernetes.**>
        @type relabel
        @label @INGRESS
      </match>
    </label>
    ...
    <label @INGRESS>
    ## filters
      @include configs.d/openshift/filter-pre-*.conf
      <filter journal>
        @type record_modifier
        <record>
          ignoreme ${if record.key?("CONTAINER_ID_FULL") && !record.key?("CONTAINER_PARTIAL_MESSAGE"); record["CONTAINER_PARTIAL_MESSAGE"] = "false"; end; "ignoreme"}
        </record>
        remove_keys ignoreme
      </filter>
      <filter journal>
        @type concat
        key MESSAGE
        separator ""
        stream_identity_key CONTAINER_ID_FULL
        partial_key CONTAINER_PARTIAL_MESSAGE
        partial_value true
      </filter>

Create a special configmap for the plugin code:

mkdir cm-fluentd-plugin
oc get pods -l component=fluentd
fpod=logging-fluentd-xxx
for file in $( oc exec $fpod -- ls /etc/fluent/plugin ) ; do
  oc exec $fpod -- cat /etc/fluent/plugin/$file > cm-fluentd-plugin/$file
done
cp cm-fluentd-plugin/filter_concat.rb cm-fluentd-plugin/filter_concat.rb.orig
cp /path/to/new/filter_concat.rb cm-fluentd-plugin/filter_concat.rb
oc create configmap fluentd-plugin --from-file=cm-fluentd-plugin/

Then, add the volume and volumemount to the fluentd daemonset:

oc edit ds/logging-fluentd

Add to volumeMounts and volumes

        volumeMounts:
        - mountPath: /etc/fluent/plugin
          name: fluentd-plugin
          readOnly: true
        ...
      volumes:
      - configMap:
          defaultMode: 420
          name: fluentd-plugin
        name: fluentd-plugin

Restart fluentd

oc delete pods -l component=fluentd

You may see errors like this in the fluentd log:

/etc/fluent/plugin/viaq_docker_audit.rb:51: warning: already initialized constant Fluent::ViaqDockerAudit::ENV_HOSTNAME

You can ignore them.

add support for USE_MULTILINE_JOURNAL

If USE_MULTILINE_JOURNAL=true, then docker log-driver=journald logs
that are spread over multiple records using CONTAINER_PARTIAL_MESSAGE
will be concatenated together as a single record.

bug fixes

dump indices upon error

@include

This commit allows fluentd to reconstruct partial lines written by the docker json-file and journald log drivers when logs exceed the 16K byte limit. There are two new environment variables: * `USE_MULTILINE_JSON` - by default this is false - if you do `oc set env ds/logging-fluentd USE_MULTILINE_JSON=true` then fluentd will be able to reconstruct docker json-file partial logs. * `USE_MULTILINE_JOURNAL` - by default this is false - if you do `oc set env ds/logging-fluentd USE_MULTILINE_JOURNAL=true` then fluentd will be able to reconstruct docker journald partial logs. For json-file logs, the "log" field ends in `\n` for the final part of the log, and does not end in `\n` for starting and continuation lines. For journald logs, the field `CONTAINER_PARTIAL_MESSAGE=true` is present for starting and continuation lines, but is omitted for final lines. fluent-plugin-concat 2.4.0 was backported to work with ruby 2.0 and fluentd 0.12. The main feature was the ability to have only `multiline_end_regexp` without `multiline_start_regexp` which is required for docker json-file log support. The partial_key support for journald was already there for cri-o. The wrinkle with journald is that _all_ records to be reconstructed must have the `CONTAINER_PARTIAL_MESSAGE` field, so a filter was added to set `CONTAINER_PARTIAL_MESSAGE=false` for container log records which did not already have the `CONTAINER_PARTIAL_MESSAGE` field, in order to make the concat filter work for partial_key. If you want to try this out without building the image, you can follow these steps: Hack fluent.conf like this: ``` #@include configs.d/dynamic/input-docker-*.conf <source> @type tail @id docker-input @Label @ingress path "/var/log/containers/*.log" pos_file "/var/log/es-containers.log.pos" time_format %Y-%m-%dT%H:%M:%S.%N%Z tag kubernetes.* format json keep_time_key true read_from_head "true" exclude_path [] @Label @concat </source> <label @concat> <filter kubernetes.**> @type concat key log multiline_end_regexp /\n$/ </filter> <match kubernetes.**> @type relabel @Label @ingress </match> </label> ... <label @ingress> ## filters @include configs.d/openshift/filter-pre-*.conf <filter journal> @type record_modifier <record> ignoreme ${if record.key?("CONTAINER_ID_FULL") && !record.key?("CONTAINER_PARTIAL_MESSAGE"); record["CONTAINER_PARTIAL_MESSAGE"] = "false"; end; "ignoreme"} </record> remove_keys ignoreme </filter> <filter journal> @type concat key MESSAGE separator "" stream_identity_key CONTAINER_ID_FULL partial_key CONTAINER_PARTIAL_MESSAGE partial_value true </filter> ``` Create a special configmap for the plugin code: ``` mkdir cm-fluentd-plugin oc get pods -l component=fluentd fpod=logging-fluentd-xxx for file in $( oc exec $fpod -- ls /etc/fluent/plugin ) ; do oc exec $fpod -- cat /etc/fluent/plugin/$file > cm-fluentd-plugin/$file done cp cm-fluentd-plugin/filter_concat.rb cm-fluentd-plugin/filter_concat.rb.orig cp /path/to/new/filter_concat.rb cm-fluentd-plugin/filter_concat.rb oc create configmap fluentd-plugin --from-file=cm-fluentd-plugin/ ``` Then, add the volume and volumemount to the fluentd daemonset: ``` oc edit ds/logging-fluentd ``` Add to volumeMounts and volumes ``` volumeMounts: - mountPath: /etc/fluent/plugin name: fluentd-plugin readOnly: true ... volumes: - configMap: defaultMode: 420 name: fluentd-plugin name: fluentd-plugin ``` Restart fluentd ``` oc delete pods -l component=fluentd ``` You may see errors like this in the fluentd log: ``` /etc/fluent/plugin/viaq_docker_audit.rb:51: warning: already initialized constant Fluent::ViaqDockerAudit::ENV_HOSTNAME ``` You can ignore them. add support for USE_MULTILINE_JOURNAL If USE_MULTILINE_JOURNAL=true, then docker log-driver=journald logs that are spread over multiple records using CONTAINER_PARTIAL_MESSAGE will be concatenated together as a single record. bug fixes dump indices upon error

openshift-ci-robot · 2019-08-14T15:21:45Z

@richm: This pull request references a valid Bugzilla bug.

In response to this:

Bug 1740263: enable the parsing of docker log-driver=json-file continuation lines using fluent concat plugin

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

richm · 2019-08-14T17:46:07Z

/test json-file

richm · 2019-08-14T20:11:45Z

/test json-file

nhosoi · 2019-08-14T20:45:08Z

/lgtm

openshift-ci-robot · 2019-08-14T20:45:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nhosoi, richm

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [nhosoi,richm]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2019-08-14T22:18:43Z

/retest