LOG-1286: Pick daemon names when configuring addLogSource field

[k-keiichi-rh]

Description

The current syslog forwarding can not report who outputted logs in journal log when configuring addLogSource field.
To identify the daemon name, the "systemd.u.SYSLOG_IDENTIFIER" in journal log record is what information we want.
So fluentd can pick their daemon names from journal log records.

This PR enhances the addLogSource field in Log Forwarding API by splitting infrastructure stream into journal log and pod log and injecting daemon names into the journal log stream.
So this change is limited to the addLogSource field related code.

This PR picks the originating process name and id(known as the "pid") from the record and injects them into the header field.
There are three type messages in the journal log and the process information are injected like the following:

1. System messages stored in the journal log
   "Tag" and "AppName" fields in CLF are ignored for the system messages when enabling the "AddLogSource" field.

<158>May 18 00:21:27 worker3.cluster.sub.nec.test crio[1738]: time="2021-05-18 00:21:25.958832751Z" level=info msg="About to del CNI network multus
-cni-network (type=multus)"

2. Kernel messages stored in the journal
   Any daemon information are not injected to the kernel messages.

<158>May 18 00:21:27 worker3.cluster.sub.nec.test kernel: device vethc2d8643c left promiscuous mode

3. System pod messages not stored in the journal
   Any daemon information are not injected to the pod messages.
   The "fluentd:" is overwritten by specifying "Tag" and "AppName" fields.

<158>May 18 00:23:00 worker4.cluster.sub.nec.test fluentd: namespace_name=openshift-image-registry, container_name=node-ca, pod_name=node-ca-q5m
n5, message=image-registry.openshift-image-registry.svc:5000

/cc @vimalk78 @alanconway

Links

• Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1891886
• JIRA: https://issues.redhat.com/browse/LOG-1286

[k-keiichi-rh]

/test e2e-operator

[k-keiichi-rh]

/test e2e-operator

[alanconway]

/approve
Looks good, will lete @vimalk78 give the final LGTM as he's the syslog output expert.

[k-keiichi-rh]

@alanconway Thank you for your review. I have an issue for e2e testing in forward_to_syslog_test.go. I will fix it and remove "WIP:".

[k-keiichi-rh]

/test lint

[k-keiichi-rh]

/test e2e-operator

[k-keiichi-rh]

/test e2e-operator

[vimalk78]

i dont catch using the syslog store template here.

[k-keiichi-rh]

Infrastructure pod logs and journal logs are sent under "infrastructure".
But the infrastructure pod logs doesn't have $.systemd.u.SYSLOG_IDENTIFIER.
So I splitted "infrastructure" stream into the pod logs and the journal logs and then injected the daemon name into the journal logs only.
This syslog store template is for the journal logs. I will add some comments to explain this.

[vimalk78]

Can you please add an e2e/functional test to highlight the use of the changes

[k-keiichi-rh]

I will add a e2d testcase for this PR.

[alanconway]

@k-keiichi-rh What's the status of this PR? Is it still WIP or is it ready to consider for merge?

[k-keiichi-rh]

@k-keiichi-rh What's the status of this PR? Is it still WIP or is it ready to consider for merge?

I believe that the function is ready to consider for merge.
But the e2e testcase for the function doesn't work well. Now I am trying to fix it.
After finishing the fix, I will remove "WIP".

[k-keiichi-rh]

/test e2e-gcp

[k-keiichi-rh]

/test e2e-gcp

[alanconway]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alanconway, k-keiichi-rh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [alanconway]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[alanconway]

Hi @k-keiichi-rh, you need to add "Bug 1891886:" in front of the title of this PR so that the CI bot will associate it with the BZ.

[openshift-ci]

@k-keiichi-rh: This pull request references Bugzilla bug 1891886, which is invalid:

• expected the bug to be open, but it isn't
• expected the bug to target the "4.8.0" release, but it targets "4.7.0" instead
• expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is CLOSED (ERRATA) instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1891886: LOG-1286: Pick daemon names when configuring addLogSource field

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k-keiichi-rh]

@alanconway Hi. Thank you for the comment. I am a little confused.
This PR is targeted to the master branch, which is for CLO5.2 now I think. So I filed a Jira ticket for this PR.
Do I need to backport this patch to CLO5.1, CLO5.0 and CLO4.7?

[openshift-ci]

@k-keiichi-rh: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

LOG-1286: Pick daemon names when configuring addLogSource field

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jcantrill]

/refresh
/bugzilla refresh

[openshift-ci]

@jcantrill: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

/refresh
/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jcantrill]

/hold cancel

[jcantrill]

@alanconway Hi. Thank you for the comment. I am a little confused.
This PR is targeted to the master branch, which is for CLO5.2 now I think. So I filed a Jira ticket for this PR.
Do I need to backport this patch to CLO5.1, CLO5.0 and CLO4.7?

Is this a bug? If so, how critical is it that it gets into 5.0 or even 4.6? Can we simply backport to 5.1 and ask user's to upgrade? We would prefer to backport as few releases as possible if user's are capable of working around the issue. If not, the BZ should be for logging 4.6 and we should have JIRA issues for the 5.x code changes

[k-keiichi-rh]

Is this a bug? If so, how critical is it that it gets into 5.0 or even 4.6? Can we simply backport to 5.1 and ask user's to upgrade? We would prefer to backport as few releases as possible if user's are capable of working around the issue. If not, the BZ should be for logging 4.6 and we should have JIRA issues for the 5.x code changes

Thank you for the clarification. This is a feature request to 5.2, not a bug. So we don't need to backport to previous versions.