Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1722380: Logging data from all projects are stored to .orphaned indexes with Elasticsearch #1680

Merged
merged 1 commit into from Jun 26, 2019
Merged

Bug 1722380: Logging data from all projects are stored to .orphaned indexes with Elasticsearch #1680

merged 1 commit into from Jun 26, 2019

Conversation

richm
Copy link
Contributor

@richm richm commented Jun 25, 2019 •
edited

https://bugzilla.redhat.com/show_bug.cgi?id=1722380
Cause: Fluentd is unable to correctly determine the docker log
driver. It thinks the log driver is journald when it is json-file.
Fluentd then looks for the CONTAINER_NAME field in the record to
hold the kubernetes metadata and it is not present.

Consequence: Fluentd is not able to add kubernetes metadata to
records. Records go to the .orphaned index. Fluentd spews lots
of errors like this:

[error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field

Fix: Fluentd should not rely on reading the docker configuration file
to determine if the record contains kubernetes metadata. It should
look at both the record tag and the record data and use whatever
kubernetes metadata it finds there.

Result: Fluentd can correctly add kubernetes metadata and assign
records to the correct indices no matter which log driver docker
is using.

Records read from files under /var/log/containers/*.log will have
a fluentd tag like kubernetes.var.log.containers.**. This applies
both to CRI-O and docker file logs. Kubernetes records read from
journald with CONTAINER_NAME will have a tag like
journal.kubernetes.**. There is no CRI-O journald log driver yet,
and it is not clear how those records will be represented, but
hopefully they will follow the same CONTAINER_NAME convention, in
which case they will Just Work.

(cherry picked from commit 33011c5)
manual cherrypick of #1678

@openshift-ci-robot
Copy link

@richm: This pull request references a valid Bugzilla bug.

In response to this:

Bug 1722380: Logging data from all projects are stored to .orphaned indexes with Elasticsearch

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 25, 2019
@richm richm requested review from jcantrill and ewolinetz and removed request for jwforres June 25, 2019 16:17
@richm richm added autoretest Please auto-retest this PR if one of the flaky tests fail component/fluentd kind/bug Categorizes issue or PR as related to a bug. release/3.11 cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. labels Jun 25, 2019
@openshift-ci-robot
Copy link

@richm: This pull request references a valid Bugzilla bug.

In response to this:

Bug 1722380: Logging data from all projects are stored to .orphaned indexes with Elasticsearch

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@richm
Copy link
Contributor Author

richm commented Jun 25, 2019

https://bugzilla.redhat.com/show_bug.cgi?id=1722380

@nhosoi
Copy link
Contributor

nhosoi commented Jun 25, 2019

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 25, 2019
@richm
Copy link
Contributor Author

richm commented Jun 25, 2019

/test logging

@richm
Bug 1722380: Logging data from all projects are stored to .orphaned i…
07a580d 
…ndexes with Elasticsearch

https://bugzilla.redhat.com/show_bug.cgi?id=1722380
Cause: Fluentd is unable to correctly determine the docker log
driver.  It thinks the log driver is journald when it is json-file.
Fluentd then looks for the `CONTAINER_NAME` field in the record to
hold the kubernetes metadata and it is not present.

Consequence: Fluentd is not able to add kubernetes metadata to
records.  Records go to the .orphaned index.  Fluentd spews lots
of errors like this:
```
[error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field
```

Fix: Fluentd should not rely on reading the docker configuration file
to determine if the record contains kubernetes metadata.  It should
look at both the record tag and the record data and use whatever
kubernetes metadata it finds there.

Result: Fluentd can correctly add kubernetes metadata and assign
records to the correct indices no matter which log driver docker
is using.

Records read from files under /var/log/containers/*.log will have
a fluentd tag like kubernetes.var.log.containers.**.  This applies
both to CRI-O and docker file logs.  Kubernetes records read from
journald with CONTAINER_NAME will have a tag like
journal.kubernetes.**.  There is no CRI-O journald log driver yet,
and it is not clear how those records will be represented, but
hopefully they will follow the same CONTAINER_NAME convention, in
which case they will Just Work.
Using the string value of `'nil'` will cause the fluentd config
parser to turn this into the ruby `nil` value.

(cherry picked from commit 33011c5)
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jun 25, 2019
@richm
Copy link
Contributor Author

richm commented Jun 26, 2019

tests are passing now - please review - see the nil trick

@ewolinetz
Copy link
Contributor

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 26, 2019
@openshift-merge-robot openshift-merge-robot merged commit 3967642 into openshift:release-3.11 Jun 26, 2019
@richm richm deleted the release-3.11-bz1722380 branch June 26, 2019 15:08
@richm
Copy link
Contributor Author

richm commented Jun 26, 2019

/cherrypick release-3.10

@openshift-cherrypick-robot
Copy link

@richm: #1680 failed to apply on top of branch "release-3.10":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	fluentd/configs.d/openshift/filter-k8s-meta.conf
M	fluentd/run.sh
Falling back to patching base and 3-way merge...
Auto-merging fluentd/run.sh
Auto-merging fluentd/configs.d/openshift/filter-k8s-meta.conf
CONFLICT (content): Merge conflict in fluentd/configs.d/openshift/filter-k8s-meta.conf
Patch failed at 0001 Bug 1722380: Logging data from all projects are stored to .orphaned indexes with Elasticsearch

In response to this:

/cherrypick release-3.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nhosoi nhosoi Awaiting requested review from nhosoi

@jcantrill jcantrill Awaiting requested review from jcantrill

@ewolinetz ewolinetz Awaiting requested review from ewolinetz

Assignees

@nhosoi nhosoi

@ewolinetz ewolinetz

Labels
autoretest Please auto-retest this PR if one of the flaky tests fail bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. component/fluentd kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. release/3.11 size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@richm @openshift-ci-robot @nhosoi @ewolinetz @openshift-cherrypick-robot @openshift-merge-robot