Generate uuid on ingestion to reduce duplicates #974

jcantrill · 2018-02-23T15:37:11Z

This PR generates record uuids at the source to resolve:

https://bugzilla.redhat.com/show_bug.cgi?id=1548104

where when ES is underload, it fails bulk indexing and retries can produce duplicate entries

richm · 2018-02-23T16:22:13Z

/test json-file

richm · 2018-02-23T20:31:34Z

/test logging

jcantrill · 2018-02-23T21:07:44Z

/test json-file

richm · 2018-02-24T03:20:43Z

/retest

jcantrill · 2018-02-24T14:56:35Z

/retest

richm · 2018-02-24T18:33:17Z

please test with #933
or, let's merge 933, then this PR can change the bulk index rejection test to look for expected count == actual count instead of expected count <= actual count

mffiedler · 2018-02-26T19:55:34Z

fluentd/configs.d/user/fluent.conf

+  <filter **>
+    @type elasticsearch_genid
+    hash_id_key _openshift_es_genid
+  </filter>


Follow up to IRC discussion - this should go above filter-post-*.conf. My successful (no duplicates) config was:

<label @INGRESS> ## filters @include configs.d/openshift/filter-pre-*.conf @include configs.d/openshift/filter-retag-journal.conf @include configs.d/openshift/filter-k8s-meta.conf @include configs.d/openshift/filter-kibana-transform.conf @include configs.d/openshift/filter-k8s-flatten-hash.conf @include configs.d/openshift/filter-k8s-record-transform.conf @include configs.d/openshift/filter-syslog-record-transform.conf @include configs.d/openshift/filter-viaq-data-model.conf <filter **> @type elasticsearch_genid hash_id_key _openshift_es_genid </filter> @include configs.d/openshift/filter-post-*.conf </label>

ewolinetz · 2018-02-27T15:03:43Z

fluentd/configs.d/user/fluent.conf

+  # requires fluent-plugin-elasticsearch
+  <filter **>
+    @type elasticsearch_genid
+    hash_id_key _openshift_es_genid


is there a reason we are using this particular key only to map and remove it later?

This is what is recommended by the filter. The filter generates the key, copies the value to the 'id' field, and then comments that ES does not like records with '' so it is removed prior to submission

I don't think this file is used.
The "real" fluent.conf comes from ansible.
If we want this in the product by default, we'll need to create configs.d/openshift/filter-post-esid.conf or something like that with this config. And - if we want to easily disable it, we'll need to make the inclusion of this file conditional based on an env var.

ack, i read the documentation for the filter type. I didn't realize that id_key in the output plugin denotes what should be used as that key... I read it as it was setting another key to be that value.

richm

/lgtm

jcantrill · 2018-02-28T17:18:31Z

/test json-file

richm · 2018-02-28T23:52:48Z

/test json-file

richm · 2018-03-01T01:36:02Z

Running test/fluentd-forward.sh:258: executing 'wait_for_fluentd_to_catch_up '' '' 2' expecting success...
FAILURE after 1245.828s: test/fluentd-forward.sh:258: executing 'wait_for_fluentd_to_catch_up '' '' 2' expecting success: the command returned the wrong error code
Standard output from the command:
1
... repeated 853 times

/test json-file

richm · 2018-03-01T03:15:45Z

I think the fluentd-forward test may be getting confused - it may be doing the esid in both fluentd - @jcantrill take a look and see if you need to disable the esid plugin in the client fluentd or the server fluentd

richm · 2018-03-01T03:27:16Z

I see what the problem is - the test is failing here:
https://github.com/openshift/origin-aggregated-logging/blob/master/test/fluentd-forward.sh#L258

The test with FORWARDCNT=2 is testing that you can have a client fluentd sending to 2 server fluentd and each one of them will write the same record, but with a different id, to Elasticsearch. However, with esid, the id is generated in the client fluentd, which is passed through to the 2 servers, one of them gets through, the other is rejected with a duplicate id.

FAILURE after 1243.062s: test/fluentd-forward.sh:258: executing 'wait_for_fluentd_to_catch_up '' '' 2' expecting success: the command returned the wrong error code
Standard output from the command:
1

That is, the test is expecting 2 matching records, but there is only 1.

I think the best fix is to disable the esid plugin on the client fluentd. If that's not possible, then the next best is to create some sort filter to strip out the esid field from the record after generating it but before sending it to the matchs.

richm · 2018-03-01T15:12:15Z

test/fluentd-forward.sh

@@ -75,6 +75,10 @@ update_current_fluentd() {
    port 24284\n\
   </server>\n\
  </store>\n\
+  <filter **>\n\
+    @type record_modifier\n\
+    remove_keys _id\n\


should this be _openshift_es_genid ?

No I dont believe so. The gen plugin creates the id and stuffs it in a key that you define. It then uses that to populate the _id field which is what ES uses. We then tell it to remove the original key because apparently ES doesnt like that. In our case are not getting a duplicate record so we need to remove the '_id' field

Also, use a record_transformer filter instead of record_modifier

richm

/lgtm

jcantrill · 2018-03-01T16:44:47Z

/test logging

richm · 2018-03-01T17:30:59Z

test/fluentd-forward.sh

@@ -61,6 +61,10 @@ update_current_fluentd() {
      </match>' | oc replace -f -
        oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "add", "path": "/data/secure-forward1.conf", "#": "generated config file secure-forward1.conf" }]' 2>&1
        oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward1.conf", "value": "\
+  <filter **>\n\


Sorry, I missed this earlier - this won't work - you cannot have a <filter> inside of a <match> block - this oc patch is putting the patch inside a <match> block - you might need a separate oc patch in order to edit fluent.conf to add the filter

richm

/lgtm

jcantrill · 2018-03-01T21:12:23Z

/cherrypick release-3.9

openshift-cherrypick-robot · 2018-03-01T21:12:24Z

@jcantrill: once the present PR merges, I will cherry-pick it on top of release-3.9 in a new PR and assign it to you.

In response to this:

/cherrypick release-3.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jcantrill · 2018-03-01T21:12:37Z

/cherrypick release-3.7

openshift-cherrypick-robot · 2018-03-01T21:12:37Z

@jcantrill: once the present PR merges, I will cherry-pick it on top of release-3.7 in a new PR and assign it to you.

In response to this:

/cherrypick release-3.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jcantrill · 2018-03-01T21:12:48Z

/cherrypick release-3.6

openshift-cherrypick-robot · 2018-03-01T21:12:48Z

@jcantrill: once the present PR merges, I will cherry-pick it on top of release-3.6 in a new PR and assign it to you.

In response to this:

/cherrypick release-3.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-merge-robot · 2018-03-01T21:59:38Z

/test all [submit-queue is verifying that this PR is safe to merge]

openshift-merge-robot · 2018-03-01T23:34:54Z

Automatic merge from submit-queue.

openshift-cherrypick-robot · 2018-03-01T23:35:01Z

@jcantrill: new pull request created: #990

In response to this:

/cherrypick release-3.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2018-03-01T23:35:09Z

@jcantrill: new pull request created: #991

In response to this:

/cherrypick release-3.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2018-03-01T23:35:17Z

@jcantrill: new pull request created: #992

In response to this:

/cherrypick release-3.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…74-to-release-3.7 Automatic merge from submit-queue. [release-3.7] Generate uuid on ingestion to reduce duplicates This is an automated cherry-pick of #974 /assign jcantrill Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1556896

…74-to-release-3.6 Automatic merge from submit-queue. [release-3.6] Generate uuid on ingestion to reduce duplicates This is an automated cherry-pick of #974 /assign jcantrill https://bugzilla.redhat.com/show_bug.cgi?id=1556897

jcantrill added backport/3.6 backport/3.7 release/3.9 labels Feb 23, 2018

jcantrill requested a review from richm February 23, 2018 15:37

openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Feb 23, 2018

richm mentioned this pull request Feb 24, 2018

add bulk index rejection test #933

Merged

jcantrill force-pushed the 1548104 branch from 79e067f to 247258a Compare February 26, 2018 14:21

mffiedler reviewed Feb 26, 2018

View reviewed changes

jcantrill force-pushed the 1548104 branch from 247258a to ad65263 Compare February 26, 2018 20:43

ewolinetz reviewed Feb 27, 2018

View reviewed changes

jcantrill force-pushed the 1548104 branch from ad65263 to 6ea1b79 Compare February 28, 2018 15:38

openshift-ci-robot assigned richm Feb 28, 2018

richm approved these changes Feb 28, 2018

View reviewed changes

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 28, 2018

jcantrill added the backport/3.9 backport this PR to release-3.9 label Feb 28, 2018

jcantrill force-pushed the 1548104 branch from 6ea1b79 to cf53c91 Compare March 1, 2018 14:59

openshift-merge-robot removed the lgtm Indicates that a PR is ready to be merged. label Mar 1, 2018

richm reviewed Mar 1, 2018

View reviewed changes

jcantrill force-pushed the 1548104 branch from cf53c91 to fa77abd Compare March 1, 2018 15:19

jcantrill force-pushed the 1548104 branch 2 times, most recently from e38d36d to 550a6e2 Compare March 1, 2018 15:35

richm approved these changes Mar 1, 2018

View reviewed changes

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 1, 2018

richm reviewed Mar 1, 2018

View reviewed changes

bug 1548104. Generate record uuids upon ingestion to minimize duplicates

5e1e372

jcantrill force-pushed the 1548104 branch from 550a6e2 to 5e1e372 Compare March 1, 2018 18:55

openshift-merge-robot removed the lgtm Indicates that a PR is ready to be merged. label Mar 1, 2018

richm approved these changes Mar 1, 2018

View reviewed changes

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 1, 2018

openshift-merge-robot merged commit 588f756 into openshift:master Mar 1, 2018

openshift-cherrypick-robot mentioned this pull request Mar 1, 2018

[release-3.9] Generate uuid on ingestion to reduce duplicates #990

Merged

18 tasks

openshift-cherrypick-robot mentioned this pull request Mar 1, 2018

[release-3.7] Generate uuid on ingestion to reduce duplicates #991

Merged

openshift-cherrypick-robot mentioned this pull request Mar 1, 2018

[release-3.6] Generate uuid on ingestion to reduce duplicates #992

Merged

jcantrill deleted the 1548104 branch March 7, 2018 16:01

jcantrill added the backport/completed label Mar 22, 2018

Generate uuid on ingestion to reduce duplicates #974

Generate uuid on ingestion to reduce duplicates #974

jcantrill commented Feb 23, 2018

richm commented Feb 23, 2018

richm commented Feb 23, 2018

jcantrill commented Feb 23, 2018

richm commented Feb 24, 2018

jcantrill commented Feb 24, 2018

richm commented Feb 24, 2018 •

edited

mffiedler Feb 26, 2018

jcantrill Feb 26, 2018

ewolinetz Feb 27, 2018

jcantrill Feb 27, 2018

richm Feb 27, 2018

ewolinetz Feb 27, 2018

richm left a comment

jcantrill commented Feb 28, 2018

richm commented Feb 28, 2018

richm commented Mar 1, 2018

richm commented Mar 1, 2018

richm commented Mar 1, 2018

richm Mar 1, 2018

jcantrill Mar 1, 2018

richm Mar 1, 2018

richm left a comment

jcantrill commented Mar 1, 2018

richm Mar 1, 2018

richm left a comment

jcantrill commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

jcantrill commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

jcantrill commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

openshift-merge-robot commented Mar 1, 2018

openshift-merge-robot commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

Generate uuid on ingestion to reduce duplicates #974

Generate uuid on ingestion to reduce duplicates #974

Conversation

jcantrill commented Feb 23, 2018

richm commented Feb 23, 2018

richm commented Feb 23, 2018

jcantrill commented Feb 23, 2018

richm commented Feb 24, 2018

jcantrill commented Feb 24, 2018

richm commented Feb 24, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richm left a comment

Choose a reason for hiding this comment

jcantrill commented Feb 28, 2018

richm commented Feb 28, 2018

richm commented Mar 1, 2018

richm commented Mar 1, 2018

richm commented Mar 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richm left a comment

Choose a reason for hiding this comment

jcantrill commented Mar 1, 2018

Choose a reason for hiding this comment

richm left a comment

Choose a reason for hiding this comment

jcantrill commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

jcantrill commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

jcantrill commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

openshift-merge-robot commented Mar 1, 2018

openshift-merge-robot commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

openshift-cherrypick-robot commented Mar 1, 2018

richm commented Feb 24, 2018 •

edited