Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add bulk index rejection test #933

Merged
merged 1 commit into from Mar 1, 2018

Conversation

richm
Copy link
Contributor

@richm richm commented Feb 8, 2018

This tests the ability of fluentd to handle bulk index rejections.
The test starts up several curl processes which hammer Elasticsearch
with bulk index requests which causes fluentd to receive bulk rejections.
The methodology is to add some messages for both projects and operations
indices which should be rejected and queued and eventually resent
successfully.
The test will ensure that Elasticsearch is rejecting bulk requests and
fluentd is receiving bulk rejections.

@openshift-ci-robot openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 8, 2018
@richm
Copy link
Contributor Author

richm commented Feb 8, 2018

/test

@richm
Copy link
Contributor Author

richm commented Feb 8, 2018

@mffiedler @lukas-vlcek

@richm
Copy link
Contributor Author

richm commented Feb 8, 2018

@richm
Copy link
Contributor Author

richm commented Feb 8, 2018

The motivation for doing this now is https://bugzilla.redhat.com/show_bug.cgi?id=1541429

@nhosoi
Copy link
Contributor

nhosoi commented Feb 8, 2018

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 8, 2018
@nhosoi
Copy link
Contributor

nhosoi commented Feb 8, 2018

/retest

@richm
Copy link
Contributor Author

richm commented Feb 8, 2018

hmm - looks like bulk index rejection test is interfering with mux test?

@richm
Copy link
Contributor Author

richm commented Feb 9, 2018

/retest

@richm
Copy link
Contributor Author

richm commented Feb 10, 2018

/retest
flakes in forward and mux tests???

The json-file test flake in mux is very strange. It looks like an extra n character is being added to the messages?

"message": "67108864 is less then the default 128m.  Setting to default.n",
"message": "2018-02-09 23:12:39 +0000 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or notn",

here is an example of a raw log message from a file:

/var/log/containers/logging-kibana-1-tvhbm_logging_kibana-962e091dcc81d53f2231df604c6c3469db2af890781a977031fbd21b3de03b7e.log:{"log":"{"type":"response","@timestamp":"2018-02-09T23:13:13Z","tags":[],"pid":66,"method":"get","statusCode":404,"req":{"url":"/bd0cf33bdd464f9f924a47c80bc877f5","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":404,"responseTime":2,"contentLength":9},"message":"GET /bd0cf33bdd464f9f924a47c80bc877f5 404 2ms - 9.0B"}n","stream":"stdout","time":"2018-02-09T23:13:13.559006504Z"}

notice the extra n after the closing }
wtf? docker --log-driver=json-file bug?

@portante
Copy link
Contributor

portante commented Feb 10, 2018 via email

@richm
Copy link
Contributor Author

richm commented Feb 10, 2018

Could that be a \n that was process as “escape the n” leaving it as a ‘n’?

Yes, looks like that's it. So, probably false alarm. Looks like there is a bug in artifact_out - it strips off the \

@richm richm added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 10, 2018
@richm richm removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 13, 2018
@openshift-merge-robot openshift-merge-robot removed the lgtm Indicates that a PR is ready to be merged. label Feb 13, 2018
@richm richm added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 13, 2018
@richm
Copy link
Contributor Author

richm commented Feb 14, 2018

/retest

1 similar comment
@richm
Copy link
Contributor Author

richm commented Feb 15, 2018

/retest

@richm
Copy link
Contributor Author

richm commented Feb 27, 2018

/retest

1 similar comment
@jcantrill
Copy link
Contributor

/retest

This tests the ability of fluentd to handle bulk index rejections.
The test starts up several curl processes which hammer Elasticsearch
with bulk index requests which causes fluentd to receive bulk rejections.
The methodology is to add some messages for both projects and operations
indices which should be rejected and queued and eventually resent
successfully.
The test will ensure that Elasticsearch is rejecting bulk requests and
fluentd is receiving bulk rejections.
The test has to run last because it will cause other CI tests to have
test flakes due to bulk index rejections, which somehow become
persistent after this test runs.
@jcantrill
Copy link
Contributor

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 1, 2018
@jcantrill jcantrill added the backport/3.9 backport this PR to release-3.9 label Mar 1, 2018
@openshift-merge-robot
Copy link
Contributor

/test all [submit-queue is verifying that this PR is safe to merge]

@openshift-merge-robot
Copy link
Contributor

Automatic merge from submit-queue.

@openshift-merge-robot openshift-merge-robot merged commit b5a8649 into openshift:master Mar 1, 2018
@richm richm deleted the bulk-rejection-test branch March 1, 2018 18:00
@richm
Copy link
Contributor Author

richm commented Mar 1, 2018

/cherrypick es5.x

@openshift-cherrypick-robot

@richm: #933 failed to apply on top of branch "es5.x":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	test/zzz-correct-index-names.sh
Falling back to patching base and 3-way merge...
Auto-merging test/zzz-correct-index-names.sh
CONFLICT (content): Merge conflict in test/zzz-correct-index-names.sh
Patch failed at 0001 add bulk index rejection test

In response to this:

/cherrypick es5.x

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@richm
Copy link
Contributor Author

richm commented Mar 1, 2018

/cherrypick release-3.9

@openshift-cherrypick-robot

@richm: #933 failed to apply on top of branch "release-3.9":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	test/zzz-correct-index-names.sh
Falling back to patching base and 3-way merge...
Auto-merging test/zzz-correct-index-names.sh
CONFLICT (content): Merge conflict in test/zzz-correct-index-names.sh
Patch failed at 0001 add bulk index rejection test

In response to this:

/cherrypick release-3.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@richm
Copy link
Contributor Author

richm commented Mar 1, 2018

/cherrypick release-3.8

@openshift-cherrypick-robot

@richm: #933 failed to apply on top of branch "release-3.8":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	hack/testing/util.sh
M	test/zzz-correct-index-names.sh
Falling back to patching base and 3-way merge...
Auto-merging test/zzz-correct-index-names.sh
CONFLICT (content): Merge conflict in test/zzz-correct-index-names.sh
Auto-merging hack/testing/util.sh
Patch failed at 0001 add bulk index rejection test

In response to this:

/cherrypick release-3.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@richm
Copy link
Contributor Author

richm commented Mar 1, 2018

/cherrypick release-3.7

@openshift-cherrypick-robot

@richm: #933 failed to apply on top of branch "release-3.7":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	hack/testing/util.sh
M	test/zzz-correct-index-names.sh
Falling back to patching base and 3-way merge...
Auto-merging test/zzz-correct-index-names.sh
CONFLICT (content): Merge conflict in test/zzz-correct-index-names.sh
Auto-merging hack/testing/util.sh
Patch failed at 0001 add bulk index rejection test

In response to this:

/cherrypick release-3.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@richm
Copy link
Contributor Author

richm commented Mar 6, 2018

/cherrypick release-3.9

@openshift-cherrypick-robot

@richm: new pull request created: #997

In response to this:

/cherrypick release-3.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@richm
Copy link
Contributor Author

richm commented Mar 6, 2018

/cherrypick release-3.8

@openshift-cherrypick-robot

@richm: #933 failed to apply on top of branch "release-3.8":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	hack/testing/util.sh
M	test/zzz-correct-index-names.sh
Falling back to patching base and 3-way merge...
Auto-merging test/zzz-correct-index-names.sh
CONFLICT (content): Merge conflict in test/zzz-correct-index-names.sh
Auto-merging hack/testing/util.sh
Patch failed at 0001 add bulk index rejection test

In response to this:

/cherrypick release-3.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@richm
Copy link
Contributor Author

richm commented Mar 6, 2018

/cherrypick release-3.7

@richm
Copy link
Contributor Author

richm commented Mar 6, 2018

/cherrypick release-3.6

@openshift-cherrypick-robot

@richm: #933 failed to apply on top of branch "release-3.7":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	hack/testing/util.sh
M	test/zzz-correct-index-names.sh
Falling back to patching base and 3-way merge...
Auto-merging test/zzz-correct-index-names.sh
CONFLICT (content): Merge conflict in test/zzz-correct-index-names.sh
Auto-merging hack/testing/util.sh
Patch failed at 0001 add bulk index rejection test

In response to this:

/cherrypick release-3.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-cherrypick-robot

@richm: #933 failed to apply on top of branch "release-3.6":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	hack/testing/util.sh
M	test/zzz-correct-index-names.sh
Falling back to patching base and 3-way merge...
Auto-merging test/zzz-correct-index-names.sh
CONFLICT (content): Merge conflict in test/zzz-correct-index-names.sh
Auto-merging hack/testing/util.sh
Patch failed at 0001 add bulk index rejection test

In response to this:

/cherrypick release-3.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-merge-robot added a commit that referenced this pull request Mar 14, 2018
…33-to-release-3.9

Automatic merge from submit-queue.

[release-3.9] add bulk index rejection test

This is an automated cherry-pick of #933

/assign richm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport/3.7 backport/3.8 backport/3.9 backport this PR to release-3.9 component/fluentd lgtm Indicates that a PR is ready to be merged. release/3.9 size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants