[system testing] Add provision for system testing negative / false-positive scenarios in packages

[bhapas]

This PR changes the way the system testing asserts at the number of hits in ElasticSearch when a system test is run.

Earlier, The testrunner exited when enough docs are found as per the config and there were more docs that got ingested after assertHitCount condition satisfied.

Now, The testrunner tries to get all the docs before checking for the assertHitCount and fails if the docs are more/less than the configured count

The retryTicker in the waitUntilTrue function is changed to 5 seconds meaning , the retry interval for each poll to ES has been changed to 5 seconds and so if for 10 seconds, there is no change in number of docs , the test passes.

How to test

Run PACKAGE_TEST_TYPE=false_positives PACKAGE_UNDER_TEST=httpjson_false_positive_asserts ./scripts/test-check-packages-false-positives.sh

The test-check-packages.sh file has been modified to catch a failure from the new package httpjson_false_positive_asserts and a testcase failure is considered as success scenario

• Fixes Reduce false negatives with assert.hit_count #1308

[jsoriano]

test integrations

[jsoriano]

Nice to have the first negative test 🙂

I think these changes are going to affect the timings of all tests, and not only the ones with hit count assertions, could we change it so these changes only affect tests with hit count assertions?

[mrodm]

test integrations

[elasticmachine]

Created or updated PR in integrations repostiory to test this vesrion. Check elastic/integrations#6778

[bhapas]

test integrations

[elasticmachine]

Created or updated PR in integrations repostiory to test this vesrion. Check elastic/integrations#6778

Changes I requested have been addressed.

[bhapas]

test integrations

[elasticmachine]

Created or updated PR in integrations repostiory to test this vesrion. Check elastic/integrations#6778

[bhapas]

/test

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c2d77cb

History

• 💚 Build #1144 succeeded 30bf3b0
• 💔 Build #1143 failed 08933fe
• 💔 Build #1142 failed 51e33f4
• 💚 Build #1136 succeeded d486799
• 💚 Build #1129 succeeded 8deca24
• 💔 Build #1128 failed 8deca24

cc @bhapas

[mrodm]

👍 Thanks for adding this negative tests @bhapas!