cloud_storage/ducktape: allow bad logs in scope

[abhijat]

A utility is added which allows logs to be ignored in a specific time frame.

The allow list is added to the cluster decorator:

@cluster(log_allow_list=[..., TimedLogAllowList(r'message123')]

Then within that test:

# message123 errors not ignored here

with bad_logs_allowed('message123'):
    # message123 errors ignored here

# message123 errors not ignored here

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.2.x
• v23.1.x
• v22.3.x

Release Notes

• none

[abhijat]

/ci-repeat 10
skip-units
dt-repeat=100
tests/rptest/tests/cloud_storage_scrubber_test.py::CloudStorageScrubberTest.test_scrubber

[abhijat]

/ci-repeat 10
skip-units
dt-repeat=100
tests/rptest/tests/cloud_storage_scrubber_test.py::CloudStorageScrubberTest.test_scrubber

[abhijat]

leaving this PR up as the tooling could be useful, but removed the changes for #14132 because it might need change in the C++ code.

[nvartolomei]

Mixed feelings. Tests that make assumptions about time/clocks are generally flaky and should be avoided. This one makes even more assumptions. I.e. that clocks are synchronised to millisecond precision across machines (in CDT ducktape runs on distinct machine from Redpanda). Truncating time and adding tolerances I think will make it just worse. I'd prefer the following alternatives over this one:

1. Ignore specific error loglines for the entire test
2. Change log level to WARN, quoting our internal logging guide (Logging in Redpanda Core on the wiki): "Issues with external systems (S3, Kafka Clients) are not considered errors: they are warnings."

[abhijat]

Mixed feelings. Tests that make assumptions about time/clocks are generally flaky and should be avoided. This one makes even more assumptions. I.e. that clocks are synchronised to millisecond precision across machines (in CDT ducktape runs on distinct machine from Redpanda). Truncating time and adding tolerances I think will make it just worse. I'd prefer the following alternatives over this one:

1. Ignore specific error loglines for the entire test
2. Change log level to WARN, quoting our internal logging guide (Logging in Redpanda Core on the wiki): "Issues with external systems (S3, Kafka Clients) are not considered errors: they are warnings."

Sorry, I missed the review. The idea here was to add some sort of tolerance to a specific error in a scope, which is unfortunately tied to time. This is useful in cases where for example we toggle a feature on and off which causes an error.

So this is better than the alternative, instead of saying the error is acceptable for the entire run of the test, we are saying it is only acceptable for a shorter time span. In worst case scenario we may get false negatives due to the time differences.

However I do not see the utility of this any more as the original issue will not be served by this change, so I will close the PR.