Nodetool scrub validator fails when logs are delayed

[soyacz]

When db node logs are delayed, nodetool scrub validator fails with

2024-05-12 23:41:09.356: (ValidatorEvent Severity.ERROR) period_type=one-time event_id=ef632a24-f02f-464e-85d5-931882c8329a during_nemesis=DecommissionStreamingErr,CreateIndex: message=No scrubbing validation message found in db logs on node: longevity-parallel-topology-schema--db-node-2a3bb187-44

Mentioned messages are present in log, but after several seconds (in sct.log).

OS / Image: ami-0b7480423a402aa95 (aws: undefined_region)

Test: longevity-schema-topology-changes-12h-test
Test id: 2a3bb187-5efb-4f8d-87a8-9508cb0a56cf
Test name: scylla-master/tier1/longevity-schema-topology-changes-12h-test
Test config file(s):

• longevity-parallel-schema-changes-12h.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor 2a3bb187-5efb-4f8d-87a8-9508cb0a56cf
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs 2a3bb187-5efb-4f8d-87a8-9508cb0a56cf

Logs:

• db-cluster-2a3bb187.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2a3bb187-5efb-4f8d-87a8-9508cb0a56cf/20240513_000730/db-cluster-2a3bb187.tar.gz
• sct-runner-events-2a3bb187.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2a3bb187-5efb-4f8d-87a8-9508cb0a56cf/20240513_000730/sct-runner-events-2a3bb187.tar.gz
• sct-2a3bb187.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2a3bb187-5efb-4f8d-87a8-9508cb0a56cf/20240513_000730/sct-2a3bb187.log.tar.gz
• loader-set-2a3bb187.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2a3bb187-5efb-4f8d-87a8-9508cb0a56cf/20240513_000730/loader-set-2a3bb187.tar.gz
• monitor-set-2a3bb187.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2a3bb187-5efb-4f8d-87a8-9508cb0a56cf/20240513_000730/monitor-set-2a3bb187.tar.gz

Jenkins job URL
Argus