CI Failure (BadLogLines: Failed to query spillover manifests: cloud_storage::error_outcome:10, query: {{timestamp: 1704785344565}}) in CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves

[andijcr]

https://buildkite.com/redpanda/redpanda/builds/43586

Test	Description	Time	Data	Summary	Detail

Module: rptest.tests.cloud_storage_timing_stress_test
Class:  CloudStorageTimingStressTest
Method: test_cloud_storage_with_partition_moves
Arguments:
{
  "cleanup_policy": "delete"
}

	

<BadLogLines nodes=docker-rp-32(1) example="ERROR 2024-01-09 07:29:40,772 [shard 1:main] cloud_storage - [fiber137 kafka/test-topic/0] - remote_partition.cc:551 - Failed to query spillover manifests: cloud_storage::error_outcome:10, query: {{timestamp: 1704785344565}}">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 142, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 1327, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-32(1) example="ERROR 2024-01-09 07:29:40,772 [shard 1:main] cloud_storage - [fiber137 kafka/test-topic/0] - remote_partition.cc:551 - Failed to query spillover manifests: cloud_storage::error_outcome:10, query: {{timestamp: 1704785344565}}">

[andrwng]

May be related to #15042, which also notes a bug in timequeries to the spillover region

[rockwotj]

https://buildkite.com/redpanda/redpanda/builds/43966#018d22e3-62fa-4677-9aaf-d87e7853adf1

[savex]

To add to the thread, recent fails here: https://buildkite.com/redpanda/redpanda/builds/44312#_
3 out of 6 with

rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-1(2) 
example="ERROR 2024-01-26 00:14:30,170 [shard 1:main] cloud_storage - [fiber541 kafka/test-topic/0] - 
remote_partition.cc:551 - Failed to query spillover manifests: cloud_storage::error_outcome:10, 
query: {{timestamp: 1706228039913}}">