[v23.3.x] CI Failure (Can't apply spillover command) in CloudStorageScrubberTest.test_scrubber

[rockwotj]

https://buildkite.com/redpanda/redpanda/builds/43854#018d1991-427d-494a-8e2d-e4ee353675cc

Module: rptest.tests.cloud_storage_scrubber_test
Class:  CloudStorageScrubberTest
Method: test_scrubber
Arguments:
{
  "cloud_storage_type": 2
}

test_id:    rptest.tests.cloud_storage_scrubber_test.CloudStorageScrubberTest.test_scrubber.cloud_storage_type=CloudStorageType.ABS
status:     FAIL
run time:   2 minutes 18.259 seconds


    <BadLogLines nodes=docker-rp-8(24),docker-rp-18(25),docker-rp-19(24) example="ERROR 2024-01-17 23:02:29,437 [shard 1:main] cluster - ntp: {kafka/topic-msuoigbzdy/0} - archival_metadata_stm.cc:1457 - Can't apply spillover_cmd: {is_compacted: false, size_bytes: 11159010, base_offset: 2176, committed_offset: 2869, base_timestamp: {timestamp: 1705532451257}, max_timestamp: {timestamp: 1705532457953}, delta_offset: 39, ntp_revision: 24, archiver_term: 1, segment_term: 1, delta_offset_end: 56, sname_format: {v3}, metadata_size_hint: 3232}">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 142, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 1327, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-8(24),docker-rp-18(25),docker-rp-19(24) example="ERROR 2024-01-17 23:02:29,437 [shard 1:main] cluster - ntp: {kafka/topic-msuoigbzdy/0} - archival_metadata_stm.cc:1457 - Can't apply spillover_cmd: {is_compacted: false, size_bytes: 11159010, base_offset: 2176, committed_offset: 2869, base_timestamp: {timestamp: 1705532451257}, max_timestamp: {timestamp: 1705532457953}, delta_offset: 39, ntp_revision: 24, archiver_term: 1, segment_term: 1, delta_offset_end: 56, sname_format: {v3}, metadata_size_hint: 3232}">

[rockwotj]

Relevant log lines:

WARN  2024-01-17 23:02:29,618 [shard 1:main] cloud_storage - partition_manifest.cc:1274 - {kafka/topic-msuoigbzdy/0} Can't apply spillover manifest because the start offsets are not aligned: 2108 vs 2176, {o=2176-2869 t={timestamp: 1705532451257}-{timestamp: 1705532457953}}
ERROR 2024-01-17 23:02:29,618 [shard 1:main] cluster - ntp: {kafka/topic-msuoigbzdy/0} - archival_metadata_stm.cc:1457 - Can't apply spillover_cmd: {is_compacted: false, size_bytes: 11159010, base_offset: 2176, committed_offset: 2869, base_timestamp: {timestamp: 1705532451257}, max_timestamp: {timestamp: 1705532457953}, delta_offset: 39, ntp_revision: 24, archiver_term: 1, segment_term: 1, delta_offset_end: 56, sname_format: {v3}, metadata_size_hint: 3232}

[rockwotj]

This was hit in a PR that added in a backport for scheduling groups for Wasm, and I'm not sure how that could introduce this: #16139

[piyushredpanda]

Not seen in at least two months, closing