[v23.2.x] log_manager::housekeeping_scan iteration with suspension point fix

[andijcr]

Backport of PR #13615

Fixes #13830

[andijcr]

https://buildkite.com/redpanda/redpanda/builds/38169#018af535-9f5c-459f-8e51-c9152ae185eb
failure is

====================================================================================================
test_id:    rptest.tests.cloud_storage_chunk_read_path_test.CloudStorageChunkReadTest.test_read_chunks
status:     FAIL
run time:   49.713 seconds


    MetricCheckFailed('vectorized_cloud_storage_partition_chunk_size', 0.0, 0.0)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 82, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/cloud_storage_chunk_read_path_test.py", line 198, in test_read_chunks
    m.expect([(metric, lambda a, b: a < b <= 2 * self.default_chunk_size)])
  File "/root/tests/rptest/services/metrics_check.py", line 135, in expect
    raise error
rptest.services.metrics_check.MetricCheckFailed: MetricCheckFailed<vectorized_cloud_storage_partition_chunk_size old=0.0 new=0.0>

And it's new. I will ask @abhijat for input on this.

in the meantime, the next force push is to fix a merge conflict