[v23.2.x] log_manager::housekeeping_scan iteration with suspension point fix

[vbotbuildovich]

Backport #13615 to branch v23.2.x. Note that this issue was created as a placeholder, since the original PR's commit(s) could not be automatically cherry-picked.
Here are the commands to execute:

git checkout v23.2.x
git checkout -b vbotbuildovich/backport-13615-v23.2.x-855
git cherry-pick -x 15d2d9fed95124263303d9dcb6f4e485ee08c003 cf50cefada95c67ee194438f22db70642b2ce4b4 9a45e40f7cbfc96b1000fd6354424249738bc4b4 6eb57258ae74e92e846c94ef6cceb7ee733ecec2 ab5a2104b4a9b46c68eb7087de9f7cd97d74da5e a21446fb66cae7110ca2fd7fe2176061275eccdd dfed2da71aa728f16e9de14a2cf5805d4045c383

git push origin vbotbuildovich/backport-13615-v23.2.x-855
gh pr create \
  --title "[v23.2.x] log_manager::housekeeping_scan iteration with suspension point fix" \
  --base "v23.2.x" \
  --label "kind/backport" \
  --head "vbotbuildovich/backport-13615-v23.2.x-855" \
  --draft \
  --repo "redpanda-data/redpanda" \
  --reviewer "dotnwat,ballard26" \
  --milestone "v23.2.x-next" \
  --body "Backport of PR https://github.com/redpanda-data/redpanda/pull/13615 "