log_manager: gate orphan file cleanup

[bharathv]

Fixes the broken promise resulting from a race during shutdown.

Fixes #17614

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.3.x
• v23.2.x

Release Notes

• none

[bharathv]

Test failure is gtest_raft_rpunit, unrelated known issue.

[vbotbuildovich]

/backport v23.3.x

[vbotbuildovich]

/backport v23.2.x

[vbotbuildovich]

Failed to create a backport PR to v23.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-17910-v23.3.x-894 remotes/upstream/v23.3.x
git cherry-pick -x d892c64c2b467ab989bc54b7b1bb07031a86b001 91fb2c2ddaa01ffbf4dd3c2094665d037f5277dc

Workflow run logs.

[vbotbuildovich]

Failed to create a backport PR to v23.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-17910-v23.2.x-215 remotes/upstream/v23.2.x
git cherry-pick -x d892c64c2b467ab989bc54b7b1bb07031a86b001 91fb2c2ddaa01ffbf4dd3c2094665d037f5277dc

Workflow run logs.