[v23.3.x] cluster: Avoid oversize allocs for topic creation and configuration

[vbotbuildovich]

Backport #16982 to branch v23.3.x. Note that this issue was created as a placeholder, since the original PR's commit(s) could not be automatically cherry-picked.
Here are the commands to execute:

git checkout v23.3.x
git checkout -b vbotbuildovich/backport-16982-v23.3.x-176
git cherry-pick -x f524df7dc255f65e3bb0e9d4d2805fea25d1d76a 12daecc409f885d046c7a47ed85b3a5b81907e8f 99247c562d9b0f263debb1678496902c996d845e d58c9fa751f676a8fa969f88040b9af95ff90530 9191f391b09909a32bb679abbc1a803a0d54de18

git push origin vbotbuildovich/backport-16982-v23.3.x-176
gh pr create \
  --title "[v23.3.x] cluster: Avoid oversize allocs for topic creation and configuration" \
  --base "v23.3.x" \
  --label "kind/backport" \
  --head "vbotbuildovich/backport-16982-v23.3.x-176" \
  --draft \
  --repo "redpanda-data/redpanda" \
  --milestone "v23.3.x-next" \
  --body "Backport of PR https://github.com/redpanda-data/redpanda/pull/16982 "