mon: The mon daemons maintain host network settings to allow change in config

[travisn]

Description of your changes:
If the host network setting is changed in the cluster CR, the existing mons must continue using the same network settings or else the operator would update their pod specs with the incorrect settings and cause mon quorum to go down. Now the network setting is preserved so the admin could switch between host and non-host network configuration without reinstalling.

Which issue is resolved by this Pull Request:
Resolves #11121

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide).
• Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[BlaineEXE]

I think this looks good. It's not explicit here: does this automatically fail over mons to the newest network config, or does it still keep the mons running using the old config?

[travisn]

I think this looks good. It's not explicit here: does this automatically fail over mons to the newest network config, or does it still keep the mons running using the old config?

It won’t automatically failover the mons. I'll add a note to the cluster CR doc that the mon failover would need to be triggered to complete the conversion.