Storage: Add mode reconciliation for modes 1 and 2

[suntala]

What is this feature?

Adds foundation for reconciling current dual writing mode with desired dual writing mode. In particular, adds support for setting current dual writing mode and reconciliation functionality to go from mode 1 to mode 2.

Why do we need this feature?

Required to implement proposal 4 in https://docs.google.com/document/d/1_FZ75QYoBqAW_7pC64fPN5GPUbHyRIWmemcO9SDhBJM/edit#heading=h.5sybau7waq2q

Who is this feature for?

US users, developers

Which issue(s) does this PR fix?:

Relates to https://github.com/grafana/search-and-storage-team/issues/16

Special notes for your reviewer:

Please check that:

• It works as expected from a user's perspective.
• If this is a pre-GA feature, it is behind a feature toggle.
• The docs are updated, and if this is a notable improvement, it's added to our What's New doc.

[diegommm]

Question: have you checked if we can modify this signature to have the first argument be a context.Context? I think it would require regenerating the wire files. In case it doesn't work:

1. It may be because we don't yet have a wire provider return a context.Context. We could try to add a wire provider that provides a context.Context, simply returning context.Background() for now.
2. In the worst case (e.g. if the above is too much work, or unrelated, etc.), we could just pass context.Background() to SetDualWritingMode here.

The idea is that we push the need for a proper context.Context as far above in dependencies as possible, so that when we take the step to properly providing a context.Context the refactor is not so big.

[suntala]

Thanks for the review! That makes sense about pushing the need for the context.Context up. Ended up going with option 2 for now.

[DanCech]

we should probably make sure that toMode[m] exists and default to mode1 if it doesn't

[suntala]

Good point. Added a check in dc147b9.