storage controller: make direct tenant creation more robust

[jcsp]

Problem

• Creations were not idempotent (unique key violation)
• Creations waited for reconciliation, which control plane blocks while an operation is in flight

Summary of changes

• Handle unique key constraint violation as an OK situation: if we're creating the same tenant ID and shard count, it's reasonable to assume this is a duplicate creation.
• Make the wait for reconcile during creation tolerate failures: this is similar to location_conf, where the cloud control plane blocks our notification calls until it is done with calling into our API (in future this constraint is expected to relax as the cloud control plane learns to run multiple operations concurrently for a tenant)

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2718 tests run: 2581 passed, 0 failed, 137 skipped (full report)

---

Flaky tests (2)

Postgres 15

• test_empty_branch_remote_storage_upload: debug

Postgres 14

• test_empty_branch_remote_storage_upload: debug

Code coverage* (full report)

• functions: 28.1% (6284 of 22346 functions)
• lines: 47.0% (44208 of 94097 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
7dcc293 at 2024-03-26T15:08:11.155Z :recycle:}