Only allocate SNAT IP when there is no ephemeral or floating IP

[bnaecker]

In the instance-creation saga, we unconditionally allocate an SNAT IP for every instance. This is regardless of whether the request explicitly asked for ephemeral addresses. In that case, OPTE always uses the ephemeral address anyway, and no network traffic will ever flow through the SNAT IP. We ought to make such allocation optional, and only provide one if no ephemeral addresses were requested.

[bnaecker]

Related to #2715. We should ideally also move the SNAT address allocation and release to the instance start saga and (maybe a new?) stop saga.

[david-crespo]

Wonder if the fact that you can now detach and attach ephemeral IPs changes things here. If you start with an ephemeral IP and no SNAT IP, then you detach the ephemeral IP, you have nothing.

[david-crespo]

Based on the discussion in #5043, we should:

• At instance create, only allocate a SNAT IP (from default pool) if there is no ephemeral or floating IP
• When the user has detached all ephemeral and floating IPs, allocate a SNAT IP (from default pool)
• When an ephemeral or floating IP is attached, remove SNAT IP if there is one

None of that sounds too hard. One wrinkle is that we'd be introducing a new way that detaching a floating or ephemeral IP can fail, because if there is no default pool or the default pool is full, then allocating the SNAT IP at detach time will fail and we probably shouldn't allow that. Fortunately the lack of a default pool means it's very hard to do anything, so we should consider that just a Bad Configuration among many possible bad configurations operators must endeavor to avoid. Default pool being full is also generally a bad situation — you can't create new instances either — so it's not that weird if a few other operations are hamstrung by that.

[david-crespo]

We're not getting this into v9.