cloud.Service updates should wait for services to be stable before completing

[lukehoban]

Our current semantics for updating a cloud.Service are that the update completes once the cluster has started the process of transitioning from the old to the new copy of the service.

This is different than Functions, for which we wait until the new code is deployed and is guaranteed to be hit on future calls.

This has lead to us needing to sprinkle aws ecs wait services-stable calls after every pulumi update in places where we care about being sure that we are hitting the new copy of the service. That doesn't seem like an acceptable answer for customers.

I believe our semantics for an update should always be that the requested new state is fully live, and the old state is fully removed, at the point that we indicate an update (or step of an update) has completed.

This is true in everything else we project as far as I'm aware, but for some reason the TF provider for ECS services decided to not align with this principle. Notably, CloudFormation does wait on service ready (and in fact rolls back the whole update if the service can't reach a stable state passing health checks in the new state).

Since the TF provider doesn't provide the behaviour we want here - we have two options:

1. Patch and upstream the TF provider with the ability to request that updates wait for stable
2. Add our own dynamic provider to orchestrate this as a separate step in our graph.

Making this change will lead to Service updates taking longer - probably in the neighborhood of 5 minutes. But this represents the reality that the update really does take that long.

[mmdriley]

Closest issue upstream: hashicorp/terraform-provider-aws#3107

[mmdriley]

hashicorp/terraform-provider-aws#3485

[joeduffy]

Rock star! Awesome to see you doing this upstream, @mmdriley.