Can't change to spot instances

[bfeliano]

Hello team.

I have a service running on Fargate and I'm trying to change it to fargate_spot.

I changed my service manifest from

environments:
  production:
    count: 1

to

environments:
  production:
    count: 
      range:
        min: 2
        max: 3
        spot_from: 2 
      memory_percentage: 80 

When trying to run a copilot svc deploy it returns an error when trying to update the service.

The following resource(s) failed to update: [Service].

Resource handler returned message: "Invalid request provided: CreateService error: SC service is already used by arn:aws:servicediscovery:us-east-1:ACCOUNT:namespace/ns-xxx (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: 25213558-e374-4986-b4d2-a19eff2ae660; Proxy: null)" (RequestToken: xxxx, HandlerErrorCode: InvalidRequest)

[huanjani]

Hello, @bfeliano!

It looks like you're encountering a bug involving Service Connect + Fargate Spot that ECS is actively working on. ☹️ I'm sorry about the inconvenience. For context see #4337 (comment) and #4313 (comment). As noted in those, the current workaround is to turn off/opt out of Service Connect by adding

network:
  connect: false

to your workload manifest.

Let us know if that works for you!

Thanks and sorry again.

[bfeliano]

Hey Janice, that did the trick. Many thanks.
I'm sorry I missed it from previous issues.

[hailwood]

Hi @huanjani

Do we have a workaround for if we do require service connect?
We deploy two services, one being our request handlers that we want to be spot instances and the other being a caching instance that all the request handlers use to communicate with. So as far as I am aware we need connect: true so that our request handlers can communicate with the caching instance?

[Lou1415926]

@hailwood Copilot automatically enables "ServiceDiscovery" for each of your services, so your instances should be able to use that, instead of service connect, to communicate.

To find the endpoint for each of your service, you can run copilot svc show. For example,

$ copilot svc show --name handler
...
Internal Service Endpoint

  Endpoint                     Environment  Type
  --------                     -----------  ----
  handler.test.myapp.local:80  test         Service Discovery. <--- THIS

Then, in your application code, replace the service connect endpoint with this.

Hope this helps!

[hailwood]

Thanks @Lou1415926,

When working correctly is Service Connect recommended over Service Discovery?
We got it working in the end by destroying the service and redeploying, but are open to switching to Service Discovery if that's recommended instead.

[iamhopaul123]

Hello @hailwood. The comparison between service discovery and service connect is summarized here. And we recommend Service Connect more.

[github-actions]

This issue is stale because it has been open 60 days with no response activity. Remove the stale label, add a comment, or this will be closed in 14 days.

[github-actions]

This issue is closed due to inactivity. Feel free to reopen the issue if you have any further questions!