is `copilot task run --generate-cmd` compatible with environments in other AWS accounts?

[codekitchen]

I did a first test run of copilot all in one AWS account. Now I'm working on deploying our application for real on top of copilot, with a "primary" account that just holds the copilot application resources, and then a separate AWS account per copilot environment.

In my initial testing, I could do copilot task run --generate-cmd myapp/staging/api to get the parameters necessary to run pre-deploy db migrations. However, now that I've spun things up in this multi-account structure I get an error doing this:

✘ generate task run command from service api of application myapp deployed in environment staging: retrieve network configuration for service api: no cluster found in environment staging

Is this an issue because my application, and the CLI profile I am using, are now in a different AWS account from the actual staging environment resources? If not, I'm not sure what else to do to debug, any pointers there would be great. Thanks.

[codekitchen]

I'm actually having trouble coming up with a workaround for this that doesn't involve bypassing copilot completely. Hopefully I'm not just missing something.

If I try to craft my own copilot task run command I run into one of two scenarios:

1. If I pass --app and --env parameters, the task runs in the correct ECS cluster but it doesn't join the security groups and subnets necessary to communicate with my RDS db. And I can't pass --security-groups or --subnets if I pass --app and --env.
2. If I skip using --app and --env and specify everything manually using --subnets, --security-groups, --cluster, --task-role, and --execution-role: copilot tries to run the task in the application AWS account rather than the environment AWS account, so creating the task fails.

[Lou1415926]

Hi @codekitchen ! Thank you for reporting the issue.

Is this an issue because my application, and the CLI profile I am using, are now in a different AWS account from the actual staging environment resources? If not, I'm not sure what else to do to debug, any pointers there would be great. Thanks.

I believe this is a bug. Basically Copilot is using the wrong profile to retrieve network configuration information, therefore the cluster is not found. I am sorry for the churn!

If I skip using --app and --env and specify everything manually using --subnets, --security-groups, --cluster, --task-role, and --execution-role: copilot tries to run the task in the application AWS account rather than the environment AWS account, so creating the task fails.

This approach should work - you just need to run copilot task run using your environment profile.

When you do not specify --app and --env, Copilot uses the default AWS profile (which is likely the account used for your app) to spin up the task. This is why the task is run in the application account.

You can switch to your environment account in your terminal, and then run copilot task run. Copilot should then run the tasks in the environment account.

Here is a recommended way of switching account. Basically, it's running export AWS_PROFILE=my-environment-account.

Let me know if this works for you! Thank you again for reporting this issue!

[codekitchen]

This approach should work - you just need to run copilot task run using your environment profile.

Oh ha, thank you for pointing that out. I was able to successfully run our migrate task after using a profile for the staging environment account.

That does make me wonder, what do you think the fix is here? Our current plan is to not give our devs user accounts in each environment account, and have them just interact with the environments via the roles that copilot sets up in the application account. If my mental model is correct, It seems like making that work will require a new flag to task run in order to specify that copilot should assume its role in the environment account, like how it does for other operations inside an env (other than env init/delete).

[Lou1415926]

That does make me wonder, what do you think the fix is here?

I already sent out a fix for the bug that you were encountering! This fixes the bug that gives you the error you saw in the first place ✘ generate task run command from service api of application myapp deployed in environment staging: retrieve network configuration for service api: no cluster found in environment staging.

If my mental model is correct, It seems like making that work will require a new flag to task run in order to specify that copilot should assume its role in the environment account, like how it does for other operations inside an env (other than env init/delete).

This raises a very good point about the workflow of copilot task run --generate-cmd! Do you need additional subnets configuration for your task, other than the public subnets created in your environment VPC? If not, my proposal in #3100 might probably solve your issue. You'd be able to run copilot --app --env --security-groups instead of copilot --security-groups --subnets --cluster. This way, Copilot would be able to automatically assume the environment manager role, and you won't need to switch the profile.

[codekitchen]

Thanks! I believe that in this specific case, being able to add --security-groups would be enough. I wouldn't want to have to look up those security groups IDs manually for each environment. But I think I could run --generate-cmd to get the security group ids, and then pass those to the --app --env version of the command. So your proposal in #3100 should work for us here.

Really what I think I want is "Option 3" from the original proposal here: #2159 but I know the decision was made to implement --generate-cmd instead. I feel like "Option 3" would be a lot easier to understand and interact with for my use case here.

[Lou1415926]

Exactly - I made a separate proposal in #3103 that generates a command that uses --app --env flags (instead of the current --subnets --cluster version), along with the --security-groups flag. With this proposal, you could just copy & paste, instead of having to look up the security group IDs and manually construct a command with the --app and --env version. Hope this would improve the experience of --generate-cmd!

[Lou1415926]

I already sent out a fix for the bug that you were encountering! This fixes the bug that gives you the error you saw in the first place ✘ generate task run command from service api of application myapp deployed in environment staging: retrieve network configuration for service api: no cluster found in environment staging.

This fix is now released 🎉 in v1.14.0 https://github.com/aws/copilot-cli/releases/tag/v1.14.0 !

[codekitchen]

Since this bug is fixed (thanks again!) I'm going to close this in favor of further discussion on #3100 and #3103