Support recreate for ECS tasks #4608

khanhtc1202 · 2023-10-06T09:31:15Z

What this PR does / why we need it:

Support recreate ECS task while deploying. In such case, we need to configure the application config file like this

apiVersion: pipecd.dev/v1beta1
kind: ECSApp
spec:
  quickSync:
    recreate: true

Which issue(s) this PR fixes:

Part of #4609

Does this PR introduce a user-facing change?:

How are users affected by this change: Add option to keep the ECS task singleton while deploying
Is this breaking change:
How to migrate (if breaking change):

Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com>

codecov · 2023-10-06T09:36:41Z

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (3809d60) 30.00% compared to head (20cfb77) 29.98%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4608      +/-   ##
==========================================
- Coverage   30.00%   29.98%   -0.02%     
==========================================
  Files         221      221              
  Lines       25955    25955              
==========================================
- Hits         7787     7783       -4     
- Misses      17519    17523       +4     
  Partials      649      649

Files	Coverage Δ
pkg/config/application_ecs.go	`9.52% <ø> (ø)`

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nghialv · 2023-10-06T09:44:05Z

pkg/app/piped/executor/ecs/deploy.go

@@ -102,7 +102,8 @@ func (e *deployExecutor) ensureSync(ctx context.Context) model.StageStatus {
 		return model.StageStatus_STAGE_FAILURE
 	}

-	if !sync(ctx, &e.Input, e.platformProviderName, e.platformProviderCfg, taskDefinition, servicedefinition, primary) {
+	singleton := e.appCfg.QuickSync.Singleton
+	if !sync(ctx, &e.Input, e.platformProviderName, e.platformProviderCfg, singleton, taskDefinition, servicedefinition, primary) {


Do we need to update for rollback too?

I think this feature is only for the sync stage, not progressive, since we only have one task which is available at a time, isn't it?

I've forgotten most of the details regarding the code.
At this point, my main concern is to ensure that the rollback process also keeps the singleton property.

I got your point 👍 Just plan to update the rollback process logic by another PR, so we can forget about that now.

nghialv · 2023-10-06T10:01:46Z

pkg/app/piped/executor/ecs/ecs.go

+		// Scale down the service tasks by set it to 0
+		in.LogPersister.Infof("Scale down ECS desired tasks count to 0")
+		service.DesiredCount = 0
+		if _, err = client.UpdateService(ctx, *service); err != nil {


Does this function wait for the completion of all tasks before proceeding?
My guess is no. 🤔
Therefore, perhaps we should have a waiting mechanism to ensure that no tasks are still running.

I think it's not necessary 🤔 AFAIK, when the service desired count is set to 0, the termination process begins. ECS will send a stop sign to any running tasks of that service and stop traffic coming to the service (since it's marked as inactivity). The tasks can keep working with its on-flight processes/requests, but no new coming traffic is accepted. So, rather than implement a wait mechanism here, I guess it's better to implement proper handling in the application code side, wdyt?

No New Traffic: However, during this process, the tasks that are being stopped will not accept new traffic. Requests sent to the tasks will be rejected or routed to other healthy tasks if you have implemented a load balancer or service discovery mechanism.

Okay, after reviewing the old code, I think this approach will work well.
Since the old task sets will have been deleted before the desired count value is set again.

pipecd/pkg/app/piped/executor/ecs/ecs.go

Lines 237 to 243 in 20cfb77

// Remove old taskSets if existed.

for _, prevTaskSet := range prevTaskSets {

if err = client.DeleteTaskSet(ctx, *prevTaskSet); err != nil {

return err

}

}

pkg/config/application_ecs.go

Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com>

nghialv · 2023-10-09T14:27:13Z

nit: change singleton to recreate in the PR description.

* Support singleton ECS task Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> * Rename singleton to recreate Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> --------- Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com>

* Display MySQL user defined error in API Key UI (#4590) * Display MySQL user defined error in UI Signed-off-by: Kenta Kozuka <kenta.kozuka@gmail.com> * Fix Unexpected empty arrow function Signed-off-by: Kenta Kozuka <kenta.kozuka@gmail.com> * Add tests Signed-off-by: Kenta Kozuka <kenta.kozuka@gmail.com> * Run subtests in parallel Signed-off-by: Kenta Kozuka <kenta.kozuka@gmail.com> --------- Signed-off-by: Kenta Kozuka <kenta.kozuka@gmail.com> * Display MySQL user defined error of piped, command, and application (#4597) Signed-off-by: Kenta Kozuka <kenta.kozuka@gmail.com> * Add service tags as takset task on create (#4598) Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> * [ECS] Fix remove all previous active tasksets on QuickSync (#4600) * Remove previous ACTIVE tasksets if present on quicksync Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> * Remove GetPrimaryTaskSet interface Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> --------- Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> * Fix unable to fetch ECS taskset tags (#4605) Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> * Support recreate for ECS tasks (#4608) * Support singleton ECS task Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> * Rename singleton to recreate Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> --------- Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> --------- Signed-off-by: Kenta Kozuka <kenta.kozuka@gmail.com> Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> Co-authored-by: Kenta Kozuka <kenta.kozuka@gmail.com>

* Support singleton ECS task Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> * Rename singleton to recreate Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> --------- Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> Signed-off-by: moko-poi <mokopoi44@gmail.com>

* Support singleton ECS task Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> * Rename singleton to recreate Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> --------- Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com>

Support singleton ECS task

5486fa3

Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com>

khanhtc1202 requested review from nghialv, kentakozuka and ffjlabo as code owners October 6, 2023 09:31

github-actions bot added the area/go label Oct 6, 2023

nghialv reviewed Oct 6, 2023

View reviewed changes

pkg/config/application_ecs.go Outdated Show resolved Hide resolved

Rename singleton to recreate

20cfb77

Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com>

khanhtc1202 changed the title ~~Support singleton ECS task~~ Support recreate for ECS tasks Oct 6, 2023

nghialv approved these changes Oct 9, 2023

View reviewed changes

nghialv self-requested a review October 9, 2023 14:21

nghialv approved these changes Oct 9, 2023

View reviewed changes

khanhtc1202 merged commit 0f6ed52 into master Oct 10, 2023
13 of 14 checks passed

khanhtc1202 deleted the support-singleton-ecs-task branch October 10, 2023 04:05

khanhtc1202 mentioned this pull request Oct 11, 2023

Cherry-pick #4590 #4597 #4598 #4600 #4605 #4608 #4610

Merged

github-actions bot mentioned this pull request Dec 1, 2023

Release v0.45.4 #4697

Merged

github-actions bot mentioned this pull request Feb 6, 2024

Release v0.46.0 #4775

Merged

t-kikuc mentioned this pull request Mar 5, 2024

Enable to keep ECS task singleton while deploy/reconcile #4609

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support recreate for ECS tasks #4608

Support recreate for ECS tasks #4608

khanhtc1202 commented Oct 6, 2023 •

edited

codecov bot commented Oct 6, 2023 •

edited

nghialv Oct 6, 2023

khanhtc1202 Oct 6, 2023

nghialv Oct 6, 2023

khanhtc1202 Oct 6, 2023

nghialv Oct 9, 2023

nghialv Oct 6, 2023

khanhtc1202 Oct 6, 2023 •

edited

nghialv Oct 9, 2023

nghialv commented Oct 9, 2023

	// Remove old taskSets if existed.
	for _, prevTaskSet := range prevTaskSets {
	if err = client.DeleteTaskSet(ctx, *prevTaskSet); err != nil {
	return err
	}
	}

Support recreate for ECS tasks #4608

Support recreate for ECS tasks #4608

Conversation

khanhtc1202 commented Oct 6, 2023 • edited

codecov bot commented Oct 6, 2023 • edited

Codecov Report

nghialv Oct 6, 2023

Choose a reason for hiding this comment

khanhtc1202 Oct 6, 2023

Choose a reason for hiding this comment

nghialv Oct 6, 2023

Choose a reason for hiding this comment

khanhtc1202 Oct 6, 2023

Choose a reason for hiding this comment

nghialv Oct 9, 2023

Choose a reason for hiding this comment

nghialv Oct 6, 2023

Choose a reason for hiding this comment

khanhtc1202 Oct 6, 2023 • edited

Choose a reason for hiding this comment

nghialv Oct 9, 2023

Choose a reason for hiding this comment

nghialv commented Oct 9, 2023

khanhtc1202 commented Oct 6, 2023 •

edited

codecov bot commented Oct 6, 2023 •

edited

khanhtc1202 Oct 6, 2023 •

edited