Skip to content
This repository has been archived by the owner on Oct 9, 2023. It is now read-only.

Fixing max parallelism #611

Merged
merged 2 commits into from
Aug 30, 2023
Merged

Fixing max parallelism #611

merged 2 commits into from
Aug 30, 2023

Conversation

pingsutw
Copy link
Member

TL;DR

I run two tasks in a workflow, but max Parallelism is being set to 4.

{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n0","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:582","tasktype":"python-task","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"Parallelism now set to [1].","ts":"2023-08-29T14:37:15-07:00"}
{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n0","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:540","tasktype":"python-task","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"Parallelism now set to [2].","ts":"2023-08-29T14:37:15-07:00"}
{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n0","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:91","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"regular node detected, (no future file found)","ts":"2023-08-29T14:37:15-07:00"}
{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n1","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:183","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"Dynamic handler.Handle's called with phase 0.","ts":"2023-08-29T14:37:15-07:00"}
{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n1","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:582","tasktype":"python-task","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"Parallelism now set to [3].","ts":"2023-08-29T14:37:15-07:00"}
{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n1","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:540","tasktype":"python-task","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"Parallelism now set to [4].","ts":"2023-08-29T14:37:15-07:00"}

We use defer func to update the max parallelism, so we don't need to update the parallelism in other places in the handle()

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

^^^

Tracking Issue

NA

Follow-up issue

NA

Signed-off-by: Kevin Su <pingsutw@apache.org>
@codecov
Copy link

codecov bot commented Aug 29, 2023

Codecov Report

Merging #611 (5198686) into master (ebc0ae7) will increase coverage by 0.42%.
The diff coverage is n/a.

❗ Current head 5198686 differs from pull request most recent head 3448fcd. Consider uploading reports for the commit 3448fcd to get more accurate results

Additional details and impacted files

@hamersaw hamersaw merged commit 476d416 into master Aug 30, 2023
14 checks passed
@hamersaw hamersaw deleted the fixing-parallelism branch August 30, 2023 01:50
eapolinario pushed a commit that referenced this pull request Sep 6, 2023
Signed-off-by: Kevin Su <pingsutw@apache.org>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
2 participants