-
Notifications
You must be signed in to change notification settings - Fork 276
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Only transform the literal that matches the map task index #2484
Conversation
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
…/flytekit into array-node-input-transformation
Signed-off-by: Kevin Su <pingsutw@apache.org>
…-input-transformation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2484 +/- ##
===========================================
- Coverage 73.97% 50.98% -23.00%
===========================================
Files 250 185 -65
Lines 21592 18677 -2915
Branches 3644 3645 +1
===========================================
- Hits 15973 9522 -6451
- Misses 4973 8674 +3701
+ Partials 646 481 -165 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
…2484) * Only transform the literal that matches the map task index Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * kevin's update Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * make it work locally Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * add a test Signed-off-by: Kevin Su <pingsutw@apache.org> * check (flyteorg#2486) Signed-off-by: Kevin Su <pingsutw@apache.org> * Only fail for unbound batch pickled inputs for map tasks Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Unit tests Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: bugra.gedik <bugra.gedik@predera.ai>
* Only transform the literal that matches the map task index Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * kevin's update Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * make it work locally Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * add a test Signed-off-by: Kevin Su <pingsutw@apache.org> * check (#2486) Signed-off-by: Kevin Su <pingsutw@apache.org> * Only fail for unbound batch pickled inputs for map tasks Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Unit tests Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jan Fiedler <jan@union.ai>
Tracking issue
fixes flyteorg/flyte#5483
Why are the changes needed?
Existing code only trims map task inputs before invoking the python subtask. This is incorrect as it leads to flytekit unnecessarily downloading all inputs for the entire maptask (potentially 10s of thousands) for each subtask.
What changes were proposed in this pull request?
For local execution, continue with the current behavior of handling all subtasks inputs when the map task is invoked.
For remote execution, index into the inputs list using the maptask index and only transform this input.
How was this patch tested?
Setup process
Screenshots
Check all the applicable boxes