[BUG] Infinite loop and memory growth in dynamic task. #5164
Labels
backlogged
For internal use. Reserved for contributor team workflow.
bug
Something isn't working
flytekit
FlyteKit Python related issue
Describe the bug
Two interesting outcomes when iterating over task output and calling sub-workflows within a dynamic task.
Within a dynamic task, if we iterate over a list of FlyteFiles outputted by another task, the dynamic task never appears to enter the first task, yet it seemingly infinitely iterates over the FlyteFile list.
e.g.
Looking at the print statements when we run this example, we see that
ff_list_task
never gets executed, yet the for loop is repeatedly executed until we manually terminate the workflow.In the above example, if we call a sub-workflow when iterating over the FlyteFile list, we see runaway memory growth.
e.g.
Even though
sub_wf
shouldn't allocate much memory and should only be called twice, simply adding thesub_wf
call causes the memory to increases until an OOM error. In example #1 the memory usage stayed flat.Expected behavior
In example #1, we would expect that
ff_list_task
would execute before the for loop. We would also expect thatfor file in batched_input_files
may error as it is iterating over a promise. Then, when the for loop does iterate, we would only expect it to iterate two times in the example (as the list only has two files) rather than infinitely.In example #2, we would expect all the same behavior as outlined above, but we would also expect memory to not increase until OOM.
Additional context to reproduce
These examples were run on union cloud and resource usage was monitored there. Flytekit version 1.11.0 was used.
Screenshots
Usage Screenshots:
![image](https://private-user-images.githubusercontent.com/40698988/318876928-3e488441-b6a7-411c-9f5a-dadcc0a10df3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5ODY4OTQsIm5iZiI6MTcyNTk4NjU5NCwicGF0aCI6Ii80MDY5ODk4OC8zMTg4NzY5MjgtM2U0ODg0NDEtYjZhNy00MTFjLTlmNWEtZGFkY2MwYTEwZGYzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDE2NDMxNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg1OWY1MzI1MTliNDljMGI5NDIzOWEwYmJmYjFkY2Q1NDZmMDk4NzM5MDA2ZDk1ZTEyMWNlNGQxZTM4MmEwZjgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.0L4pOggwgOT8WJln8YN71LfB92fyWLB1s20qzmXOliw)
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?
The text was updated successfully, but these errors were encountered: