Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Foreach over list parameter results in None value for input #651

lsimoneau opened this issue Aug 19, 2021 · 3 comments

Foreach over list parameter results in None value for input #651

lsimoneau opened this issue Aug 19, 2021 · 3 comments


Copy link

We have a flow with a JSONType parameter containing a list of values. We attempted to split into a foreach step across the parameter. Metaflow splits off the correct number of tasks (one per value in the list), but in each task the input value is None.

I would expect this to either fail validation (tell us we can't use a parameter in a foreach) or Just Work™ and make the value available in the split tasks.

Example flow:

from metaflow import FlowSpec, Parameter, step, JSONType

class ForeachFlow(FlowSpec):
    numbers_param = Parameter(
    def start(self):
        # This works, and passes each number to the run_number step:
        # self.numbers = self.numbers_param
        #, foreach='numbers')
        # But this doesn't:, foreach='numbers_param')
    def run_number(self):
        print(f"number is {self.input}")
    def join(self, inputs):
    def end(self):
if __name__ == '__main__':

This results in:

2021-08-19 21:50:37.718 Workflow starting (run-id 390):
2021-08-19 21:50:37.801 [390/start/82041 (pid 8328)] Task is starting.
2021-08-19 21:50:40.221 [390/start/82041 (pid 8328)] Foreach yields 3 child steps.
2021-08-19 21:50:40.222 [390/start/82041 (pid 8328)] Task finished successfully.
2021-08-19 21:50:40.558 [390/run_number/82042 (pid 8368)] Task is starting.
2021-08-19 21:50:40.684 [390/run_number/82043 (pid 8371)] Task is starting.
2021-08-19 21:50:40.868 [390/run_number/82044 (pid 8374)] Task is starting.
2021-08-19 21:50:42.566 [390/run_number/82043 (pid 8371)] number is None
2021-08-19 21:50:42.667 [390/run_number/82042 (pid 8368)] number is None
2021-08-19 21:50:42.748 [390/run_number/82044 (pid 8374)] number is None
2021-08-19 21:50:43.879 [390/run_number/82043 (pid 8371)] Task finished successfully.
2021-08-19 21:50:44.633 [390/run_number/82042 (pid 8368)] Task finished successfully.
2021-08-19 21:50:45.196 [390/run_number/82044 (pid 8374)] Task finished successfully.
2021-08-19 21:50:45.602 [390/join/82045 (pid 8500)] Task is starting.
2021-08-19 21:50:48.733 [390/join/82045 (pid 8500)] Task finished successfully.
2021-08-19 21:50:48.998 [390/end/82046 (pid 8561)] Task is starting.
2021-08-19 21:50:51.324 [390/end/82046 (pid 8561)] Task finished successfully.
Copy link

savingoyal commented Aug 19, 2021

Taking a look! This looks like a bug.

Edit - can confirm this is a bug. We will roll out a fix.

Copy link

#652 fixes and closes this issue.

Copy link

This issue has been fixed in Metaflow 2.3.5 -

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

No branches or pull requests

3 participants