Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Foreach over list parameter results in None value for input #651

Closed
lsimoneau opened this issue Aug 19, 2021 · 3 comments
Closed

Foreach over list parameter results in None value for input #651

lsimoneau opened this issue Aug 19, 2021 · 3 comments

Comments

@lsimoneau
Copy link

We have a flow with a JSONType parameter containing a list of values. We attempted to split into a foreach step across the parameter. Metaflow splits off the correct number of tasks (one per value in the list), but in each task the input value is None.

I would expect this to either fail validation (tell us we can't use a parameter in a foreach) or Just Work™ and make the value available in the split tasks.

Example flow:

from metaflow import FlowSpec, Parameter, step, JSONType

class ForeachFlow(FlowSpec):
    numbers_param = Parameter(
        "numbers_param",
        type=JSONType,
        default='[1,2,3]'
    )
    
    @step
    def start(self):
        # This works, and passes each number to the run_number step:
        #
        # self.numbers = self.numbers_param
        # self.next(self.run_number, foreach='numbers')
        
        # But this doesn't:
        self.next(self.run_number, foreach='numbers_param')
        
    @step
    def run_number(self):
        print(f"number is {self.input}")
        self.next(self.join)
        
    @step
    def join(self, inputs):
        self.next(self.end)
        
    @step
    def end(self):
        pass
    
if __name__ == '__main__':
    ForeachFlow()

This results in:

2021-08-19 21:50:37.718 Workflow starting (run-id 390):
2021-08-19 21:50:37.801 [390/start/82041 (pid 8328)] Task is starting.
2021-08-19 21:50:40.221 [390/start/82041 (pid 8328)] Foreach yields 3 child steps.
2021-08-19 21:50:40.222 [390/start/82041 (pid 8328)] Task finished successfully.
2021-08-19 21:50:40.558 [390/run_number/82042 (pid 8368)] Task is starting.
2021-08-19 21:50:40.684 [390/run_number/82043 (pid 8371)] Task is starting.
2021-08-19 21:50:40.868 [390/run_number/82044 (pid 8374)] Task is starting.
2021-08-19 21:50:42.566 [390/run_number/82043 (pid 8371)] number is None
2021-08-19 21:50:42.667 [390/run_number/82042 (pid 8368)] number is None
2021-08-19 21:50:42.748 [390/run_number/82044 (pid 8374)] number is None
2021-08-19 21:50:43.879 [390/run_number/82043 (pid 8371)] Task finished successfully.
2021-08-19 21:50:44.633 [390/run_number/82042 (pid 8368)] Task finished successfully.
2021-08-19 21:50:45.196 [390/run_number/82044 (pid 8374)] Task finished successfully.
2021-08-19 21:50:45.602 [390/join/82045 (pid 8500)] Task is starting.
2021-08-19 21:50:48.733 [390/join/82045 (pid 8500)] Task finished successfully.
2021-08-19 21:50:48.998 [390/end/82046 (pid 8561)] Task is starting.
2021-08-19 21:50:51.324 [390/end/82046 (pid 8561)] Task finished successfully.
@savingoyal
Copy link
Collaborator

savingoyal commented Aug 19, 2021

Taking a look! This looks like a bug.

Edit - can confirm this is a bug. We will roll out a fix.

@savingoyal
Copy link
Collaborator

#652 fixes and closes this issue.

@savingoyal
Copy link
Collaborator

This issue has been fixed in Metaflow 2.3.5 - https://github.com/Netflix/metaflow/releases/tag/2.3.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants