Windows support

[visr]

Thanks for open sourcing this library. I was quite excited to take it for a spin, only to get an error "no module named 'fcntl'", and learning through #10, #23 and #46 that Windows is not supported, and there are no active plans for Windows support.

That is of course fine, but I have a few related questions.

• I see ModuleNotFoundError: No module named 'fcntl' on Windows #10 has a wontfix and No Module named 'fcntl' #46 has a help wanted label. That begs the question; would you be open to accepting contributions that add Windows support?

• Do you know what the major technical obstacles to Windows support are?

• Just now I see on the Installing Metaflow page "Metaflow is available as a Python package for MacOS and Linux." Perhaps if it was followed by a more explicit "Windows is not supported.", fewer people would miss this.

• Would it be suitable to place this on the roadmap, perhaps stating that there are no Netflix plans but outside contributions are welcome?

[savingoyal]

@visr, thanks for opening the issue -

• Windows support would be a complex undertaking (given our liberal use of multiprocessing for starters). We are definitely open to contributions and a good way to kick start the conversation would be to do a POC.
• We are updating our contributing guidelines to reflect this. Expect some progress on that front in the next couple of days.

[pombredanne]

@savingoyal re:

Windows support would be a complex undertaking (given our liberal use of multiprocessing for starters). We are definitely open to contributions and a good way to kick start the conversation would be to do a POC.

FWIW, all data scientists may not be running a *nix Linux or mac. Having official support for Windows would be a big plus. I wish I could coerce them though but that's a dream. (And FWIW, WSL or Cygwin are their own can of frankenstein worms)

Multiprocessing may not be the main issue IMHO but rather:

1. keeping Python 2 and 3 compatibility is going to be a major pain on Windows

2. using signals such as here https://github.com/Netflix/metaflow/blob/c2f70c73782549fdce6c418410ecd9b305199b28/metaflow/plugins/timeout_decorator.py

3. possibly some assumptions about calling some functions and file system semantics that do not apply or are not available on Windows

For 2. Windows is a pain but I have some POSIX/windows utilities that have been battle tested for timeouts here https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/interrupt.py

For 1. we did that in https://github.com/nexB/scancode-toolkit and I would never do it again which would mean to support Python 3 for Windows. Anything else is a sure way to destroy grey matter very quickly ;)

[pablodz]

And what about news about Windows supporting Linux subsystem?

[pombredanne]

My latest thinking is that Windows support is too mucho pita and with the new WSL2, running on Windows means running on Linux, which is a sane thing. I would close this once some doc is added that WSL2 must be used

[erowan]

I was hoping that a full remote k8s execution of a flow would work on windows but it doesn't. I stubbed out fcntl but hit this polling dependency. I am in a corporate setup so windows is unfortunately my client. I could develop flows locally in docker but it would be nice to be able to execute the flow in k8s from windows.

$ python 00-helloworld/helloworld.py run --with kubernetes:service_account=auser                                
Metaflow 2.5.2.post7+git6b07668 executing HelloFlow for user:U038178                                                
Validating your flow...                                                                                             
    The graph looks good!                                                                                           
Running pylint...                                                                                                   
    Pylint is happy!                                                                                                
Polling is not supported on your operating system (Windows)                                                         
2022-03-17 13:23:24.242 Workflow starting (run-id 1647523400851411):                                                
2022-03-17 13:23:24.628 [1647523400851411/start/1 (pid 34040)] Task is starting.                                    
2022-03-17 13:23:24.629 Workflow failed.                                                                            
2022-03-17 13:23:24.630 Terminating 1 active tasks...                                                               
2022-03-17 13:23:24.631 [1647523400851411/start/1 (pid 34040)] [KILLED BY ORCHESTRATOR]                             
2022-03-17 13:23:24.632 [1647523400851411/start/1 (pid 34040)] [KILLED BY ORCHESTRATOR]                             
2022-03-17 13:23:29.000 Killing 1 remaining tasks after having waited for 5 seconds -- some tasks may not exit clean
2022-03-17 13:23:29.002 Flushing logs...                                                                            
    Internal error                                                                                                  
Traceback (most recent call last):                                                                                  
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 209, in execute                     
    self._launch_workers()                                                                                          
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 564, in _launch_workers             
    self._launch_worker(task)                                                                                       
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 584, in _launch_worker              
    self._poll.add(fd)                                                                                              
AttributeError: 'NoneType' object has no attribute 'add'                                                            
                                                                                                                    
During handling of the above exception, another exception occurred:                                                 
                                                                                                                    
Traceback (most recent call last):                                                                                  
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 1102, in main                           
    start(auto_envvar_prefix="METAFLOW", obj=state)                                                                 
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 829, in __call__         
    return self.main(args, kwargs)                                                                                  
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 782, in main             
    rv = self.invoke(ctx)                                                                                           
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 1259, in invoke          
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                         
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 1066, in invoke          
    return ctx.invoke(self.callback, ctx.params)                                                                    
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 610, in invoke           
    return callback(args, kwargs)                                                                                   
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 666, in wrapper                         
    return func(args, kwargs)                                                                                       
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\decorators.py", line 33, in new_func    
    return f(get_current_context().obj, args, kwargs)                                                               
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 789, in run                             
    runtime.execute()                                                                                               
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 230, in execute                     
    self._killall()                                                                                                 
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 275, in _killall                    
    list(self._poll_workers())                                                                                      
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 521, in _poll_workers               
    for event in self._poll.poll(PROGRESS_INTERVAL):                                                                
AttributeError: 'NoneType' object has no attribute 'poll'                   

I see Argo support might be forthcoming #50 would this be an option for starting flows in k8s from windows or is windows as a client completely out?

Also its not clear to me how the code is packaged? I see decorators for conda & image and reference to the code being tarball'd to S3. Can you explain or point me to the docs that explain the code packaging for k8s. Thanks.

[savingoyal]

@erowan Would you like to try out #992 in your windows environment? Here is the code that packages up your flow.