New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiprocessing runtime error in Windows 64 bit #399

Closed
Garyfallidis opened this Issue Jul 30, 2014 · 3 comments

Comments

Projects
None yet
2 participants
@Garyfallidis
Member

Garyfallidis commented Jul 30, 2014

This one is for @gabknight .

In 64 bit Windows we get the following runtime error when we run peaks_from_models with parallel=True

File "C:\Users\Eleftherios\Anaconda\lib\multiprocessing\process.py", line 130,
in start
self._popen = Popen(self)
File "C:\Users\Eleftherios\Anaconda\lib\multiprocessing\forking.py", line 258,
in init
cmd = get_command_line() + [rhandle]
File "C:\Users\Eleftherios\Anaconda\lib\multiprocessing\forking.py", line 358,
in get_command_line
is not going to be frozen to produce a Windows executable.''')
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.

        This probably means that you are on Windows and you have
        forgotten to use the proper idiom in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce a Windows executable.

The error suggests to add freeze_support which actually we tried and worked. But that means that we need to change our tutorials to check if Windows then add freeze_support.

@gabknight can you check this problem and see how it can be written in a nice way and maybe be hidden from the user?
It's a bit ugly to have to add freeze_support in your main function, it would be nicer if that actually happens inside the peaks_from_model function.

Thx!

@gabknight

This comment has been minimized.

Contributor

gabknight commented Jul 31, 2014

@Garyfallidis I did not know this issue. From what I found, freeze_support() needs to be at top level, straight after

if __name__ == '__main__':
    freeze_support()

multiprocessing.freeze_support does a platform check:

def freeze_support():
    '''
    Check whether this is a fake forked process in a frozen executable.
    If so then run code specified by commandline and exit.
    '''
    if sys.platform == 'win32' and getattr(sys, 'frozen', False):
        from multiprocessing.forking import freeze_support
        freeze_support()

An other option could be to do this in _peaks_from_model_parallel:

if sys.platform == 'win32' and getattr(sys, 'frozen', False):
    warn("Multiprocessing is not supported for frozen windows executable. \
                 returns peaks_from_model(..., parallel=False).")
    return peaks_from_model(model, data, sphere,
                                    relative_peak_threshold,
                                    min_separation_angle, mask, return_odf,
                                    return_sh, gfa_thr, normalize_peaks,
                                    sh_order, sh_basis_type, npeaks,
                                    parallel=False)
@Garyfallidis

This comment has been minimized.

Member

Garyfallidis commented Jul 31, 2014

It seems that if you just use

if name == 'main':

in windows then the problem stops even without running freeze_support().
But maybe using freeze_support is even safer.

If you do what you suggest. I mean write a warning inside peaks_from_models you should also suggest the solution in the warning. Say that they have to add freeze_support() in their main.

Also you may need to update our tutorials that use peaks_from_models to have a main. Which means also we need to make sure that the tutorials render nicely in the website.

I would suggest to take action asap because this is a serious error. The tutorials never end and use a lot of memory for our windows users.

@Garyfallidis

This comment has been minimized.

Member

Garyfallidis commented Jul 31, 2014

Okay, what I said before that you may not need to call freeze_support (just use main) in some cases is not true. I tested it with two different windows machines and one of the two worked only with freeze_support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment