-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
initializer does not override global variable #206
Comments
Hi Lukas, This has to do with the behavior of To use a global variable in an initializer, the function and the initializer should be moved in a separate module: # module.py
import os
testvar = 1
def func(i):
global testvar
return testvar
def init(i):
global testvar
testvar = i + os.getpid()
print(testvar) # main.py
from concurrent.futures import ProcessPoolExecutor
import os
import sys
import loky
from module import init, func
print(f"loky: {loky.__version__}")
print(f"Python: {sys.version}")
exe = loky.get_reusable_executor(max_workers=2, initializer=init, initargs=(2,))
print(list(exe.map(func, range(4))))
exe.shutdown()
print("-"*30)
with ProcessPoolExecutor(max_workers=2, initializer=init, initargs=(2,)) as exe:
print(list(exe.map(func, range(4)))) $ python main.py
loky: 2.4.2
Python: 3.7.0+ (heads/3.7:f9ae07441c, Aug 16 2018, 11:02:05)
[GCC 8.2.0]
13685
13684
[13685, 13684, 13685, 13685]
------------------------------
13688
13689
[13688, 13689, 13688, 13688] |
Thank you very much for your answer ! Would you happen to have an idea how this could be achieved inside a single Juypter Notebook? :) I just found out about this project a couple of weeks ago and like it very much - thanks again for developing it ! |
Thanks for your kind words! If you really cannot have an extra module, a quick and dirty way to do it is to mutate a global variable from a module (such as the from concurrent.futures import ProcessPoolExecutor
import os
import sys
import loky
print(f"loky: {loky.__version__}")
print(f"Python: {sys.version}")
testvar = 1
def func(i):
testvar = os.testvar
return testvar
def init(i):
os.testvar = i + os.getpid()
print(os.testvar)
exe = loky.get_reusable_executor(max_workers=2, initializer=init, initargs=(2,))
print(list(exe.map(func, range(4))))
exe.shutdown()
print("-"*30)
with ProcessPoolExecutor(max_workers=2, initializer=init, initargs=(2,)) as exe:
print(list(exe.map(func, range(4)))) This gives
Note that this is "unsafe" as it can lead to weird behavior so it should not be used for code that need to be reliable. |
Oh wow - and there is something new I learned about python today. Thank you ! |
Hey there,
i am currently trying to use the initializer feature from the get_reusable_executor function.
Because i could not get the expected results, ich compared get_reusable_executor with ProcessPoolExecutor in saw a difference in outputs.
The reusable executor does not seem to override the global variable. Can you help me to figure this out?
A minimal example can be found below, the results are identical on loky version 2.4.2 and 2.5.0dev0.
thank you very much in advance.
Greetings,
Lukas
The text was updated successfully, but these errors were encountered: