Problem with buckshot and lattice when using ray

[yolking]

Hi @mmckerns, thank you for such powerful project!
I am trying to use mystic for optimization of multiple functions in parallel on Windows. They are dependent on other functions in file and in helper .py files. I tried pathos way at first, but couldn't figure out how to import all such functions (on which optimized functions depend) at once to Pool. So i decided to try recently ported ray library on Windows with mystic. It works perfectly fine if I use diffev2, but it looks like diffev2 is quite bad at global optimization for me, so I am leaning towards lattice or buckshot. They work fine without parallelization. Unfortunately, with ray if I change solver to one of those I start getting error messages about imported classes from helper .py files and libraries such as NameError: MyClass is not defined and NameError: name 'pandas' is not defined.

Simplified version of my code looks like this:

import ray
from helper.Calculator import MyClass #my class i have problem importing
from mystic.monitors import VerboseMonitor
from mystic.solvers import diffev2, buckshot, lattice

#data - my data
#func_set - set of functions I am trying to optimise

ray.init()
def opt_func(x, somefunc, data):
    w = somefunc(x , data)
    e = MyClass(w,data) 
    res = e.mymethod(w)
    return( res*-1)
@integers()
def round(x):
    return x

@ray.remote
def f(key,data):
    numb= 3
    P_bounds = 1,1000
    result = diffev2(opt_func, x0= [100] * numb, bounds=[P_bounds] * numb,  gtol=10, npop=30, constraints=round, 
          disp=True, full_output=True, itermon=VerboseMonitor(), args = (func,data)) #this works
    result =  buckshot(opt_func, npts =10, ndim = numb, bounds=[P_bounds] * numb, gtol=20, constraints=round,
          disp=True, full_output=True, itermon=VerboseMonitor(), args = ( func[key],data))  #this doesn't work
    result = lattice(opt_func, nbins =10, ndim = numb, bounds=[P_bounds] * numb, gtol=20, constraints=round,
          disp=True, full_output=True, itermon=VerboseMonitor(), args = ( func[key],data)) #this doesn't work

data_id = ray.put(data) 
result_ids = []
for myfunc in func_set:
        result_ids.append(f.remote(myfunc, data_id))
ray.shutdown()

Since diffev2 works fine, I am guessing there is something different about how pseudo-global methods import stuff. I tried importing MyClass inside opt_func since there Error happens, but I receive new error ImportError: __import__ not found. Same happens with full definitions of these optimizers. Can I somehow make these optimizers work too?

[mmckerns]

@yolking: a few comments... (1) diffev2, and most solvers, need tuning. Which means, they can work poorly until you play with the settings, termination conditions, constraints, and so on until it works. (2) It'd be helpful for you to post your traceback. (3) You may want to try to use a ThreadPoolto see if that works, just to collect more information.

I believe that the issue may be that ray cannot serialize your code. You said you also had difficulty with pathos? It may require you to clean up the import structure of your code. Again, tracebacks can help here. Generally, any serializer is going to struggle with locally imported helper files, so typically you want to avoid that.

I should add that I haven't tried using mystic with ray before.

[yolking]

Thank you for your comments, @mmckerns. I agree, I haven't looked close enough at diffev2 parameters. I've seen only examples using standard parameters, but now I see there is more to it. I'll look into it. Though since I don't have any idea about starting points I think this can be the slowest way.
I tried using ThreadPool and it works without errors. I think it's obvious with other pools the problem is that I have a lot of dependencies, which are not in a scope of optimizers during parallel optimization due to how multiprocess works. It's difficult to rewrite my code to fix this and that's why I switched to ray.

About ray problem I thought this has something to do with how lattice and buckshot classes inherit scopes with variables differently from diffev2 or something like that since diffev2 works fine. That's the only reason I opened this issue since ray support on Windows is currently in alpha phase there are many reasons why it can have problems with mystic on ray side.

Here is traceback from lattice launch with ray:

2020-10-08 22:19:42,566	ERROR worker.py:1018 -- Possible unhandled error from worker: ray::f() (pid=29764, ip=192.168.31.253)
  File "python\ray\_raylet.pyx", line 484, in ray._raylet.execute_task
  File "<ipython-input-60-7ab75046726e>", line 53, in f
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\ensemble.py", line 241, in lattice
    ExtraArgs=args,callback=callback)
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\abstract_solver.py", line 937, in Solve
    self._Solve(cost, ExtraArgs, **settings)
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\abstract_ensemble_solver.py", line 653, in _Solve
    results = list(self._map(_solve, op, iv, vb, cb, **self._mapconfig))
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\python_map.py", line 66, in python_map
    result = list(map(func, *arglist)) #     see pathos.pyina.ez_map
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\abstract_ensemble_solver.py", line 638, in _solve
    solver.Solve(disp=disp,callback=callback)
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\scipy_optimize.py", line 360, in Solve
    ExtraArgs, **kwds)
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\abstract_solver.py", line 937, in Solve
    self._Solve(cost, ExtraArgs, **settings)
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\abstract_solver.py", line 880, in _Solve
    stop = self.Step(**settings) #XXX: remove need to pass settings?
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\abstract_solver.py", line 853, in Step
    self._Step(**kwds) #FIXME: not all kwds are given in __doc__
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\scipy_optimize.py", line 233, in _Step
    fsim[0] = cost(x0)
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\tools.py", line 353, in function_wrapper
    return outer_function(inner_function(_x))
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\tools.py", line 364, in function_wrapper
    return cost_function(_x) + penalty_function(_x)
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\tools.py", line 402, in function_wrapper
    return target_function(x)
  File "C:\Users\user\Anaconda3\envs\env\lib\site-packages\mystic\tools.py", line 375, in function_wrapper
    fval = the_function(x, *extra_args)
  File "<ipython-input-60-7ab75046726e>", line 30, in opt_func
NameError: name 'MyClass' is not defined