Type Error: unsopported operand type(s) for -: 'function and function'

[mlittmanabbvie]

Look into the FAQ of the readme. Can the bug be resolved by one of those solutions?
Not in the FAQs

Describe the bug

TypeError: unsupported operand type(s) for -: 'function' and 'function'

Code to reproduce the behavior
'''
from hyperactive import Hyperactive
from hyperactive import RepulsingHillClimbingOptimizer, RandomAnnealingOptimizer

  class Parameters:
      def __init__(self):
          self.x = 5
          
  finp = Parameters()
  def ret_df():
      return df
  
  def func_minl(opts):
      return opts['slope'] + opts['exp']
    
  h = Hyperactive(["progress_bar", "print_results", "print_times"])
  search_space = {'exp':list(range(0, 5)),
                 'slope': list(np.arange(.001,10,step = .05)),
                 'freq_mult':list(np.arange(1,2.5,.005)),
                 'clust':[5],
                  'df': [ret_df],
                  'finp': [finp],
                  'asc': [False],
                  'use_pca':[False],
                  'last':[False],
                  'disc_type':['type']
                  }
  h.add_search(func_minl, search_space = search_space, n_iter = 10, optimizer = 
      RepulsingHillClimbingOptimizer(epsilon=0.05,
      distribution="normal",n_neighbours=3,rand_rest_p=0.03,repulsion_factor=3), n_jobs = 1, max_score = None,initialize = 
      {'warm_start' : [{'exp':2,'slope':5,'freq_mult':1.5,'clust':5,
                  'df': ret_df,
                  'finp': finp,
                  'asc': False,
                  'use_pca':False,
                  'last':False,
                  'disc_type':'type'
                  }]}, early_stopping = {'tol_rel':1, 'n_iter_no_change':3},random_state = 0, memory= True, memory_warm_start = None)

'''

Error message from command line
When adding a dataframe as a parameter in the search space, by using a function as mentioned in the documentation, I am receiving an error
--> 973 h.add_search(func_minl, search_space = search_space, n_iter = maxiter, optimizer = RepulsingHillClimbingOptimizer(epsilon=0.05,
974 distribution="normal",
975 n_neighbours=3,

~\Anaconda3\lib\site-packages\hyperactive\hyperactive.py in add_search(self, objective_function, search_space, n_iter, search_id, optimizer, n_jobs, initialize, max_score, early_stopping, random_state, memory, memory_warm_start, progress_board)
148 self.check_list(search_space)
149
--> 150 optimizer.init(search_space, initialize, progress_collector)
151
152 self._add_search_processes(

~\Anaconda3\lib\site-packages\hyperactive\optimizers\gfo_wrapper.py in init(self, search_space, initialize, progress_collector)
76 self.trafo = HyperGradientTrafo(search_space)
77
---> 78 initialize = self.trafo.trafo_initialize(initialize)
79 search_space_positions = self.trafo.search_space_positions
80

~\Anaconda3\lib\site-packages\hyperactive\optimizers\hyper_gradient_trafo.py in trafo_initialize(self, initialize)
113 for warm_start_ in warm_start:
114 value = self.para2value(warm_start_)
--> 115 position = self.value2position(value)
116 pos_para = self.value2para(position)
117

~\Anaconda3\lib\site-packages\hyperactive\optimizers\hyper_gradient_trafo.py in value2position(self, value)
16 position = []
17 for n, space_dim in enumerate(self.search_space_values):
---> 18 pos = np.abs(value[n] - np.array(space_dim)).argmin()
19 position.append(int(pos))
20
TypeError: unsupported operand type(s) for -: 'function' and 'function'

System information:

• OS Platform and Distribution
• Windows 10
• Python version 3.8.8
• Hyperactive version 3.3.2

Additional context

[SimonBlanke]

Hello @mlittmanabbvie,

thank you for the information about this issue, but I think there is some code missing. I must be able to take your example, run it and reproduce the error.

Only part of your code is in a code-block. Please make sure to put the code into a code-block like this:
```python

```
This makes it much easier for me to help you.

[mlittmanabbvie]

Fixed the code block, it was right below the actual block. Thank you for your help in advance!

[SimonBlanke]

Thank you very much!
There is still the problem, that I cannot run the code. I need a full example, that I can run. Without a complete script it is almost impossible for me to figure out the issue.

I need a code block that I can just copy into a *.py file and execute (to get the same error as you).

[mlittmanabbvie]

Updated code block, you can run the block directly now

[SimonBlanke]

Very good! :-) I will look into this issue later today.

[SimonBlanke]

I looked into this issue. I am certain, that it occurs because of a problem of the transformation between the hyperactive warm start into gfo warm start. I am surprised i missed this test case! I will fix this tomorrow (and write multiple tests) and release it in v3.3.3.

Thank you very much for pointing me to this problem! :-)

[mlittmanabbvie]

Thank you for looking into this! By the way, it also fails for strings, it says unsupported for type string and string. My guess would be that the line pos = np.abs(value[n] - np.array(space_dim)).argmin() is essentially assuming the value to be a numeric type which is why you then can't subtract a function from a function or a string from a string.

[SimonBlanke]

Your guess was right! ;-) It is a part of the code that converts the hyperactive parameter into gfo parameter. I already fixed it by checking the types in the search space beforehand and using the fast method for numeric search space dimensions:

pos = np.abs(value[n] - np.array(space_dim)).argmin()

If the type is not numeric but string or other python objects another method is used to convert the parameter.

This works fine for most cases! But there is an issue if the search space contains class instances and the number of jobs is > 1. For some reason the class instances change their position in memory, so that the comparison during the hyper -> gfo-conversion fails. I think this could be a problem with pickle.

The solution for all this would be something I am already planing for Hyperactive v4:
The search space should only contain numbers, strings or functions. If you want to put a class into the search space you can wrap it into a function:

class test_class:
  def __init__(self):
    pass

def class_wrapper1():
  return test_class

I think this should get around this problem.

I will continue (and probably finish) this tomorrow. I want to make sure that this is very stable and thoroughly tested.

[SimonBlanke]

After some more testing I discovered a problem with classes wrapped in functions in the search space when using multiprocessing (n_jobs>1 or multiple .add_search()). The package dill cannot pickle those objects. A solution could be to use cloudpickle, which is included in joblib for parallel computing. Joblib can already be used in Hyperactive see joblib tests (test_joblib_0 and follows). I will test if joblib (+ cloudpickle) solves the problem of classes in the search space.

@mlittmanabbvie If you need a solution fast you can clone the master, go into the dir and run make reinstall. This patches the warm start error (if you do not use multiprocessing). Otherwise you can wait for the official solution in v3.3.3.

[mlittmanabbvie]

Thank you so much for all of your help with this! I have another question, but I will open a new ticket for it

[SimonBlanke]

I released v3.3.3 of Hyperactive yesterday. It contains the fix to solve this issue, multiple new tests and more fixes and stability improvements for some related problems that might occur.

It is recommended to only use numbers, strings or functions in the search space. If you want to put an array, dataframe, class, ... into the search space you should wrap it into a function beforehand and put the wrapper-function into the search space. This is important for services like the warm-start (initialization) or the memory-warm-start, because Hyperactive needs an identifier for the parameter in the search space. Numbers and strings are good identifiers, because you can compare them with == or is, but this can be much more complicated with classes, lists or other objects.
This is why functions as wrappers are a better solution: Hyperactive recognizes the function by getting its name (a string) via .__name__. This is a proper identifier to compare functions with each other. For now Hyperactive throws a warning if the search space does not contain numbers, strings or functions, but in future versions (v4) Hyperactive will raise a ValueError.

There was also a problem pickling classes, objects and objects that are not top level. To solve this problem I added support for the pathos library. It uses cloudpickle to give the user more flexibility for pickling objects during parallel computing.

[mlittmanabbvie]

Is there an example of how the pathos library gets used? I just tried and I am getting the error "AttributeError: Can't pickle local object 'optimize..func_min'" when passing a local function into the optimizer. Will open new ticket for it.